[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:30:00 dist-test-slave-gwfx krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:30:00 dist-test-slave-gwfx krb5kdc[395](info): set up 2 sockets
Sep 26 23:30:00 dist-test-slave-gwfx krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 23:30:01.800031 388 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.003s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:30:01 dist-test-slave-gwfx krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:30:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:30:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:30:01.993049 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35737
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35737
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:30:02.436853 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:02.437389 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:02.437503 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:02.446792 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:30:02.446973 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:02.447024 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:02.447114 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:30:02.447173 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:30:02.460243 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:02.462172 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:02.464114 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:02.482342 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:02.482453 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:02.482329 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:02.488370 408 server_base.cc:1034] running on GCE node
I20260926 23:30:02.489622 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:02.491897 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:02.495414 408 hybrid_clock.cc:648] HybridClock initialized: now 1790465402495214 us; error 136 us; skew 500 ppm
I20260926 23:30:02.496140 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:02 dist-test-slave-gwfx krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:02.503159 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:30:02.506249 408 webserver.cc:460] Webserver started at http://127.0.97.62:45167/ using document root <none> and password file <none>
I20260926 23:30:02.507825 408 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:02.508019 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:02.508528 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:02.514325 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "be2e4886ca0a4a9e8dfe15aee1ebd58d"
format_stamp: "Formatted at 2026-09-26 23:30:02 on dist-test-slave-gwfx"
server_key: "1ff008ee3c33da270e3a1306ebb684c4"
server_key_iv: "1462155b32069a7b181b358969150c2b"
server_key_version: "encryptionkey@0"
I20260926 23:30:02.515626 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "be2e4886ca0a4a9e8dfe15aee1ebd58d"
format_stamp: "Formatted at 2026-09-26 23:30:02 on dist-test-slave-gwfx"
server_key: "1ff008ee3c33da270e3a1306ebb684c4"
server_key_iv: "1462155b32069a7b181b358969150c2b"
server_key_version: "encryptionkey@0"
I20260926 23:30:02.525168 408 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.002s sys 0.005s
I20260926 23:30:02.531147 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:02.533453 408 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260926 23:30:02.533732 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "be2e4886ca0a4a9e8dfe15aee1ebd58d"
format_stamp: "Formatted at 2026-09-26 23:30:02 on dist-test-slave-gwfx"
server_key: "1ff008ee3c33da270e3a1306ebb684c4"
server_key_iv: "1462155b32069a7b181b358969150c2b"
server_key_version: "encryptionkey@0"
I20260926 23:30:02.533986 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:02.572238 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:02.578550 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:02.578950 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:02.603777 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35737
I20260926 23:30:02.603785 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35737 every 8 connection(s)
I20260926 23:30:02.606277 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:30:02.609863 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 408
I20260926 23:30:02.610282 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:30:02.613318 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:02.613636 388 external_mini_cluster.cc:1428] Setting key 35da22c41619f00d2410392cc19caeee
I20260926 23:30:02.627768 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be2e4886ca0a4a9e8dfe15aee1ebd58d: Bootstrap starting.
I20260926 23:30:02.631505 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be2e4886ca0a4a9e8dfe15aee1ebd58d: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:02.633121 475 log.cc:824] T 00000000000000000000000000000000 P be2e4886ca0a4a9e8dfe15aee1ebd58d: Log is configured to *not* fsync() on all Append() calls
I20260926 23:30:02.637581 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be2e4886ca0a4a9e8dfe15aee1ebd58d: No bootstrap required, opened a new log
I20260926 23:30:02.649066 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P be2e4886ca0a4a9e8dfe15aee1ebd58d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be2e4886ca0a4a9e8dfe15aee1ebd58d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35737 } }
I20260926 23:30:02.649524 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P be2e4886ca0a4a9e8dfe15aee1ebd58d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:02.649627 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P be2e4886ca0a4a9e8dfe15aee1ebd58d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be2e4886ca0a4a9e8dfe15aee1ebd58d, State: Initialized, Role: FOLLOWER
I20260926 23:30:02.650327 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P be2e4886ca0a4a9e8dfe15aee1ebd58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be2e4886ca0a4a9e8dfe15aee1ebd58d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35737 } }
I20260926 23:30:02.650604 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P be2e4886ca0a4a9e8dfe15aee1ebd58d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:30:02.650760 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P be2e4886ca0a4a9e8dfe15aee1ebd58d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:30:02.650979 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be2e4886ca0a4a9e8dfe15aee1ebd58d [term 0 FOLLOWER]: Advancing to term 1
Sep 26 23:30:02 dist-test-slave-gwfx krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:02.667044 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:02.620066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55178 (local address 127.0.97.62:35737)
0926 23:30:02.621320 (+ 1254us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:02.621342 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:02.623185 (+ 1843us) server_negotiation.cc:407] Connection header received
0926 23:30:02.634361 (+ 11176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:02.634409 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:02.635067 (+ 658us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:02.635620 (+ 553us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:02.641301 (+ 5681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:02.643780 (+ 2479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:02.646520 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:02.648214 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:02.648494 (+ 280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:02.657446 (+ 8952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:02.657458 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:02.657477 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:02.657528 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:02.661113 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:02.662226 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:02.662235 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:02.662241 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:02.662444 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:02.663022 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:02.663026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:02.663028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:02.663656 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:02.663961 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:02.664951 (+ 990us) server_negotiation.cc:299] Negotiation successful
0926 23:30:02.665397 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":906,"thread_start_us":566,"threads_started":1}
I20260926 23:30:02.726956 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P be2e4886ca0a4a9e8dfe15aee1ebd58d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be2e4886ca0a4a9e8dfe15aee1ebd58d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35737 } }
I20260926 23:30:02.727746 475 leader_election.cc:302] T 00000000000000000000000000000000 P be2e4886ca0a4a9e8dfe15aee1ebd58d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be2e4886ca0a4a9e8dfe15aee1ebd58d; no voters:
I20260926 23:30:02.728490 475 leader_election.cc:288] T 00000000000000000000000000000000 P be2e4886ca0a4a9e8dfe15aee1ebd58d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:30:02.728768 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be2e4886ca0a4a9e8dfe15aee1ebd58d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:30:02.730139 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P be2e4886ca0a4a9e8dfe15aee1ebd58d [term 1 LEADER]: Becoming Leader. State: Replica: be2e4886ca0a4a9e8dfe15aee1ebd58d, State: Running, Role: LEADER
I20260926 23:30:02.731026 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P be2e4886ca0a4a9e8dfe15aee1ebd58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be2e4886ca0a4a9e8dfe15aee1ebd58d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35737 } }
I20260926 23:30:02.731405 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P be2e4886ca0a4a9e8dfe15aee1ebd58d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:30:02.736052 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P be2e4886ca0a4a9e8dfe15aee1ebd58d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be2e4886ca0a4a9e8dfe15aee1ebd58d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be2e4886ca0a4a9e8dfe15aee1ebd58d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35737 } } }
I20260926 23:30:02.736476 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P be2e4886ca0a4a9e8dfe15aee1ebd58d [sys.catalog]: SysCatalogTable state changed. Reason: New leader be2e4886ca0a4a9e8dfe15aee1ebd58d. Latest consensus state: current_term: 1 leader_uuid: "be2e4886ca0a4a9e8dfe15aee1ebd58d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be2e4886ca0a4a9e8dfe15aee1ebd58d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35737 } } }
I20260926 23:30:02.736934 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P be2e4886ca0a4a9e8dfe15aee1ebd58d [sys.catalog]: This master's current role is: LEADER
I20260926 23:30:02.736935 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P be2e4886ca0a4a9e8dfe15aee1ebd58d [sys.catalog]: This master's current role is: LEADER
I20260926 23:30:02.738557 489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:30:02.746816 489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:30:02.760911 489 catalog_manager.cc:1347] Generated new cluster ID: a10ada034bff4909b315dcf8ede4c049
I20260926 23:30:02.761066 489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:30:02.784740 489 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:30:02.786310 489 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:30:02.795791 489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be2e4886ca0a4a9e8dfe15aee1ebd58d: Generated new TSK 0
I20260926 23:30:02.796868 489 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
W20260926 23:30:03.630625 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.1: real 0.809s user 0.004s sys 0.009s
I20260926 23:30:03.632158 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35737
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:30:04.086787 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:04.087198 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:04.087355 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:04.096618 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:04.096822 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:04.096998 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:30:04.108691 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35737
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:04.110616 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:04.112565 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:04.131319 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:04.131425 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:04.131845 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:04.132865 501 server_base.cc:1034] running on GCE node
I20260926 23:30:04.134016 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:04.135586 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:04.136940 501 hybrid_clock.cc:648] HybridClock initialized: now 1790465404136828 us; error 82 us; skew 500 ppm
I20260926 23:30:04.137732 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:04 dist-test-slave-gwfx krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:04.142678 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:30:04.144917 501 webserver.cc:460] Webserver started at http://127.0.97.1:32979/ using document root <none> and password file <none>
I20260926 23:30:04.146029 501 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:04.146368 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:04.146965 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:04.151278 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-0/data/instance:
uuid: "5ed9b207dfdc4a3896abce8be2e1bdf4"
format_stamp: "Formatted at 2026-09-26 23:30:04 on dist-test-slave-gwfx"
server_key: "efe80e64feca790814998a8e528ad8f0"
server_key_iv: "3b3eb8d6f706211815c899fdecdd6553"
server_key_version: "encryptionkey@0"
I20260926 23:30:04.152352 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5ed9b207dfdc4a3896abce8be2e1bdf4"
format_stamp: "Formatted at 2026-09-26 23:30:04 on dist-test-slave-gwfx"
server_key: "efe80e64feca790814998a8e528ad8f0"
server_key_iv: "3b3eb8d6f706211815c899fdecdd6553"
server_key_version: "encryptionkey@0"
I20260926 23:30:04.159358 501 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260926 23:30:04.164086 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:04.166201 501 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 23:30:04.166448 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "5ed9b207dfdc4a3896abce8be2e1bdf4"
format_stamp: "Formatted at 2026-09-26 23:30:04 on dist-test-slave-gwfx"
server_key: "efe80e64feca790814998a8e528ad8f0"
server_key_iv: "3b3eb8d6f706211815c899fdecdd6553"
server_key_version: "encryptionkey@0"
I20260926 23:30:04.166719 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:04.185583 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:04.191545 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:04.191949 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:04.193102 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:30:04.194841 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:30:04.194940 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:04.195076 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:30:04.195129 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:04.237908 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45673
I20260926 23:30:04.237936 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45673 every 8 connection(s)
I20260926 23:30:04.239391 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:30:04.247040 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 501
I20260926 23:30:04.247289 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:30:04.248139 388 external_mini_cluster.cc:1428] Setting key c5c2244ed4e053223eb3a0a478a0f2da
Sep 26 23:30:04 dist-test-slave-gwfx krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 23:30:04.267683 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:04.242783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46283 (local address 127.0.97.62:35737)
0926 23:30:04.243368 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:04.243373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:04.244583 (+ 1210us) server_negotiation.cc:407] Connection header received
0926 23:30:04.248305 (+ 3722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:04.248312 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:04.248421 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:04.248560 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:04.250763 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:04.251588 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:04.254464 (+ 2876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:04.255016 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:04.255160 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:04.259885 (+ 4725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:04.259895 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:04.259900 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:04.259958 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:04.263370 (+ 3412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:04.264529 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:04.264537 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:04.264541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:04.264749 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:04.265316 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:04.265322 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:04.265325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:04.265721 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:04.265908 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:04.266992 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0926 23:30:04.267256 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":348,"threads_started":1}
I20260926 23:30:04.268896 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:04.243420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35737 (local address 127.0.97.1:46283)
0926 23:30:04.244358 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:04.244411 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:04.247720 (+ 3309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:04.248963 (+ 1243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:04.248976 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:04.249663 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:04.250458 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:04.250481 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:04.251785 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:04.251792 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:04.254191 (+ 2399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:04.254211 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:04.255323 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:04.255330 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:04.255662 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:04.256620 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:04.256666 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:04.259442 (+ 2776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:04.263672 (+ 4230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:04.263684 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:04.263705 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:04.264329 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:04.264924 (+ 595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:04.264928 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:04.264931 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:04.265184 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:04.265918 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:04.265926 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:04.266457 (+ 531us) client_negotiation.cc:769] Sending connection context
0926 23:30:04.266849 (+ 392us) client_negotiation.cc:240] Negotiation successful
0926 23:30:04.267391 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":337,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 23:30:04.271365 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35737
I20260926 23:30:04.271885 629 heartbeater.cc:464] Registering TS with master...
I20260926 23:30:04.272933 629 heartbeater.cc:511] Master 127.0.97.62:35737 requested a full tablet report, sending...
I20260926 23:30:04.275933 440 ts_manager.cc:194] Registered new tserver with Master: 5ed9b207dfdc4a3896abce8be2e1bdf4 (127.0.97.1:45673)
I20260926 23:30:04.278206 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:30:04.299155 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35737
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:30:04.746713 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:04.747102 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:04.747220 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:04.755442 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:04.755592 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:04.755734 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:30:04.766716 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35737
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:04.768427 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:04.770085 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:04.786980 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:04.787603 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:04.787670 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:04.788271 635 server_base.cc:1034] running on GCE node
I20260926 23:30:04.789048 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:04.790194 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:04.791461 635 hybrid_clock.cc:648] HybridClock initialized: now 1790465404791402 us; error 46 us; skew 500 ppm
I20260926 23:30:04.791930 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:04 dist-test-slave-gwfx krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:04.796299 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:30:04.798420 635 webserver.cc:460] Webserver started at http://127.0.97.2:46031/ using document root <none> and password file <none>
I20260926 23:30:04.799419 635 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:04.799552 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:04.799907 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:04.803606 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-1/data/instance:
uuid: "87d8352eb4a643e28e2c008f622cef70"
format_stamp: "Formatted at 2026-09-26 23:30:04 on dist-test-slave-gwfx"
server_key: "bbd95eddf47a490035db9c915d877f00"
server_key_iv: "93f96239ccef57962e73ccbe7ae937b3"
server_key_version: "encryptionkey@0"
I20260926 23:30:04.804553 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-1/wal/instance:
uuid: "87d8352eb4a643e28e2c008f622cef70"
format_stamp: "Formatted at 2026-09-26 23:30:04 on dist-test-slave-gwfx"
server_key: "bbd95eddf47a490035db9c915d877f00"
server_key_iv: "93f96239ccef57962e73ccbe7ae937b3"
server_key_version: "encryptionkey@0"
I20260926 23:30:04.812065 635 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260926 23:30:04.818037 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:04.820643 635 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260926 23:30:04.820960 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "87d8352eb4a643e28e2c008f622cef70"
format_stamp: "Formatted at 2026-09-26 23:30:04 on dist-test-slave-gwfx"
server_key: "bbd95eddf47a490035db9c915d877f00"
server_key_iv: "93f96239ccef57962e73ccbe7ae937b3"
server_key_version: "encryptionkey@0"
I20260926 23:30:04.821280 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:04.841145 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:04.847434 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:04.847811 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:04.849128 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:30:04.851008 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:30:04.851105 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:04.851215 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:30:04.851259 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:04.901677 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34201
I20260926 23:30:04.901721 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34201 every 8 connection(s)
I20260926 23:30:04.903697 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:30:04.908932 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 635
I20260926 23:30:04.909119 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:30:04.909711 388 external_mini_cluster.cc:1428] Setting key 91f374f7de50632a1ff1b6bb77ad552a
Sep 26 23:30:04 dist-test-slave-gwfx krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 23:30:04.936955 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:04.908762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47329 (local address 127.0.97.62:35737)
0926 23:30:04.909402 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:04.909408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:04.911560 (+ 2152us) server_negotiation.cc:407] Connection header received
0926 23:30:04.914551 (+ 2991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:04.914559 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:04.914740 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:04.914865 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:04.917631 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:04.919266 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:04.921564 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:04.922081 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:04.922244 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:04.927900 (+ 5656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:04.927908 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:04.927912 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:04.927968 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:04.931159 (+ 3191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:04.932493 (+ 1334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:04.932500 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:04.932503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:04.932817 (+ 314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:04.933965 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:04.933972 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:04.933976 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:04.934441 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:04.934650 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:04.935995 (+ 1345us) server_negotiation.cc:299] Negotiation successful
0926 23:30:04.936468 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":376,"threads_started":1}
I20260926 23:30:04.938930 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:04.909406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35737 (local address 127.0.97.2:47329)
0926 23:30:04.911288 (+ 1882us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:04.911365 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:04.914035 (+ 2670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:04.915289 (+ 1254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:04.915305 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:04.916115 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:04.917076 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:04.917115 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:04.919615 (+ 2500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:04.919621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:04.921306 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:04.921331 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:04.922418 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:04.922427 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:04.922824 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:04.923693 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:04.923724 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:04.927364 (+ 3640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:04.931487 (+ 4123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:04.931498 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:04.931518 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:04.932278 (+ 760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:04.933137 (+ 859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:04.933142 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:04.933146 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:04.933669 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:04.934721 (+ 1052us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:04.934736 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:04.935324 (+ 588us) client_negotiation.cc:769] Sending connection context
0926 23:30:04.935712 (+ 388us) client_negotiation.cc:240] Negotiation successful
0926 23:30:04.936681 (+ 969us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1328,"thread_start_us":842,"threads_started":1}
I20260926 23:30:04.942274 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35737
I20260926 23:30:04.942812 763 heartbeater.cc:464] Registering TS with master...
I20260926 23:30:04.943822 763 heartbeater.cc:511] Master 127.0.97.62:35737 requested a full tablet report, sending...
I20260926 23:30:04.946314 440 ts_manager.cc:194] Registered new tserver with Master: 87d8352eb4a643e28e2c008f622cef70 (127.0.97.2:34201)
I20260926 23:30:04.948122 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47329
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:30:04.968134 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35737
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:30:05.282440 629 heartbeater.cc:503] Master 127.0.97.62:35737 was elected leader, sending a full tablet report...
W20260926 23:30:05.447314 769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:05.447650 769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:05.447758 769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:05.456343 769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:05.456481 769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:05.456599 769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:30:05.468204 769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35737
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:05.470160 769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:05.472013 769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:05.488847 775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:05.489032 774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:05.489651 777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:05.490183 769 server_base.cc:1034] running on GCE node
I20260926 23:30:05.491235 769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:05.492555 769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:05.493855 769 hybrid_clock.cc:648] HybridClock initialized: now 1790465405493766 us; error 75 us; skew 500 ppm
I20260926 23:30:05.494266 769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:05 dist-test-slave-gwfx krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465405, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:05.498347 769 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:30:05.501008 769 webserver.cc:460] Webserver started at http://127.0.97.3:41531/ using document root <none> and password file <none>
I20260926 23:30:05.501868 769 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:05.502007 769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:05.502341 769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:05.505798 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-2/data/instance:
uuid: "94176d8daef24c9589c2038789fc411e"
format_stamp: "Formatted at 2026-09-26 23:30:05 on dist-test-slave-gwfx"
server_key: "002da2e0536c1f6a3d9eb6e397a3e0d0"
server_key_iv: "064fee5fa4ba332842cdc0c4386386b8"
server_key_version: "encryptionkey@0"
I20260926 23:30:05.506762 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-2/wal/instance:
uuid: "94176d8daef24c9589c2038789fc411e"
format_stamp: "Formatted at 2026-09-26 23:30:05 on dist-test-slave-gwfx"
server_key: "002da2e0536c1f6a3d9eb6e397a3e0d0"
server_key_iv: "064fee5fa4ba332842cdc0c4386386b8"
server_key_version: "encryptionkey@0"
I20260926 23:30:05.514348 769 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260926 23:30:05.520212 784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:05.522341 769 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:30:05.522593 769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "94176d8daef24c9589c2038789fc411e"
format_stamp: "Formatted at 2026-09-26 23:30:05 on dist-test-slave-gwfx"
server_key: "002da2e0536c1f6a3d9eb6e397a3e0d0"
server_key_iv: "064fee5fa4ba332842cdc0c4386386b8"
server_key_version: "encryptionkey@0"
I20260926 23:30:05.523116 769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:05.538812 769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:05.543951 769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:05.544237 769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:05.545305 769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:30:05.547634 769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:30:05.547726 769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:05.547816 769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:30:05.547875 769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:05.590600 769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35693
I20260926 23:30:05.590750 896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35693 every 8 connection(s)
I20260926 23:30:05.592051 769 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:30:05.596153 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 769
I20260926 23:30:05.596349 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790465400694755-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:30:05.596875 388 external_mini_cluster.cc:1428] Setting key 2a0788ca7946354017b49cc9bd89cafa
Sep 26 23:30:05 dist-test-slave-gwfx krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465405, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:05.620765 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:05.595926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51963 (local address 127.0.97.62:35737)
0926 23:30:05.596507 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:05.596515 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:05.598308 (+ 1793us) server_negotiation.cc:407] Connection header received
0926 23:30:05.600845 (+ 2537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:05.600852 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:05.601054 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:05.601172 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:05.603616 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:05.605069 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:05.608018 (+ 2949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:05.608554 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:05.608712 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:05.614063 (+ 5351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:05.614069 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:05.614072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:05.614118 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:05.617231 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:05.618169 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:05.618173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:05.618176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:05.618317 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:05.618793 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:05.618798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:05.618801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:05.619136 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:05.619274 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:05.620077 (+ 803us) server_negotiation.cc:299] Negotiation successful
0926 23:30:05.620361 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":320,"threads_started":1}
I20260926 23:30:05.621858 900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:05.596711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35737 (local address 127.0.97.3:51963)
0926 23:30:05.598015 (+ 1304us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:05.598101 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:05.600469 (+ 2368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:05.601515 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:05.601527 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:05.602414 (+ 887us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:05.603301 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:05.603332 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:05.605316 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:05.605323 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:05.607735 (+ 2412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:05.607758 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:05.608879 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:05.608886 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:05.609237 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:05.610219 (+ 982us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:05.610258 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:05.613746 (+ 3488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:05.617451 (+ 3705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:05.617461 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:05.617481 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:05.618019 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:05.618431 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:05.618436 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:05.618440 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:05.618641 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:05.619269 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:05.619289 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:05.619691 (+ 402us) client_negotiation.cc:769] Sending connection context
0926 23:30:05.619955 (+ 264us) client_negotiation.cc:240] Negotiation successful
0926 23:30:05.620558 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"thread_start_us":476,"threads_started":1}
I20260926 23:30:05.624248 897 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35737
I20260926 23:30:05.624668 897 heartbeater.cc:464] Registering TS with master...
I20260926 23:30:05.625645 897 heartbeater.cc:511] Master 127.0.97.62:35737 requested a full tablet report, sending...
I20260926 23:30:05.627876 440 ts_manager.cc:194] Registered new tserver with Master: 94176d8daef24c9589c2038789fc411e (127.0.97.3:35693)
I20260926 23:30:05.629171 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51963
I20260926 23:30:05.631207 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 23:30:05 dist-test-slave-gwfx krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465405, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 23:30:05 dist-test-slave-gwfx krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465405, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:30:05.688549 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:05.668135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59798 (local address 127.0.97.1:45673)
0926 23:30:05.668830 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:05.668837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:05.668989 (+ 152us) server_negotiation.cc:407] Connection header received
0926 23:30:05.669237 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:05.669246 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:05.669501 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:05.669674 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:05.671463 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:05.672919 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:05.674813 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:05.675505 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:05.675666 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:05.680557 (+ 4891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:05.680580 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:05.680604 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:05.680678 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:05.684561 (+ 3883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:05.685706 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:05.685717 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:05.685720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:05.685937 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:05.686659 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:05.686666 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:05.686669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:05.687215 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:05.687553 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:05.687976 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 23:30:05.688232 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":359,"threads_started":1}
W20260926 23:30:05.690119 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59798
Sep 26 23:30:05 dist-test-slave-gwfx krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 23:30:05 dist-test-slave-gwfx krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:30:05.745263 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:05.724283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59812 (local address 127.0.97.1:45673)
0926 23:30:05.724624 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:05.724631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:05.725765 (+ 1134us) server_negotiation.cc:407] Connection header received
0926 23:30:05.726024 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:05.726029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:05.726117 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:05.726218 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:05.728760 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:05.729761 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:05.731336 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:05.731802 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:05.731969 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:05.737192 (+ 5223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:05.737200 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:05.737203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:05.737259 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:05.741016 (+ 3757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:05.742068 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:05.742074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:05.742077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:05.742350 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:05.742844 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:05.742848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:05.742850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:05.743212 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:05.743434 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:05.743861 (+ 427us) server_negotiation.cc:299] Negotiation successful
0926 23:30:05.744109 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260926 23:30:05.747568 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 501
I20260926 23:30:05.763013 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:05.748926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59824 (local address 127.0.97.1:45673)
0926 23:30:05.749165 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:05.749170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:05.749513 (+ 343us) server_negotiation.cc:407] Connection header received
0926 23:30:05.749637 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:05.749642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:05.749718 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:05.749828 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:05.751174 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:05.751999 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:05.753473 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:05.753958 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:05.754070 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:05.756791 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:05.756797 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:05.756800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:05.756850 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:05.759987 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:05.760885 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:05.760890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:05.760894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:05.761085 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:05.761496 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:05.761501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:05.761503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:05.761813 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:05.762053 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:05.762418 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 23:30:05.762636 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 23:30:05.764021 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:30:05.850946 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 501
I20260926 23:30:05.867343 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 635
Sep 26 23:30:05 dist-test-slave-gwfx krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:30:05.898316 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:05.868331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42928 (local address 127.0.97.2:34201)
0926 23:30:05.868978 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:05.868987 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:05.869017 (+ 30us) server_negotiation.cc:407] Connection header received
0926 23:30:05.869082 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:05.869089 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:05.869317 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:05.869491 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:05.870680 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:05.871789 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:05.873081 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:05.873659 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:05.873780 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:05.891824 (+ 18044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:05.891844 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:05.891863 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:05.891958 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:05.895338 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:05.896110 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:05.896118 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:05.896120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:05.896259 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:05.896630 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:05.896634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:05.896637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:05.897006 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:05.897322 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:05.897680 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 23:30:05.897964 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":290,"threads_started":1}
I20260926 23:30:05.900283 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:30:05.989423 763 heartbeater.cc:503] Master 127.0.97.62:35737 was elected leader, sending a full tablet report...
I20260926 23:30:05.990839 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 635
W20260926 23:30:06.013417 426 connection.cc:535] server connection from 127.0.97.2:47329 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 23:30:06.013818 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 769
Sep 26 23:30:06 dist-test-slave-gwfx krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:30:06.034814 914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:06.015197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54538 (local address 127.0.97.3:35693)
0926 23:30:06.016154 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:06.016164 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:06.016205 (+ 41us) server_negotiation.cc:407] Connection header received
0926 23:30:06.016283 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:06.016289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:06.016692 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:06.016990 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:06.018798 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:06.019880 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:06.021642 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:06.022262 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:06.022445 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:06.027986 (+ 5541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:06.028006 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:06.028036 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:06.028094 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:06.031527 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:06.032660 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:06.032669 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:06.032671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:06.032839 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:06.033241 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:06.033244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:06.033245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:06.033641 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:06.033925 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:06.034213 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 23:30:06.034442 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":549,"threads_started":1}
I20260926 23:30:06.036674 892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:30:06.127137 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 769
I20260926 23:30:06.144024 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 408
I20260926 23:30:06.145617 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:30:06.253382 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 408
2026-09-26T23:30:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5505 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:30:06 dist-test-slave-gwfx krb5kdc[919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:30:06 dist-test-slave-gwfx krb5kdc[919](info): set up 2 sockets
Sep 26 23:30:06 dist-test-slave-gwfx krb5kdc[919](info): commencing operation
krb5kdc: starting...
W20260926 23:30:07.287220 388 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:30:07 dist-test-slave-gwfx krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:30:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:30:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:30:07.503757 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38685
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38685
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:30:07.915243 932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:07.915696 932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:07.915802 932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:07.924595 932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:30:07.924844 932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:07.924909 932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:07.924988 932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:30:07.925148 932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:30:07.937323 932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:07.939244 932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:07.941038 932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:07.956930 938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:07.957247 940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:07.957217 937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:07.964308 932 server_base.cc:1034] running on GCE node
I20260926 23:30:07.965838 932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:07.969147 932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:07.970727 932 hybrid_clock.cc:648] HybridClock initialized: now 1790465407970597 us; error 67 us; skew 500 ppm
I20260926 23:30:07.971375 932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:07 dist-test-slave-gwfx krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:07.975973 932 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:30:07.978609 932 webserver.cc:460] Webserver started at http://127.0.97.62:34111/ using document root <none> and password file <none>
I20260926 23:30:07.979756 932 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:07.979948 932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:07.980355 932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:07.984725 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "1064b530962a4d5d92f5a5ac694cad9d"
format_stamp: "Formatted at 2026-09-26 23:30:07 on dist-test-slave-gwfx"
server_key: "94007cfa8931ff7a0c81c18a5c1a1c71"
server_key_iv: "41a85bb11071100c144a8cf37ffa4181"
server_key_version: "encryptionkey@0"
I20260926 23:30:07.985919 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "1064b530962a4d5d92f5a5ac694cad9d"
format_stamp: "Formatted at 2026-09-26 23:30:07 on dist-test-slave-gwfx"
server_key: "94007cfa8931ff7a0c81c18a5c1a1c71"
server_key_iv: "41a85bb11071100c144a8cf37ffa4181"
server_key_version: "encryptionkey@0"
I20260926 23:30:07.995272 932 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.011s sys 0.000s
I20260926 23:30:08.000582 947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:08.002883 932 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.001s
I20260926 23:30:08.003155 932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "1064b530962a4d5d92f5a5ac694cad9d"
format_stamp: "Formatted at 2026-09-26 23:30:07 on dist-test-slave-gwfx"
server_key: "94007cfa8931ff7a0c81c18a5c1a1c71"
server_key_iv: "41a85bb11071100c144a8cf37ffa4181"
server_key_version: "encryptionkey@0"
I20260926 23:30:08.003417 932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:08.022764 932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:08.029168 932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:08.029505 932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:08.053568 932 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38685
I20260926 23:30:08.053671 998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38685 every 8 connection(s)
I20260926 23:30:08.055918 932 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:30:08.057088 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 932
I20260926 23:30:08.057329 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:30:08.058028 388 external_mini_cluster.cc:1428] Setting key be2a56d0a31bd55026abeba07630365b
I20260926 23:30:08.061971 999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 23:30:08 dist-test-slave-gwfx krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:08.077605 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1064b530962a4d5d92f5a5ac694cad9d: Bootstrap starting.
I20260926 23:30:08.082864 999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1064b530962a4d5d92f5a5ac694cad9d: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:08.084569 999 log.cc:824] T 00000000000000000000000000000000 P 1064b530962a4d5d92f5a5ac694cad9d: Log is configured to *not* fsync() on all Append() calls
I20260926 23:30:08.088056 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:08.060642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38490 (local address 127.0.97.62:38685)
0926 23:30:08.061630 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:08.061650 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:08.061717 (+ 67us) server_negotiation.cc:407] Connection header received
0926 23:30:08.064301 (+ 2584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:08.064341 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:08.065014 (+ 673us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:08.065587 (+ 573us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:08.067177 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:08.069385 (+ 2208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:08.070963 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:08.072106 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:08.072341 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:08.077021 (+ 4680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:08.077035 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:08.077064 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:08.077140 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:08.081740 (+ 4600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:08.082885 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:08.082897 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:08.082910 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:08.083202 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:08.083797 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:08.083802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:08.083806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:08.084751 (+ 945us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:08.085146 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:08.085740 (+ 594us) server_negotiation.cc:299] Negotiation successful
0926 23:30:08.086296 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":385,"threads_started":1}
I20260926 23:30:08.089095 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1064b530962a4d5d92f5a5ac694cad9d: No bootstrap required, opened a new log
I20260926 23:30:08.097834 999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1064b530962a4d5d92f5a5ac694cad9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1064b530962a4d5d92f5a5ac694cad9d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38685 } }
I20260926 23:30:08.098371 999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1064b530962a4d5d92f5a5ac694cad9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:08.098507 999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1064b530962a4d5d92f5a5ac694cad9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1064b530962a4d5d92f5a5ac694cad9d, State: Initialized, Role: FOLLOWER
I20260926 23:30:08.099347 999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1064b530962a4d5d92f5a5ac694cad9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1064b530962a4d5d92f5a5ac694cad9d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38685 } }
I20260926 23:30:08.099661 999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1064b530962a4d5d92f5a5ac694cad9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:30:08.099869 999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1064b530962a4d5d92f5a5ac694cad9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:30:08.100080 999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1064b530962a4d5d92f5a5ac694cad9d [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:08.103209 999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1064b530962a4d5d92f5a5ac694cad9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1064b530962a4d5d92f5a5ac694cad9d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38685 } }
I20260926 23:30:08.103852 999 leader_election.cc:302] T 00000000000000000000000000000000 P 1064b530962a4d5d92f5a5ac694cad9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1064b530962a4d5d92f5a5ac694cad9d; no voters:
I20260926 23:30:08.104512 999 leader_election.cc:288] T 00000000000000000000000000000000 P 1064b530962a4d5d92f5a5ac694cad9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:30:08.104759 1004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1064b530962a4d5d92f5a5ac694cad9d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:30:08.107013 1004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1064b530962a4d5d92f5a5ac694cad9d [term 1 LEADER]: Becoming Leader. State: Replica: 1064b530962a4d5d92f5a5ac694cad9d, State: Running, Role: LEADER
I20260926 23:30:08.107231 999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1064b530962a4d5d92f5a5ac694cad9d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:30:08.107802 1004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1064b530962a4d5d92f5a5ac694cad9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1064b530962a4d5d92f5a5ac694cad9d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38685 } }
I20260926 23:30:08.111461 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1064b530962a4d5d92f5a5ac694cad9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1064b530962a4d5d92f5a5ac694cad9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1064b530962a4d5d92f5a5ac694cad9d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38685 } } }
I20260926 23:30:08.111907 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1064b530962a4d5d92f5a5ac694cad9d [sys.catalog]: This master's current role is: LEADER
I20260926 23:30:08.112602 1006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1064b530962a4d5d92f5a5ac694cad9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1064b530962a4d5d92f5a5ac694cad9d. Latest consensus state: current_term: 1 leader_uuid: "1064b530962a4d5d92f5a5ac694cad9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1064b530962a4d5d92f5a5ac694cad9d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38685 } } }
I20260926 23:30:08.113026 1006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1064b530962a4d5d92f5a5ac694cad9d [sys.catalog]: This master's current role is: LEADER
I20260926 23:30:08.114459 1014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:30:08.120798 1014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:30:08.134133 1014 catalog_manager.cc:1347] Generated new cluster ID: caef19b52dee41679bd4405cc8fad7f7
I20260926 23:30:08.134315 1014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:30:08.159708 1014 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:30:08.161248 1014 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:30:08.173619 1014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1064b530962a4d5d92f5a5ac694cad9d: Generated new TSK 0
I20260926 23:30:08.174602 1014 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:30:08.240675 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38685
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:30:08.672933 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:08.673365 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:08.673503 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:08.681579 1025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:08.681744 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:08.681914 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:30:08.693113 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38685
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:08.694906 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:08.696650 1025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:08.713529 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:08.713588 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:08.714361 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:08.715006 1025 server_base.cc:1034] running on GCE node
I20260926 23:30:08.715869 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:08.717147 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:08.718403 1025 hybrid_clock.cc:648] HybridClock initialized: now 1790465408718337 us; error 58 us; skew 500 ppm
I20260926 23:30:08.718834 1025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:08 dist-test-slave-gwfx krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:08.722666 1025 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:30:08.724874 1025 webserver.cc:460] Webserver started at http://127.0.97.1:42265/ using document root <none> and password file <none>
I20260926 23:30:08.725765 1025 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:08.725899 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:08.726249 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:08.730062 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-0/data/instance:
uuid: "2eab0413912640f6a98b09125dee59ed"
format_stamp: "Formatted at 2026-09-26 23:30:08 on dist-test-slave-gwfx"
server_key: "ef7507e1a2504e9a1a9fc42f0dc187ff"
server_key_iv: "2a35311ecd85bbdc7fae90bc37f36922"
server_key_version: "encryptionkey@0"
I20260926 23:30:08.731122 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2eab0413912640f6a98b09125dee59ed"
format_stamp: "Formatted at 2026-09-26 23:30:08 on dist-test-slave-gwfx"
server_key: "ef7507e1a2504e9a1a9fc42f0dc187ff"
server_key_iv: "2a35311ecd85bbdc7fae90bc37f36922"
server_key_version: "encryptionkey@0"
I20260926 23:30:08.738037 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s
I20260926 23:30:08.743235 1040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:08.745261 1025 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 23:30:08.745481 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "2eab0413912640f6a98b09125dee59ed"
format_stamp: "Formatted at 2026-09-26 23:30:08 on dist-test-slave-gwfx"
server_key: "ef7507e1a2504e9a1a9fc42f0dc187ff"
server_key_iv: "2a35311ecd85bbdc7fae90bc37f36922"
server_key_version: "encryptionkey@0"
I20260926 23:30:08.745715 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:08.777822 1025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:08.783442 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:08.783756 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:08.785096 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:30:08.786852 1025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:30:08.786955 1025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:08.787061 1025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:30:08.787117 1025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:08.825488 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42615
I20260926 23:30:08.825598 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42615 every 8 connection(s)
I20260926 23:30:08.827275 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:30:08.836441 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 1025
I20260926 23:30:08.836654 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:30:08.837410 388 external_mini_cluster.cc:1428] Setting key c55f2dcb887a64b030b5ee0527ebadd5
Sep 26 23:30:08 dist-test-slave-gwfx krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:08.853678 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:08.830864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45385 (local address 127.0.97.62:38685)
0926 23:30:08.831451 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:08.831456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:08.832924 (+ 1468us) server_negotiation.cc:407] Connection header received
0926 23:30:08.835010 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:08.835016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:08.835116 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:08.835259 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:08.837616 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:08.838598 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:08.840944 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:08.841432 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:08.841588 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:08.847187 (+ 5599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:08.847194 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:08.847198 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:08.847247 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:08.849771 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:08.851092 (+ 1321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:08.851098 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:08.851101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:08.851284 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:08.851860 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:08.851864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:08.851866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:08.852152 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:08.852304 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:08.853081 (+ 777us) server_negotiation.cc:299] Negotiation successful
0926 23:30:08.853328 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":302,"threads_started":1}
I20260926 23:30:08.854856 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:08.831699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38685 (local address 127.0.97.1:45385)
0926 23:30:08.832698 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:08.832761 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:08.834628 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:08.835536 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:08.835546 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:08.836337 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:08.837297 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:08.837325 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:08.838857 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:08.838861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:08.840705 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:08.840724 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:08.841740 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:08.841748 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:08.842064 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:08.843141 (+ 1077us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:08.843173 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:08.846794 (+ 3621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:08.850033 (+ 3239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:08.850049 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:08.850074 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:08.850932 (+ 858us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:08.851446 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:08.851452 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:08.851455 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:08.851716 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:08.852293 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:08.852303 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:08.852682 (+ 379us) client_negotiation.cc:769] Sending connection context
0926 23:30:08.852951 (+ 269us) client_negotiation.cc:240] Negotiation successful
0926 23:30:08.853543 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":387,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 23:30:08.857184 1153 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38685
I20260926 23:30:08.857584 1153 heartbeater.cc:464] Registering TS with master...
I20260926 23:30:08.858500 1153 heartbeater.cc:511] Master 127.0.97.62:38685 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 23:30:08.861806 964 ts_manager.cc:194] Registered new tserver with Master: 2eab0413912640f6a98b09125dee59ed (127.0.97.1:42615)
I20260926 23:30:08.864125 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45385
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:30:08.888423 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38685
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:30:09.287793 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:09.288113 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:09.288234 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:09.295707 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:09.295917 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:09.296059 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:30:09.307047 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38685
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:09.308969 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:09.310546 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:09.330621 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:09.330953 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:09.330953 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:09.332660 1159 server_base.cc:1034] running on GCE node
I20260926 23:30:09.333513 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:09.334822 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:09.338318 1159 hybrid_clock.cc:648] HybridClock initialized: now 1790465409338225 us; error 62 us; skew 500 ppm
I20260926 23:30:09.338851 1159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:09 dist-test-slave-gwfx krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:09.343266 1159 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:30:09.345183 1159 webserver.cc:460] Webserver started at http://127.0.97.2:41647/ using document root <none> and password file <none>
I20260926 23:30:09.345971 1159 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:09.346192 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:09.346984 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:09.350487 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-1/data/instance:
uuid: "1b05042674514bbca2c8fa3971d4cd59"
format_stamp: "Formatted at 2026-09-26 23:30:09 on dist-test-slave-gwfx"
server_key: "d940b23374cf1e9c60159660b82c25cf"
server_key_iv: "579191a82cadfaa25653f0af33ea51a9"
server_key_version: "encryptionkey@0"
I20260926 23:30:09.351517 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1b05042674514bbca2c8fa3971d4cd59"
format_stamp: "Formatted at 2026-09-26 23:30:09 on dist-test-slave-gwfx"
server_key: "d940b23374cf1e9c60159660b82c25cf"
server_key_iv: "579191a82cadfaa25653f0af33ea51a9"
server_key_version: "encryptionkey@0"
I20260926 23:30:09.358273 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 23:30:09.363317 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:09.365254 1159 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 23:30:09.365458 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "1b05042674514bbca2c8fa3971d4cd59"
format_stamp: "Formatted at 2026-09-26 23:30:09 on dist-test-slave-gwfx"
server_key: "d940b23374cf1e9c60159660b82c25cf"
server_key_iv: "579191a82cadfaa25653f0af33ea51a9"
server_key_version: "encryptionkey@0"
I20260926 23:30:09.365713 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:09.380461 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:09.387604 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:09.387943 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:09.389158 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:30:09.390919 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:30:09.391019 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:09.391132 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:30:09.391179 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:09.430454 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41963
I20260926 23:30:09.430500 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41963 every 8 connection(s)
I20260926 23:30:09.432287 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:30:09.432924 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 1159
I20260926 23:30:09.433103 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:30:09.433712 388 external_mini_cluster.cc:1428] Setting key f36a98195ee534b64a3fbc4a92060fe5
Sep 26 23:30:09 dist-test-slave-gwfx krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 23:30:09.463049 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:09.436306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35451 (local address 127.0.97.62:38685)
0926 23:30:09.436823 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:09.436828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:09.438943 (+ 2115us) server_negotiation.cc:407] Connection header received
0926 23:30:09.441920 (+ 2977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:09.441927 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:09.442061 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:09.442180 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:09.445025 (+ 2845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:09.446263 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:09.448957 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:09.449684 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:09.449910 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:09.455035 (+ 5125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:09.455046 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:09.455052 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:09.455118 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:09.458759 (+ 3641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:09.460018 (+ 1259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:09.460025 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:09.460027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:09.460234 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:09.460880 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:09.460887 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:09.460891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:09.461247 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:09.461419 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:09.462380 (+ 961us) server_negotiation.cc:299] Negotiation successful
0926 23:30:09.462652 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":264,"threads_started":1}
I20260926 23:30:09.463971 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:09.437279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38685 (local address 127.0.97.2:35451)
0926 23:30:09.438648 (+ 1369us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:09.438782 (+ 134us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:09.441456 (+ 2674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:09.442587 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:09.442600 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:09.443696 (+ 1096us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:09.444627 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:09.444659 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:09.446602 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:09.446608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:09.448638 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:09.448672 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:09.450116 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:09.450123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:09.450538 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:09.451753 (+ 1215us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:09.451788 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:09.454545 (+ 2757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:09.459056 (+ 4511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:09.459103 (+ 47us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:09.459122 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:09.459714 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:09.460408 (+ 694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:09.460412 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:09.460415 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:09.460675 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:09.461423 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:09.461431 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:09.461867 (+ 436us) client_negotiation.cc:769] Sending connection context
0926 23:30:09.462167 (+ 300us) client_negotiation.cc:240] Negotiation successful
0926 23:30:09.462651 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"thread_start_us":674,"threads_started":1}
I20260926 23:30:09.466306 1287 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38685
I20260926 23:30:09.467034 1287 heartbeater.cc:464] Registering TS with master...
I20260926 23:30:09.468231 1287 heartbeater.cc:511] Master 127.0.97.62:38685 requested a full tablet report, sending...
I20260926 23:30:09.470394 964 ts_manager.cc:194] Registered new tserver with Master: 1b05042674514bbca2c8fa3971d4cd59 (127.0.97.2:41963)
I20260926 23:30:09.471457 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
W20260926 23:30:09.579741 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.3: real 0.142s user 0.000s sys 0.011s
I20260926 23:30:09.580564 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38685
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:30:09.868561 1153 heartbeater.cc:503] Master 127.0.97.62:38685 was elected leader, sending a full tablet report...
W20260926 23:30:10.003214 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:10.003616 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:10.003755 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:10.012691 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:10.012859 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:10.013038 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:30:10.024418 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38685
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:10.026249 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:10.028059 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:10.044623 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:10.044559 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:10.044564 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:10.045329 1293 server_base.cc:1034] running on GCE node
I20260926 23:30:10.046202 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:10.047492 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:10.048795 1293 hybrid_clock.cc:648] HybridClock initialized: now 1790465410048665 us; error 119 us; skew 500 ppm
I20260926 23:30:10.049171 1293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:10 dist-test-slave-gwfx krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:10.053030 1293 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:30:10.054920 1293 webserver.cc:460] Webserver started at http://127.0.97.3:36935/ using document root <none> and password file <none>
I20260926 23:30:10.055792 1293 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:10.055912 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:10.056280 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:10.059947 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-2/data/instance:
uuid: "e1409a856d1e4d57810ec5632ecf1dd4"
format_stamp: "Formatted at 2026-09-26 23:30:10 on dist-test-slave-gwfx"
server_key: "b336ce46a4fe815e82753aaa026c8b0a"
server_key_iv: "a62926c423cd8f0eb27d08ea6c220294"
server_key_version: "encryptionkey@0"
I20260926 23:30:10.060894 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e1409a856d1e4d57810ec5632ecf1dd4"
format_stamp: "Formatted at 2026-09-26 23:30:10 on dist-test-slave-gwfx"
server_key: "b336ce46a4fe815e82753aaa026c8b0a"
server_key_iv: "a62926c423cd8f0eb27d08ea6c220294"
server_key_version: "encryptionkey@0"
I20260926 23:30:10.067927 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260926 23:30:10.073244 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:10.075346 1293 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 23:30:10.075619 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "e1409a856d1e4d57810ec5632ecf1dd4"
format_stamp: "Formatted at 2026-09-26 23:30:10 on dist-test-slave-gwfx"
server_key: "b336ce46a4fe815e82753aaa026c8b0a"
server_key_iv: "a62926c423cd8f0eb27d08ea6c220294"
server_key_version: "encryptionkey@0"
I20260926 23:30:10.075886 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:10.098064 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:10.103713 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:10.104096 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:10.105391 1293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:30:10.107358 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:30:10.107462 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:10.107589 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:30:10.107645 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:10.146759 1293 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33091
I20260926 23:30:10.146864 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33091 every 8 connection(s)
I20260926 23:30:10.148290 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:30:10.154873 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 1293
I20260926 23:30:10.155115 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790465400694755-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:30:10.155691 388 external_mini_cluster.cc:1428] Setting key 991ce46c8ed4ab74a85f10802846a120
Sep 26 23:30:10 dist-test-slave-gwfx krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:10.176883 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:10.151842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50003 (local address 127.0.97.62:38685)
0926 23:30:10.152450 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:10.152456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:10.154289 (+ 1833us) server_negotiation.cc:407] Connection header received
0926 23:30:10.157142 (+ 2853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:10.157148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:10.157266 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:10.157374 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:10.159765 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.160629 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.162762 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.163259 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.163613 (+ 354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:10.169047 (+ 5434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:10.169054 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:10.169058 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:10.169105 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:10.172887 (+ 3782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:10.173985 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:10.173990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:10.173993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:10.174153 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:10.174830 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:10.174835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:10.174837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:10.175265 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:10.175404 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:10.176341 (+ 937us) server_negotiation.cc:299] Negotiation successful
0926 23:30:10.176579 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":301,"threads_started":1}
I20260926 23:30:10.178104 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:10.152967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38685 (local address 127.0.97.3:50003)
0926 23:30:10.154032 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:10.154094 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:10.156741 (+ 2647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:10.157773 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:10.157791 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:10.158617 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:10.159502 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:10.159530 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.160834 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.160840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:10.162469 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:10.162490 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.163709 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.163716 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:10.164041 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:10.164824 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:10.164856 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:10.168596 (+ 3740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:10.173091 (+ 4495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:10.173103 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:10.173130 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:10.173797 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:10.174296 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:10.174300 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:10.174303 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:10.174590 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:10.175408 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:10.175416 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:10.175881 (+ 465us) client_negotiation.cc:769] Sending connection context
0926 23:30:10.176165 (+ 284us) client_negotiation.cc:240] Negotiation successful
0926 23:30:10.176703 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":366,"threads_started":1}
I20260926 23:30:10.180482 1421 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38685
I20260926 23:30:10.180881 1421 heartbeater.cc:464] Registering TS with master...
I20260926 23:30:10.181865 1421 heartbeater.cc:511] Master 127.0.97.62:38685 requested a full tablet report, sending...
I20260926 23:30:10.183863 964 ts_manager.cc:194] Registered new tserver with Master: e1409a856d1e4d57810ec5632ecf1dd4 (127.0.97.3:33091)
I20260926 23:30:10.185017 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50003
I20260926 23:30:10.186199 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:30:10.210457 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:10.194448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38498 (local address 127.0.97.62:38685)
0926 23:30:10.194672 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:10.194678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:10.195295 (+ 617us) server_negotiation.cc:407] Connection header received
0926 23:30:10.195474 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:10.195480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:10.195580 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:10.195699 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:10.197345 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.199057 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.200656 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.201134 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.201257 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:10.204145 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:10.204151 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:10.204155 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:10.204204 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:10.207047 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:10.208114 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:10.208119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:10.208122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:10.208275 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:10.208963 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:10.208967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:10.208970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:10.209344 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:10.209564 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:10.209971 (+ 407us) server_negotiation.cc:299] Negotiation successful
0926 23:30:10.210183 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 23:30:10.219887 964 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:30:10.228578 964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:30:10.270664 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:10.245695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60296 (local address 127.0.97.2:41963)
0926 23:30:10.256268 (+ 10573us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:10.256280 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:10.260495 (+ 4215us) server_negotiation.cc:407] Connection header received
0926 23:30:10.260640 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:10.260649 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:10.260943 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:10.261174 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:30:10.263182 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.264385 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.267784 (+ 3399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.269432 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.269580 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:10.269824 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:10.270052 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 23:30:10.270263 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10373,"thread_start_us":8657,"threads_started":1}
I20260926 23:30:10.271188 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:10.246108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44718 (local address 127.0.97.1:42615)
0926 23:30:10.255327 (+ 9219us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:10.255351 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:10.259806 (+ 4455us) server_negotiation.cc:407] Connection header received
0926 23:30:10.260263 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:10.260273 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:10.260604 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:10.260768 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:30:10.262910 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.264779 (+ 1869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.267910 (+ 3131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.269433 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.269580 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:10.269852 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:10.270568 (+ 716us) server_negotiation.cc:299] Negotiation successful
0926 23:30:10.270822 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8946,"thread_start_us":8573,"threads_started":1}
I20260926 23:30:10.272535 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:10.244795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42615 (local address 127.0.0.1:44718)
0926 23:30:10.259613 (+ 14818us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:10.259643 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:10.260007 (+ 364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:10.261036 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:10.261041 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:10.261082 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:30:10.262564 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:10.262589 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.265029 (+ 2440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.265035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:10.267692 (+ 2657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:10.267712 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.270082 (+ 2370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.270091 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:10.270281 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:10.270379 (+ 98us) client_negotiation.cc:769] Sending connection context
0926 23:30:10.271461 (+ 1082us) client_negotiation.cc:240] Negotiation successful
0926 23:30:10.271602 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14561,"mutex_wait_us":45,"spinlock_wait_cycles":11008,"thread_start_us":15013,"threads_started":1}
I20260926 23:30:10.272938 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:10.244710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33091 (local address 127.0.0.1:51524)
0926 23:30:10.257593 (+ 12883us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:10.257643 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:10.257924 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:10.261190 (+ 3266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:10.261198 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:10.261232 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:30:10.262588 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:10.262598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.264952 (+ 2354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.264958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:10.267518 (+ 2560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:10.267541 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.269128 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.269135 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:10.269370 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:10.269428 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 23:30:10.269558 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 23:30:10.269807 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12556,"mutex_wait_us":108,"spinlock_wait_cycles":768,"thread_start_us":14829,"threads_started":1}
I20260926 23:30:10.270763 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:10.245018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51524 (local address 127.0.97.3:33091)
0926 23:30:10.260584 (+ 15566us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:10.260593 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:10.260622 (+ 29us) server_negotiation.cc:407] Connection header received
0926 23:30:10.260706 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:10.260711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:10.260943 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:10.261174 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:30:10.263013 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.264779 (+ 1766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.267741 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.268913 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.269042 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:10.269382 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:10.270053 (+ 671us) server_negotiation.cc:299] Negotiation successful
0926 23:30:10.270237 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15359,"thread_start_us":15157,"threads_started":1}
I20260926 23:30:10.273252 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:10.245226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41963 (local address 127.0.0.1:60296)
0926 23:30:10.260157 (+ 14931us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:10.260181 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:10.260343 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:10.261516 (+ 1173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:10.261522 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:10.261548 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:30:10.262952 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:10.262967 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.264550 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.264556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:10.267518 (+ 2962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:10.267541 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.269651 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.269657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:10.269838 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:10.269870 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 23:30:10.269971 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 23:30:10.270120 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14799,"mutex_wait_us":5,"thread_start_us":11876,"threads_started":1}
I20260926 23:30:10.277483 1088 tablet_service.cc:1467] Processing CreateTablet for tablet 1a5c6592fd704158a8dc6e77a5708022 (DEFAULT_TABLE table=test-table [id=688aab15d3fe43f8a071d155381582af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:30:10.277709 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 1a5c6592fd704158a8dc6e77a5708022 (DEFAULT_TABLE table=test-table [id=688aab15d3fe43f8a071d155381582af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:30:10.277709 1356 tablet_service.cc:1467] Processing CreateTablet for tablet 1a5c6592fd704158a8dc6e77a5708022 (DEFAULT_TABLE table=test-table [id=688aab15d3fe43f8a071d155381582af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:30:10.279578 1088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a5c6592fd704158a8dc6e77a5708022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:10.279610 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a5c6592fd704158a8dc6e77a5708022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:10.279603 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a5c6592fd704158a8dc6e77a5708022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:10.296841 1440 tablet_bootstrap.cc:492] T 1a5c6592fd704158a8dc6e77a5708022 P 2eab0413912640f6a98b09125dee59ed: Bootstrap starting.
I20260926 23:30:10.298643 1441 tablet_bootstrap.cc:492] T 1a5c6592fd704158a8dc6e77a5708022 P e1409a856d1e4d57810ec5632ecf1dd4: Bootstrap starting.
I20260926 23:30:10.300824 1442 tablet_bootstrap.cc:492] T 1a5c6592fd704158a8dc6e77a5708022 P 1b05042674514bbca2c8fa3971d4cd59: Bootstrap starting.
I20260926 23:30:10.305532 1442 tablet_bootstrap.cc:654] T 1a5c6592fd704158a8dc6e77a5708022 P 1b05042674514bbca2c8fa3971d4cd59: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:10.306370 1441 tablet_bootstrap.cc:654] T 1a5c6592fd704158a8dc6e77a5708022 P e1409a856d1e4d57810ec5632ecf1dd4: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:10.307157 1442 log.cc:824] T 1a5c6592fd704158a8dc6e77a5708022 P 1b05042674514bbca2c8fa3971d4cd59: Log is configured to *not* fsync() on all Append() calls
I20260926 23:30:10.307731 1440 tablet_bootstrap.cc:654] T 1a5c6592fd704158a8dc6e77a5708022 P 2eab0413912640f6a98b09125dee59ed: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:10.308128 1441 log.cc:824] T 1a5c6592fd704158a8dc6e77a5708022 P e1409a856d1e4d57810ec5632ecf1dd4: Log is configured to *not* fsync() on all Append() calls
I20260926 23:30:10.310544 1440 log.cc:824] T 1a5c6592fd704158a8dc6e77a5708022 P 2eab0413912640f6a98b09125dee59ed: Log is configured to *not* fsync() on all Append() calls
I20260926 23:30:10.313520 1442 tablet_bootstrap.cc:492] T 1a5c6592fd704158a8dc6e77a5708022 P 1b05042674514bbca2c8fa3971d4cd59: No bootstrap required, opened a new log
I20260926 23:30:10.313987 1442 ts_tablet_manager.cc:1397] T 1a5c6592fd704158a8dc6e77a5708022 P 1b05042674514bbca2c8fa3971d4cd59: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260926 23:30:10.314247 1441 tablet_bootstrap.cc:492] T 1a5c6592fd704158a8dc6e77a5708022 P e1409a856d1e4d57810ec5632ecf1dd4: No bootstrap required, opened a new log
I20260926 23:30:10.314622 1441 ts_tablet_manager.cc:1397] T 1a5c6592fd704158a8dc6e77a5708022 P e1409a856d1e4d57810ec5632ecf1dd4: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.003s
I20260926 23:30:10.314639 1440 tablet_bootstrap.cc:492] T 1a5c6592fd704158a8dc6e77a5708022 P 2eab0413912640f6a98b09125dee59ed: No bootstrap required, opened a new log
I20260926 23:30:10.315052 1440 ts_tablet_manager.cc:1397] T 1a5c6592fd704158a8dc6e77a5708022 P 2eab0413912640f6a98b09125dee59ed: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.004s
I20260926 23:30:10.325641 1441 raft_consensus.cc:348] T 1a5c6592fd704158a8dc6e77a5708022 P e1409a856d1e4d57810ec5632ecf1dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1409a856d1e4d57810ec5632ecf1dd4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33091 } } peers { permanent_uuid: "2eab0413912640f6a98b09125dee59ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42615 } } peers { permanent_uuid: "1b05042674514bbca2c8fa3971d4cd59" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41963 } }
I20260926 23:30:10.326130 1441 raft_consensus.cc:374] T 1a5c6592fd704158a8dc6e77a5708022 P e1409a856d1e4d57810ec5632ecf1dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:10.326233 1441 raft_consensus.cc:729] T 1a5c6592fd704158a8dc6e77a5708022 P e1409a856d1e4d57810ec5632ecf1dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1409a856d1e4d57810ec5632ecf1dd4, State: Initialized, Role: FOLLOWER
I20260926 23:30:10.327006 1441 consensus_queue.cc:260] T 1a5c6592fd704158a8dc6e77a5708022 P e1409a856d1e4d57810ec5632ecf1dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1409a856d1e4d57810ec5632ecf1dd4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33091 } } peers { permanent_uuid: "2eab0413912640f6a98b09125dee59ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42615 } } peers { permanent_uuid: "1b05042674514bbca2c8fa3971d4cd59" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41963 } }
I20260926 23:30:10.327425 1442 raft_consensus.cc:348] T 1a5c6592fd704158a8dc6e77a5708022 P 1b05042674514bbca2c8fa3971d4cd59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1409a856d1e4d57810ec5632ecf1dd4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33091 } } peers { permanent_uuid: "2eab0413912640f6a98b09125dee59ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42615 } } peers { permanent_uuid: "1b05042674514bbca2c8fa3971d4cd59" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41963 } }
I20260926 23:30:10.328114 1442 raft_consensus.cc:374] T 1a5c6592fd704158a8dc6e77a5708022 P 1b05042674514bbca2c8fa3971d4cd59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:10.328233 1442 raft_consensus.cc:729] T 1a5c6592fd704158a8dc6e77a5708022 P 1b05042674514bbca2c8fa3971d4cd59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b05042674514bbca2c8fa3971d4cd59, State: Initialized, Role: FOLLOWER
I20260926 23:30:10.328258 1440 raft_consensus.cc:348] T 1a5c6592fd704158a8dc6e77a5708022 P 2eab0413912640f6a98b09125dee59ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1409a856d1e4d57810ec5632ecf1dd4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33091 } } peers { permanent_uuid: "2eab0413912640f6a98b09125dee59ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42615 } } peers { permanent_uuid: "1b05042674514bbca2c8fa3971d4cd59" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41963 } }
I20260926 23:30:10.328773 1440 raft_consensus.cc:374] T 1a5c6592fd704158a8dc6e77a5708022 P 2eab0413912640f6a98b09125dee59ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:10.328871 1440 raft_consensus.cc:729] T 1a5c6592fd704158a8dc6e77a5708022 P 2eab0413912640f6a98b09125dee59ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eab0413912640f6a98b09125dee59ed, State: Initialized, Role: FOLLOWER
I20260926 23:30:10.329278 1441 ts_tablet_manager.cc:1428] T 1a5c6592fd704158a8dc6e77a5708022 P e1409a856d1e4d57810ec5632ecf1dd4: Time spent starting tablet: real 0.015s user 0.012s sys 0.001s
I20260926 23:30:10.329303 1442 consensus_queue.cc:260] T 1a5c6592fd704158a8dc6e77a5708022 P 1b05042674514bbca2c8fa3971d4cd59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1409a856d1e4d57810ec5632ecf1dd4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33091 } } peers { permanent_uuid: "2eab0413912640f6a98b09125dee59ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42615 } } peers { permanent_uuid: "1b05042674514bbca2c8fa3971d4cd59" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41963 } }
I20260926 23:30:10.329648 1440 consensus_queue.cc:260] T 1a5c6592fd704158a8dc6e77a5708022 P 2eab0413912640f6a98b09125dee59ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1409a856d1e4d57810ec5632ecf1dd4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33091 } } peers { permanent_uuid: "2eab0413912640f6a98b09125dee59ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42615 } } peers { permanent_uuid: "1b05042674514bbca2c8fa3971d4cd59" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41963 } }
I20260926 23:30:10.330050 1421 heartbeater.cc:503] Master 127.0.97.62:38685 was elected leader, sending a full tablet report...
I20260926 23:30:10.333335 1440 ts_tablet_manager.cc:1428] T 1a5c6592fd704158a8dc6e77a5708022 P 2eab0413912640f6a98b09125dee59ed: Time spent starting tablet: real 0.018s user 0.016s sys 0.001s
I20260926 23:30:10.335410 1287 heartbeater.cc:503] Master 127.0.97.62:38685 was elected leader, sending a full tablet report...
I20260926 23:30:10.336501 1442 ts_tablet_manager.cc:1428] T 1a5c6592fd704158a8dc6e77a5708022 P 1b05042674514bbca2c8fa3971d4cd59: Time spent starting tablet: real 0.022s user 0.011s sys 0.013s
I20260926 23:30:10.358217 1448 raft_consensus.cc:482] T 1a5c6592fd704158a8dc6e77a5708022 P 2eab0413912640f6a98b09125dee59ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:30:10.358572 1448 raft_consensus.cc:504] T 1a5c6592fd704158a8dc6e77a5708022 P 2eab0413912640f6a98b09125dee59ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1409a856d1e4d57810ec5632ecf1dd4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33091 } } peers { permanent_uuid: "2eab0413912640f6a98b09125dee59ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42615 } } peers { permanent_uuid: "1b05042674514bbca2c8fa3971d4cd59" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41963 } }
I20260926 23:30:10.360523 1448 leader_election.cc:288] T 1a5c6592fd704158a8dc6e77a5708022 P 2eab0413912640f6a98b09125dee59ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1409a856d1e4d57810ec5632ecf1dd4 (127.0.97.3:33091), 1b05042674514bbca2c8fa3971d4cd59 (127.0.97.2:41963)
I20260926 23:30:10.371206 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:10.361102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60657 (local address 127.0.97.2:41963)
0926 23:30:10.361283 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:10.361288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:10.362139 (+ 851us) server_negotiation.cc:407] Connection header received
0926 23:30:10.362392 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:10.362397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:10.362474 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:10.362601 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:30:10.363447 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.364614 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.368262 (+ 3648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.369613 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.369844 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:10.369937 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:10.370668 (+ 731us) server_negotiation.cc:299] Negotiation successful
0926 23:30:10.370860 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 23:30:10.371206 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:10.361072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34485 (local address 127.0.97.3:33091)
0926 23:30:10.361282 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:10.361288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:10.362140 (+ 852us) server_negotiation.cc:407] Connection header received
0926 23:30:10.362369 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:10.362375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:10.362474 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:10.362601 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:30:10.363436 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.364613 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.368251 (+ 3638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.369640 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.369795 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:10.369937 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:10.370670 (+ 733us) server_negotiation.cc:299] Negotiation successful
0926 23:30:10.370860 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 23:30:10.371716 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:10.360942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33091 (local address 127.0.97.1:34485)
0926 23:30:10.361971 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:10.361994 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:10.362194 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:10.362662 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:10.362666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:10.362724 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:30:10.363240 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:10.363255 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.364787 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.364793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:10.368009 (+ 3216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:10.368030 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.370127 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.370132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:10.370403 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:10.370474 (+ 71us) client_negotiation.cc:769] Sending connection context
0926 23:30:10.370610 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 23:30:10.370828 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"mutex_wait_us":485,"spinlock_wait_cycles":14976,"thread_start_us":497,"threads_started":1}
I20260926 23:30:10.371731 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:10.361092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41963 (local address 127.0.97.1:60657)
0926 23:30:10.361971 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:10.361994 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:10.362194 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:10.362849 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:10.362858 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:10.362885 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:30:10.363295 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:10.363304 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.364788 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.364793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:10.368009 (+ 3216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:10.368030 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.370062 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.370075 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:10.370400 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:10.370477 (+ 77us) client_negotiation.cc:769] Sending connection context
0926 23:30:10.370610 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 23:30:10.370816 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":200,"spinlock_wait_cycles":17536,"thread_start_us":457,"threads_started":1}
I20260926 23:30:10.373557 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5c6592fd704158a8dc6e77a5708022" candidate_uuid: "2eab0413912640f6a98b09125dee59ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b05042674514bbca2c8fa3971d4cd59" is_pre_election: true
I20260926 23:30:10.373557 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5c6592fd704158a8dc6e77a5708022" candidate_uuid: "2eab0413912640f6a98b09125dee59ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1409a856d1e4d57810ec5632ecf1dd4" is_pre_election: true
I20260926 23:30:10.374271 1376 raft_consensus.cc:2454] T 1a5c6592fd704158a8dc6e77a5708022 P e1409a856d1e4d57810ec5632ecf1dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2eab0413912640f6a98b09125dee59ed in term 0.
I20260926 23:30:10.374271 1242 raft_consensus.cc:2454] T 1a5c6592fd704158a8dc6e77a5708022 P 1b05042674514bbca2c8fa3971d4cd59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2eab0413912640f6a98b09125dee59ed in term 0.
I20260926 23:30:10.375213 1042 leader_election.cc:302] T 1a5c6592fd704158a8dc6e77a5708022 P 2eab0413912640f6a98b09125dee59ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b05042674514bbca2c8fa3971d4cd59, 2eab0413912640f6a98b09125dee59ed; no voters:
I20260926 23:30:10.375852 1448 raft_consensus.cc:2781] T 1a5c6592fd704158a8dc6e77a5708022 P 2eab0413912640f6a98b09125dee59ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:30:10.376103 1448 raft_consensus.cc:482] T 1a5c6592fd704158a8dc6e77a5708022 P 2eab0413912640f6a98b09125dee59ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:30:10.376202 1448 raft_consensus.cc:3037] T 1a5c6592fd704158a8dc6e77a5708022 P 2eab0413912640f6a98b09125dee59ed [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:10.379001 1448 raft_consensus.cc:504] T 1a5c6592fd704158a8dc6e77a5708022 P 2eab0413912640f6a98b09125dee59ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1409a856d1e4d57810ec5632ecf1dd4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33091 } } peers { permanent_uuid: "2eab0413912640f6a98b09125dee59ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42615 } } peers { permanent_uuid: "1b05042674514bbca2c8fa3971d4cd59" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41963 } }
I20260926 23:30:10.379743 1448 leader_election.cc:288] T 1a5c6592fd704158a8dc6e77a5708022 P 2eab0413912640f6a98b09125dee59ed [CANDIDATE]: Term 1 election: Requested vote from peers e1409a856d1e4d57810ec5632ecf1dd4 (127.0.97.3:33091), 1b05042674514bbca2c8fa3971d4cd59 (127.0.97.2:41963)
I20260926 23:30:10.380602 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5c6592fd704158a8dc6e77a5708022" candidate_uuid: "2eab0413912640f6a98b09125dee59ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b05042674514bbca2c8fa3971d4cd59"
I20260926 23:30:10.380702 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5c6592fd704158a8dc6e77a5708022" candidate_uuid: "2eab0413912640f6a98b09125dee59ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1409a856d1e4d57810ec5632ecf1dd4"
I20260926 23:30:10.380880 1242 raft_consensus.cc:3037] T 1a5c6592fd704158a8dc6e77a5708022 P 1b05042674514bbca2c8fa3971d4cd59 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:10.381048 1376 raft_consensus.cc:3037] T 1a5c6592fd704158a8dc6e77a5708022 P e1409a856d1e4d57810ec5632ecf1dd4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:10.384032 1242 raft_consensus.cc:2454] T 1a5c6592fd704158a8dc6e77a5708022 P 1b05042674514bbca2c8fa3971d4cd59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2eab0413912640f6a98b09125dee59ed in term 1.
I20260926 23:30:10.384119 1376 raft_consensus.cc:2454] T 1a5c6592fd704158a8dc6e77a5708022 P e1409a856d1e4d57810ec5632ecf1dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2eab0413912640f6a98b09125dee59ed in term 1.
I20260926 23:30:10.384689 1042 leader_election.cc:302] T 1a5c6592fd704158a8dc6e77a5708022 P 2eab0413912640f6a98b09125dee59ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1b05042674514bbca2c8fa3971d4cd59, 2eab0413912640f6a98b09125dee59ed, e1409a856d1e4d57810ec5632ecf1dd4; no voters:
I20260926 23:30:10.385217 1448 raft_consensus.cc:2781] T 1a5c6592fd704158a8dc6e77a5708022 P 2eab0413912640f6a98b09125dee59ed [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:30:10.385766 1448 raft_consensus.cc:686] T 1a5c6592fd704158a8dc6e77a5708022 P 2eab0413912640f6a98b09125dee59ed [term 1 LEADER]: Becoming Leader. State: Replica: 2eab0413912640f6a98b09125dee59ed, State: Running, Role: LEADER
I20260926 23:30:10.386304 1448 consensus_queue.cc:237] T 1a5c6592fd704158a8dc6e77a5708022 P 2eab0413912640f6a98b09125dee59ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1409a856d1e4d57810ec5632ecf1dd4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33091 } } peers { permanent_uuid: "2eab0413912640f6a98b09125dee59ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42615 } } peers { permanent_uuid: "1b05042674514bbca2c8fa3971d4cd59" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41963 } }
I20260926 23:30:10.391770 962 catalog_manager.cc:5495] T 1a5c6592fd704158a8dc6e77a5708022 P 2eab0413912640f6a98b09125dee59ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 2eab0413912640f6a98b09125dee59ed (127.0.97.1). New cstate: current_term: 1 leader_uuid: "2eab0413912640f6a98b09125dee59ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1409a856d1e4d57810ec5632ecf1dd4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2eab0413912640f6a98b09125dee59ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b05042674514bbca2c8fa3971d4cd59" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41963 } health_report { overall_health: UNKNOWN } } }
W20260926 23:30:10.400197 1422 tablet.cc:2369] T 1a5c6592fd704158a8dc6e77a5708022 P e1409a856d1e4d57810ec5632ecf1dd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:30:10.435956 1288 tablet.cc:2369] T 1a5c6592fd704158a8dc6e77a5708022 P 1b05042674514bbca2c8fa3971d4cd59: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 23:30:10 dist-test-slave-gwfx krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:30:10.445972 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:10.425680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44734 (local address 127.0.97.1:42615)
0926 23:30:10.425971 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:10.425980 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:10.426514 (+ 534us) server_negotiation.cc:407] Connection header received
0926 23:30:10.426841 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:10.426846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:10.426939 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:10.427053 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:10.429544 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.430460 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.432015 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.432496 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.432632 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:10.438144 (+ 5512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:10.438170 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:10.438191 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:10.438246 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:10.441825 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:10.442804 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:10.442813 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:10.442816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:10.442987 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:10.443421 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:10.443425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:10.443428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:10.443855 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:10.444193 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:10.444524 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 23:30:10.444803 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
Sep 26 23:30:10 dist-test-slave-gwfx krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465410, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 23:30:10 dist-test-slave-gwfx krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465410, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:30:10.505107 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:10.484136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44750 (local address 127.0.97.1:42615)
0926 23:30:10.484460 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:10.484468 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:10.484942 (+ 474us) server_negotiation.cc:407] Connection header received
0926 23:30:10.485191 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:10.485196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:10.485284 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:10.485405 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:10.487691 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.488559 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.490310 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.490963 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.491105 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:10.497361 (+ 6256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:10.497372 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:10.497378 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:10.497447 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:10.501508 (+ 4061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:10.502594 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:10.502601 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:10.502604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:10.502926 (+ 322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:10.503528 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:10.503533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:10.503536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:10.503893 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:10.504188 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:10.504484 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 23:30:10.504736 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
Sep 26 23:30:10 dist-test-slave-gwfx krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 23:30:10 dist-test-slave-gwfx krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:30:10.559684 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:10.542210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44760 (local address 127.0.97.1:42615)
0926 23:30:10.542499 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:10.542523 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:10.543145 (+ 622us) server_negotiation.cc:407] Connection header received
0926 23:30:10.543363 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:10.543367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:10.543461 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:10.543551 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:10.545531 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.546305 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.547684 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.548108 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.548227 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:10.552959 (+ 4732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:10.552966 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:10.552969 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:10.553015 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:10.556805 (+ 3790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:10.557631 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:10.557636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:10.557639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:10.557783 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:10.558166 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:10.558170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:10.558172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:10.558463 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:10.558833 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:10.559062 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 23:30:10.559322 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
W20260926 23:30:10.561893 1068 tablet_service.cc:2908] Rejecting scan request for tablet 1a5c6592fd704158a8dc6e77a5708022: Uninitialized: clean time has not yet been initialized
I20260926 23:30:10.566277 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 1025
W20260926 23:30:10.581267 1154 tablet.cc:2369] T 1a5c6592fd704158a8dc6e77a5708022 P 2eab0413912640f6a98b09125dee59ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:30:10.583560 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:10.567356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44768 (local address 127.0.97.1:42615)
0926 23:30:10.567534 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:10.567539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:10.567933 (+ 394us) server_negotiation.cc:407] Connection header received
0926 23:30:10.568219 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:10.568224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:10.568299 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:10.568411 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:10.569931 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.570904 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.572396 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.572896 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.573055 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:10.576371 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:10.576378 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:10.576382 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:10.576427 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:10.579260 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:10.580461 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:10.580468 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:10.580472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:10.580685 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:10.581309 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:10.581315 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:10.581319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:10.581970 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:10.582330 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:10.582980 (+ 650us) server_negotiation.cc:299] Negotiation successful
0926 23:30:10.583265 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 23:30:10.584420 1148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:30:10.688942 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 1025
I20260926 23:30:10.706022 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 1159
Sep 26 23:30:10 dist-test-slave-gwfx krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:30:10.724045 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:10.706959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60300 (local address 127.0.97.2:41963)
0926 23:30:10.707176 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:10.707181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:10.707205 (+ 24us) server_negotiation.cc:407] Connection header received
0926 23:30:10.707372 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:10.707377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:10.707455 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:10.707581 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:10.709092 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.710017 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.711987 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.712484 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.712618 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:10.717344 (+ 4726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:10.717366 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:10.717387 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:10.717448 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:10.720814 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:10.721697 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:10.721706 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:10.721709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:10.721857 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:10.722266 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:10.722270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:10.722273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:10.722795 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:10.723132 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:10.723431 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 23:30:10.723679 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 23:30:10.724951 1282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:30:10.820618 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 1159
I20260926 23:30:10.838312 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 1293
Sep 26 23:30:10 dist-test-slave-gwfx krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:30:10.857076 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:10.839305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51536 (local address 127.0.97.3:33091)
0926 23:30:10.839516 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:10.839521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:10.839545 (+ 24us) server_negotiation.cc:407] Connection header received
0926 23:30:10.839622 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:10.839627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:10.839707 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:10.839857 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:10.841477 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.842265 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.844192 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:10.844655 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:10.844788 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:10.849931 (+ 5143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:10.849951 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:10.849974 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:10.850033 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:10.853581 (+ 3548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:10.854576 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:10.854587 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:10.854590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:10.854830 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:10.855265 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:10.855277 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:10.855280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:10.855743 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:10.856088 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:10.856401 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 23:30:10.856670 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 23:30:10.857940 1416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:30:10.954088 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 1293
I20260926 23:30:10.970121 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 932
I20260926 23:30:10.971486 994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:30:11.098469 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 932
2026-09-26T23:30:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4849 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:30:11 dist-test-slave-gwfx krb5kdc[1470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:30:11 dist-test-slave-gwfx krb5kdc[1470](info): set up 2 sockets
Sep 26 23:30:11 dist-test-slave-gwfx krb5kdc[1470](info): commencing operation
krb5kdc: starting...
W20260926 23:30:12.147086 388 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:30:12 dist-test-slave-gwfx krb5kdc[1470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:30:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:30:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:30:12.351595 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42647
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:30:12.732489 1483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:12.732935 1483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:12.733074 1483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:12.741312 1483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:30:12.741489 1483 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 23:30:12.741554 1483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:12.741649 1483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:12.741719 1483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:30:12.741814 1483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:30:12.753276 1483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42647
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:12.755045 1483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:12.756700 1483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:12.770404 1488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:12.770432 1489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:12.770419 1491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:12.771211 1483 server_base.cc:1034] running on GCE node
I20260926 23:30:12.772195 1483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:12.774048 1483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:12.775352 1483 hybrid_clock.cc:648] HybridClock initialized: now 1790465412775267 us; error 46 us; skew 500 ppm
I20260926 23:30:12.775864 1483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:12 dist-test-slave-gwfx krb5kdc[1470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:12.779742 1483 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:30:12.781844 1483 webserver.cc:460] Webserver started at http://127.0.97.62:46209/ using document root <none> and password file <none>
I20260926 23:30:12.782820 1483 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:12.782948 1483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:12.783332 1483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:12.787129 1483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "beae4f80ff99489785e3c26ed7c186cb"
format_stamp: "Formatted at 2026-09-26 23:30:12 on dist-test-slave-gwfx"
server_key: "2ad04809bcad65ec85fe25bc71efb9d6"
server_key_iv: "e13af741161cda965e656c2cc87c9f2b"
server_key_version: "encryptionkey@0"
I20260926 23:30:12.788151 1483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "beae4f80ff99489785e3c26ed7c186cb"
format_stamp: "Formatted at 2026-09-26 23:30:12 on dist-test-slave-gwfx"
server_key: "2ad04809bcad65ec85fe25bc71efb9d6"
server_key_iv: "e13af741161cda965e656c2cc87c9f2b"
server_key_version: "encryptionkey@0"
I20260926 23:30:12.795260 1483 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260926 23:30:12.800427 1498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:12.802402 1483 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 23:30:12.802675 1483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "beae4f80ff99489785e3c26ed7c186cb"
format_stamp: "Formatted at 2026-09-26 23:30:12 on dist-test-slave-gwfx"
server_key: "2ad04809bcad65ec85fe25bc71efb9d6"
server_key_iv: "e13af741161cda965e656c2cc87c9f2b"
server_key_version: "encryptionkey@0"
I20260926 23:30:12.803038 1483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:12.832716 1483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:12.838963 1483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:12.839299 1483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:12.866144 1483 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42647
I20260926 23:30:12.866184 1559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42647 every 8 connection(s)
I20260926 23:30:12.867983 1483 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:30:12.872514 1560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:12.875011 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 1483
I20260926 23:30:12.875255 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:30:12.876103 388 external_mini_cluster.cc:1428] Setting key 00fa622396874fc6afd40f965bc593fc
I20260926 23:30:12.886941 1560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beae4f80ff99489785e3c26ed7c186cb: Bootstrap starting.
I20260926 23:30:12.891103 1560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P beae4f80ff99489785e3c26ed7c186cb: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:12.892558 1560 log.cc:824] T 00000000000000000000000000000000 P beae4f80ff99489785e3c26ed7c186cb: Log is configured to *not* fsync() on all Append() calls
Sep 26 23:30:12 dist-test-slave-gwfx krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:12.898214 1560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beae4f80ff99489785e3c26ed7c186cb: No bootstrap required, opened a new log
I20260926 23:30:12.906406 1563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:12.878951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43956 (local address 127.0.97.62:42647)
0926 23:30:12.880092 (+ 1141us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:12.880108 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:12.880170 (+ 62us) server_negotiation.cc:407] Connection header received
0926 23:30:12.882246 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:12.882279 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:12.882921 (+ 642us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:12.883398 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:12.885324 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:12.887060 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:12.889029 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:12.890078 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:12.890338 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:12.896497 (+ 6159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:12.896509 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:12.896535 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:12.896591 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:12.900499 (+ 3908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:12.901591 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:12.901602 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:12.901613 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:12.901889 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:12.902359 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:12.902364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:12.902366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:12.903261 (+ 895us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:12.903649 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:12.904205 (+ 556us) server_negotiation.cc:299] Negotiation successful
0926 23:30:12.904865 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":825,"thread_start_us":534,"threads_started":1}
I20260926 23:30:12.907361 1560 raft_consensus.cc:348] T 00000000000000000000000000000000 P beae4f80ff99489785e3c26ed7c186cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beae4f80ff99489785e3c26ed7c186cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42647 } }
I20260926 23:30:12.907878 1560 raft_consensus.cc:374] T 00000000000000000000000000000000 P beae4f80ff99489785e3c26ed7c186cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:12.907992 1560 raft_consensus.cc:729] T 00000000000000000000000000000000 P beae4f80ff99489785e3c26ed7c186cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beae4f80ff99489785e3c26ed7c186cb, State: Initialized, Role: FOLLOWER
I20260926 23:30:12.908849 1560 consensus_queue.cc:260] T 00000000000000000000000000000000 P beae4f80ff99489785e3c26ed7c186cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beae4f80ff99489785e3c26ed7c186cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42647 } }
I20260926 23:30:12.909125 1560 raft_consensus.cc:388] T 00000000000000000000000000000000 P beae4f80ff99489785e3c26ed7c186cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:30:12.909286 1560 raft_consensus.cc:482] T 00000000000000000000000000000000 P beae4f80ff99489785e3c26ed7c186cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:30:12.909643 1560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P beae4f80ff99489785e3c26ed7c186cb [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:12.912792 1560 raft_consensus.cc:504] T 00000000000000000000000000000000 P beae4f80ff99489785e3c26ed7c186cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beae4f80ff99489785e3c26ed7c186cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42647 } }
I20260926 23:30:12.913419 1560 leader_election.cc:302] T 00000000000000000000000000000000 P beae4f80ff99489785e3c26ed7c186cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: beae4f80ff99489785e3c26ed7c186cb; no voters:
I20260926 23:30:12.914103 1560 leader_election.cc:288] T 00000000000000000000000000000000 P beae4f80ff99489785e3c26ed7c186cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:30:12.914340 1565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P beae4f80ff99489785e3c26ed7c186cb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:30:12.915431 1565 raft_consensus.cc:686] T 00000000000000000000000000000000 P beae4f80ff99489785e3c26ed7c186cb [term 1 LEADER]: Becoming Leader. State: Replica: beae4f80ff99489785e3c26ed7c186cb, State: Running, Role: LEADER
I20260926 23:30:12.916275 1565 consensus_queue.cc:237] T 00000000000000000000000000000000 P beae4f80ff99489785e3c26ed7c186cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beae4f80ff99489785e3c26ed7c186cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42647 } }
I20260926 23:30:12.916642 1560 sys_catalog.cc:564] T 00000000000000000000000000000000 P beae4f80ff99489785e3c26ed7c186cb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:30:12.920565 1566 sys_catalog.cc:455] T 00000000000000000000000000000000 P beae4f80ff99489785e3c26ed7c186cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "beae4f80ff99489785e3c26ed7c186cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beae4f80ff99489785e3c26ed7c186cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42647 } } }
I20260926 23:30:12.920490 1567 sys_catalog.cc:455] T 00000000000000000000000000000000 P beae4f80ff99489785e3c26ed7c186cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader beae4f80ff99489785e3c26ed7c186cb. Latest consensus state: current_term: 1 leader_uuid: "beae4f80ff99489785e3c26ed7c186cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beae4f80ff99489785e3c26ed7c186cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42647 } } }
I20260926 23:30:12.921123 1566 sys_catalog.cc:458] T 00000000000000000000000000000000 P beae4f80ff99489785e3c26ed7c186cb [sys.catalog]: This master's current role is: LEADER
I20260926 23:30:12.921140 1567 sys_catalog.cc:458] T 00000000000000000000000000000000 P beae4f80ff99489785e3c26ed7c186cb [sys.catalog]: This master's current role is: LEADER
I20260926 23:30:12.922399 1573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:30:12.930029 1573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:30:12.945011 1573 catalog_manager.cc:1347] Generated new cluster ID: 6cd3851f26ff4cfa995aa52e33de04d7
I20260926 23:30:12.945155 1573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:30:12.973569 1573 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:30:12.975543 1573 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:30:12.990499 1573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P beae4f80ff99489785e3c26ed7c186cb: Generated new TSK 0
I20260926 23:30:12.991874 1573 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:30:13.060389 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42647
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:30:13.445951 1586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:13.446259 1586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:13.446357 1586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:13.453979 1586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:13.454124 1586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:13.454214 1586 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 23:30:13.454303 1586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:30:13.464605 1586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:13.466257 1586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:13.467870 1586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:13.483345 1594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:13.483337 1591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:13.483379 1592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:13.485532 1586 server_base.cc:1034] running on GCE node
I20260926 23:30:13.486310 1586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:13.487524 1586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:13.488813 1586 hybrid_clock.cc:648] HybridClock initialized: now 1790465413488740 us; error 53 us; skew 500 ppm
I20260926 23:30:13.489217 1586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:13 dist-test-slave-gwfx krb5kdc[1470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:13.493600 1586 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:30:13.495759 1586 webserver.cc:460] Webserver started at http://127.0.97.1:40047/ using document root <none> and password file <none>
I20260926 23:30:13.496698 1586 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:13.496821 1586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:13.497210 1586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:13.501093 1586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-0/data/instance:
uuid: "762530e791834c2cb07652dbde301eb1"
format_stamp: "Formatted at 2026-09-26 23:30:13 on dist-test-slave-gwfx"
server_key: "b6b9afeb0c1e753d0573008e0fb21ed9"
server_key_iv: "7ca8babf71f64fe43e3129f7b361d3f1"
server_key_version: "encryptionkey@0"
I20260926 23:30:13.502130 1586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-0/wal/instance:
uuid: "762530e791834c2cb07652dbde301eb1"
format_stamp: "Formatted at 2026-09-26 23:30:13 on dist-test-slave-gwfx"
server_key: "b6b9afeb0c1e753d0573008e0fb21ed9"
server_key_iv: "7ca8babf71f64fe43e3129f7b361d3f1"
server_key_version: "encryptionkey@0"
I20260926 23:30:13.509140 1586 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260926 23:30:13.513830 1601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:13.515902 1586 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:30:13.516139 1586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "762530e791834c2cb07652dbde301eb1"
format_stamp: "Formatted at 2026-09-26 23:30:13 on dist-test-slave-gwfx"
server_key: "b6b9afeb0c1e753d0573008e0fb21ed9"
server_key_iv: "7ca8babf71f64fe43e3129f7b361d3f1"
server_key_version: "encryptionkey@0"
I20260926 23:30:13.516417 1586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:13.536320 1586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:13.540989 1586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:13.541266 1586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:13.544440 1586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:30:13.544553 1586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:13.544687 1586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:30:13.544746 1586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 23:30:13 dist-test-slave-gwfx krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:13.575112 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:13.545548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56423 (local address 127.0.97.62:42647)
0926 23:30:13.546121 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:13.546126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:13.547632 (+ 1506us) server_negotiation.cc:407] Connection header received
0926 23:30:13.551043 (+ 3411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:13.551052 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:13.551177 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:13.551349 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:13.554349 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:13.555368 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:13.559141 (+ 3773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:13.559865 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:13.560056 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:13.565715 (+ 5659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:13.565722 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:13.565726 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:13.565771 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:13.570045 (+ 4274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:13.571510 (+ 1465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:13.571515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:13.571518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:13.571715 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:13.572383 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:13.572389 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:13.572392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:13.572770 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:13.572948 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:13.574374 (+ 1426us) server_negotiation.cc:299] Negotiation successful
0926 23:30:13.574679 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":348,"threads_started":1}
I20260926 23:30:13.576937 1611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:13.546236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42647 (local address 127.0.97.1:56423)
0926 23:30:13.547405 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:13.547476 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:13.550424 (+ 2948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:13.551781 (+ 1357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:13.551793 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:13.552896 (+ 1103us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:13.554063 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:13.554098 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:13.555668 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:13.555676 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:13.558813 (+ 3137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:13.558853 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:13.560249 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:13.560259 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:13.560543 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:13.561420 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:13.561450 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:13.565326 (+ 3876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:13.570307 (+ 4981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:13.570326 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:13.570359 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:13.571251 (+ 892us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:13.571875 (+ 624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:13.571881 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:13.571889 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:13.572215 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:13.573052 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:13.573064 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:13.573738 (+ 674us) client_negotiation.cc:769] Sending connection context
0926 23:30:13.574145 (+ 407us) client_negotiation.cc:240] Negotiation successful
0926 23:30:13.574950 (+ 805us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"thread_start_us":512,"threads_started":1}
I20260926 23:30:13.597390 1586 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45241
I20260926 23:30:13.597406 1721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45241 every 8 connection(s)
I20260926 23:30:13.599386 1586 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:30:13.602538 1722 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42647
I20260926 23:30:13.602739 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 1586
I20260926 23:30:13.602890 1722 heartbeater.cc:464] Registering TS with master...
I20260926 23:30:13.602970 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:30:13.603853 388 external_mini_cluster.cc:1428] Setting key 9c9385c126345f172f592aa4259834f3
I20260926 23:30:13.603892 1722 heartbeater.cc:511] Master 127.0.97.62:42647 requested a full tablet report, sending...
I20260926 23:30:13.603997 1711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:13.585634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42647 (local address 127.0.97.1:39421)
0926 23:30:13.586505 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:13.586534 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:13.586791 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:13.587325 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:13.587330 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:13.588115 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:13.588625 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:13.588685 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:13.590009 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:13.590018 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:13.591999 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:13.592024 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:13.593109 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:13.593118 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:13.593348 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:13.594288 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:13.594314 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:13.595640 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:13.599811 (+ 4171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:13.599819 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:13.599825 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:13.600684 (+ 859us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:13.601314 (+ 630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:13.601321 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:13.601325 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:13.601566 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:13.602305 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:13.602312 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:13.602618 (+ 306us) client_negotiation.cc:769] Sending connection context
0926 23:30:13.602812 (+ 194us) client_negotiation.cc:240] Negotiation successful
0926 23:30:13.603279 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":452,"threads_started":1}
I20260926 23:30:13.605086 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:13.585798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39421 (local address 127.0.97.62:42647)
0926 23:30:13.586097 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:13.586106 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:13.586738 (+ 632us) server_negotiation.cc:407] Connection header received
0926 23:30:13.587034 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:13.587039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:13.587126 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:13.587248 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:13.588965 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:13.589780 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:13.592259 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:13.592791 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:13.592933 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:13.595932 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:13.595938 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:13.595943 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:13.595990 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:13.599488 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:13.600945 (+ 1457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:13.600952 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:13.600956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:13.601151 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:13.601709 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:13.601714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:13.601717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:13.602137 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:13.602291 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:13.604420 (+ 2129us) server_negotiation.cc:299] Negotiation successful
0926 23:30:13.604722 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 23:30:13.606729 1514 ts_manager.cc:194] Registered new tserver with Master: 762530e791834c2cb07652dbde301eb1 (127.0.97.1:45241)
I20260926 23:30:13.608651 1514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56423
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:30:13.651083 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42647
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:30:14.064074 1726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:14.064494 1726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:14.064630 1726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:14.074431 1726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:14.074621 1726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:14.074816 1726 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 23:30:14.074930 1726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:30:14.086792 1726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:14.088546 1726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:14.090221 1726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:14.108072 1732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:14.108474 1731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:14.108532 1734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:14.109570 1726 server_base.cc:1034] running on GCE node
I20260926 23:30:14.110616 1726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:14.112357 1726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:14.113737 1726 hybrid_clock.cc:648] HybridClock initialized: now 1790465414113607 us; error 81 us; skew 500 ppm
I20260926 23:30:14.114239 1726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:14 dist-test-slave-gwfx krb5kdc[1470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:14.118676 1726 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:30:14.121028 1726 webserver.cc:460] Webserver started at http://127.0.97.2:38403/ using document root <none> and password file <none>
I20260926 23:30:14.122212 1726 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:14.122378 1726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:14.122855 1726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:14.127460 1726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-1/data/instance:
uuid: "c7a26f6aa4ca4af587543e3d7ad5670f"
format_stamp: "Formatted at 2026-09-26 23:30:14 on dist-test-slave-gwfx"
server_key: "850d26a03e396a3b657f502f5b8fb437"
server_key_iv: "af9ad6c77e075e70e5f8bd8726efc5ff"
server_key_version: "encryptionkey@0"
I20260926 23:30:14.128693 1726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c7a26f6aa4ca4af587543e3d7ad5670f"
format_stamp: "Formatted at 2026-09-26 23:30:14 on dist-test-slave-gwfx"
server_key: "850d26a03e396a3b657f502f5b8fb437"
server_key_iv: "af9ad6c77e075e70e5f8bd8726efc5ff"
server_key_version: "encryptionkey@0"
I20260926 23:30:14.138413 1726 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.003s
I20260926 23:30:14.144403 1741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:14.146934 1726 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260926 23:30:14.147197 1726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "c7a26f6aa4ca4af587543e3d7ad5670f"
format_stamp: "Formatted at 2026-09-26 23:30:14 on dist-test-slave-gwfx"
server_key: "850d26a03e396a3b657f502f5b8fb437"
server_key_iv: "af9ad6c77e075e70e5f8bd8726efc5ff"
server_key_version: "encryptionkey@0"
I20260926 23:30:14.147501 1726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:14.164165 1726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:14.169914 1726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:14.170276 1726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:14.174230 1726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:30:14.174367 1726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:14.174510 1726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:30:14.174576 1726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 23:30:14 dist-test-slave-gwfx krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:14.209503 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:14.175174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50009 (local address 127.0.97.62:42647)
0926 23:30:14.175731 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:14.175738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:14.177753 (+ 2015us) server_negotiation.cc:407] Connection header received
0926 23:30:14.182479 (+ 4726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:14.182488 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:14.182733 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:14.183046 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:14.186497 (+ 3451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:14.187960 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:14.192940 (+ 4980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:14.193949 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:14.194198 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:14.200163 (+ 5965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:14.200169 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:14.200173 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:14.200216 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:14.203862 (+ 3646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:14.205802 (+ 1940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:14.205809 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:14.205813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:14.206130 (+ 317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:14.206776 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:14.206782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:14.206786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:14.207116 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:14.207258 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:14.208619 (+ 1361us) server_negotiation.cc:299] Negotiation successful
0926 23:30:14.209037 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":260,"threads_started":1}
I20260926 23:30:14.211311 1751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:14.176144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42647 (local address 127.0.97.2:50009)
0926 23:30:14.177393 (+ 1249us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:14.177483 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:14.181735 (+ 4252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:14.183646 (+ 1911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:14.183664 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:14.184694 (+ 1030us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:14.186196 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:14.186237 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:14.188315 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:14.188324 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:14.192394 (+ 4070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:14.192448 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:14.194751 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:14.194763 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:14.195108 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:14.196114 (+ 1006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:14.196143 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:14.199816 (+ 3673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:14.204239 (+ 4423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:14.204258 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:14.204298 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:14.205473 (+ 1175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:14.206325 (+ 852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:14.206330 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:14.206333 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:14.206617 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:14.207274 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:14.207284 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:14.207743 (+ 459us) client_negotiation.cc:769] Sending connection context
0926 23:30:14.208325 (+ 582us) client_negotiation.cc:240] Negotiation successful
0926 23:30:14.209162 (+ 837us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"thread_start_us":471,"threads_started":1}
I20260926 23:30:14.249408 1726 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41071
I20260926 23:30:14.249497 1861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41071 every 8 connection(s)
I20260926 23:30:14.251226 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:14.222166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47101 (local address 127.0.97.62:42647)
0926 23:30:14.222380 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:14.222386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:14.228485 (+ 6099us) server_negotiation.cc:407] Connection header received
0926 23:30:14.228958 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:14.228965 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:14.229079 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:14.229233 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:14.231276 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:14.232196 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:14.236472 (+ 4276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:14.237248 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:14.237421 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:14.241301 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:14.241312 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:14.241320 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:14.241489 (+ 169us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:14.246813 (+ 5324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:14.247889 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:14.247896 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:14.247899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:14.248094 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:14.248995 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:14.249003 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:14.249007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:14.249751 (+ 744us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:14.249957 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:14.250614 (+ 657us) server_negotiation.cc:299] Negotiation successful
0926 23:30:14.250939 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 23:30:14.251441 1726 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:30:14.251528 1813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:14.222225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42647 (local address 127.0.97.2:47101)
0926 23:30:14.228228 (+ 6003us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:14.228283 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:14.228694 (+ 411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:14.229435 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:14.229440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:14.230295 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:14.231054 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:14.231071 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:14.232403 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:14.232410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:14.236052 (+ 3642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:14.236093 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:14.237596 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:14.237619 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:14.237866 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:14.239056 (+ 1190us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:14.239089 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:14.240754 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:14.247069 (+ 6315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:14.247075 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:14.247078 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:14.247635 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:14.248311 (+ 676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:14.248317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:14.248321 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:14.248696 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:14.249959 (+ 1263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:14.249964 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:14.250260 (+ 296us) client_negotiation.cc:769] Sending connection context
0926 23:30:14.250445 (+ 185us) client_negotiation.cc:240] Negotiation successful
0926 23:30:14.250920 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5721,"thread_start_us":5004,"threads_started":1}
I20260926 23:30:14.256038 1862 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42647
I20260926 23:30:14.256346 1862 heartbeater.cc:464] Registering TS with master...
I20260926 23:30:14.257056 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 1726
I20260926 23:30:14.257275 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:30:14.257546 1862 heartbeater.cc:511] Master 127.0.97.62:42647 requested a full tablet report, sending...
I20260926 23:30:14.257892 388 external_mini_cluster.cc:1428] Setting key af270c8a141340114f557a0571a59e1d
I20260926 23:30:14.259624 1514 ts_manager.cc:194] Registered new tserver with Master: c7a26f6aa4ca4af587543e3d7ad5670f (127.0.97.2:41071)
I20260926 23:30:14.260563 1514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50009
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:30:14.328207 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42647
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:30:14.613116 1722 heartbeater.cc:503] Master 127.0.97.62:42647 was elected leader, sending a full tablet report...
W20260926 23:30:14.744747 1866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:14.745136 1866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:14.745247 1866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:14.753688 1866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:14.753841 1866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:14.753945 1866 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 23:30:14.754038 1866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:30:14.765617 1866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:14.767402 1866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:14.769068 1866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:14.785003 1871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:14.785055 1872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:14.785162 1874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:14.786135 1866 server_base.cc:1034] running on GCE node
I20260926 23:30:14.787016 1866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:14.788225 1866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:14.789472 1866 hybrid_clock.cc:648] HybridClock initialized: now 1790465414789390 us; error 56 us; skew 500 ppm
I20260926 23:30:14.789841 1866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:14 dist-test-slave-gwfx krb5kdc[1470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:14.793506 1866 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:30:14.795531 1866 webserver.cc:460] Webserver started at http://127.0.97.3:44435/ using document root <none> and password file <none>
I20260926 23:30:14.796413 1866 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:14.796517 1866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:14.796840 1866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:14.800354 1866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-2/data/instance:
uuid: "596576a3490444db809867705df4339d"
format_stamp: "Formatted at 2026-09-26 23:30:14 on dist-test-slave-gwfx"
server_key: "842681c8eff651817938b4ba22f4f596"
server_key_iv: "632906a0c1671962ab224bb29af44e23"
server_key_version: "encryptionkey@0"
I20260926 23:30:14.801254 1866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-2/wal/instance:
uuid: "596576a3490444db809867705df4339d"
format_stamp: "Formatted at 2026-09-26 23:30:14 on dist-test-slave-gwfx"
server_key: "842681c8eff651817938b4ba22f4f596"
server_key_iv: "632906a0c1671962ab224bb29af44e23"
server_key_version: "encryptionkey@0"
I20260926 23:30:14.807994 1866 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 23:30:14.812944 1881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:14.815135 1866 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260926 23:30:14.815382 1866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "596576a3490444db809867705df4339d"
format_stamp: "Formatted at 2026-09-26 23:30:14 on dist-test-slave-gwfx"
server_key: "842681c8eff651817938b4ba22f4f596"
server_key_iv: "632906a0c1671962ab224bb29af44e23"
server_key_version: "encryptionkey@0"
I20260926 23:30:14.815683 1866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:14.833082 1866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:14.838236 1866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:14.838557 1866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:14.841846 1866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:30:14.841969 1866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:14.842116 1866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:30:14.842187 1866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 23:30:14 dist-test-slave-gwfx krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:14.871469 1890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:14.842802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60139 (local address 127.0.97.62:42647)
0926 23:30:14.843271 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:14.843278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:14.844884 (+ 1606us) server_negotiation.cc:407] Connection header received
0926 23:30:14.847561 (+ 2677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:14.847566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:14.847654 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:14.847757 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:14.851090 (+ 3333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:14.852146 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:14.854446 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:14.855122 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:14.855283 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:14.861260 (+ 5977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:14.861269 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:14.861276 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:14.861327 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:14.865530 (+ 4203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:14.867070 (+ 1540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:14.867077 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:14.867081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:14.867303 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:14.868111 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:14.868116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:14.868119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:14.868580 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:14.868750 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:14.870748 (+ 1998us) server_negotiation.cc:299] Negotiation successful
0926 23:30:14.871016 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":264,"threads_started":1}
I20260926 23:30:14.872979 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:14.843640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42647 (local address 127.0.97.3:60139)
0926 23:30:14.844698 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:14.844754 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:14.847199 (+ 2445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:14.848326 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:14.848348 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:14.849412 (+ 1064us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:14.850727 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:14.850770 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:14.852328 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:14.852340 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:14.854251 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:14.854266 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:14.855413 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:14.855419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:14.855648 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:14.856794 (+ 1146us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:14.856840 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:14.860838 (+ 3998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:14.865827 (+ 4989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:14.865839 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:14.865865 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:14.866825 (+ 960us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:14.867481 (+ 656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:14.867487 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:14.867491 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:14.867937 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:14.868847 (+ 910us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:14.868858 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:14.869584 (+ 726us) client_negotiation.cc:769] Sending connection context
0926 23:30:14.870014 (+ 430us) client_negotiation.cc:240] Negotiation successful
0926 23:30:14.870868 (+ 854us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":391,"threads_started":1}
I20260926 23:30:14.893219 1866 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:32889
I20260926 23:30:14.893302 2001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:32889 every 8 connection(s)
I20260926 23:30:14.895077 1866 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:30:14.898252 2002 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42647
I20260926 23:30:14.899111 2002 heartbeater.cc:464] Registering TS with master...
I20260926 23:30:14.900097 2002 heartbeater.cc:511] Master 127.0.97.62:42647 requested a full tablet report, sending...
I20260926 23:30:14.901454 1890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:14.883029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40851 (local address 127.0.97.62:42647)
0926 23:30:14.883247 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:14.883254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:14.884021 (+ 767us) server_negotiation.cc:407] Connection header received
0926 23:30:14.884362 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:14.884370 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:14.884490 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:14.884567 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:14.886301 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:14.887314 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:14.890117 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:14.890759 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:14.890898 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:14.894316 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:14.894324 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:14.894330 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:14.894435 (+ 105us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:14.898113 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:14.899156 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:14.899170 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:14.899173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:14.899357 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:14.899830 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:14.899834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:14.899837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:14.900189 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:14.900341 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:14.900850 (+ 509us) server_negotiation.cc:299] Negotiation successful
0926 23:30:14.901058 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 23:30:14.901604 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:14.883029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42647 (local address 127.0.97.3:40851)
0926 23:30:14.883875 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:14.883898 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:14.884175 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:14.884854 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:14.884859 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:14.885564 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:14.886127 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:14.886142 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:14.887549 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:14.887558 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:14.889780 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:14.889810 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:14.891044 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:14.891054 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:14.891319 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:14.892322 (+ 1003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:14.892349 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:14.893965 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:14.898312 (+ 4347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:14.898319 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:14.898323 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:14.898963 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:14.899503 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:14.899508 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:14.899512 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:14.899685 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:14.900348 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:14.900353 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:14.900574 (+ 221us) client_negotiation.cc:769] Sending connection context
0926 23:30:14.900738 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 23:30:14.901163 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":509,"threads_started":1}
I20260926 23:30:14.902123 1514 ts_manager.cc:194] Registered new tserver with Master: 596576a3490444db809867705df4339d (127.0.97.3:32889)
I20260926 23:30:14.903193 1514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60139
I20260926 23:30:14.905072 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 1866
I20260926 23:30:14.905256 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:30:14.905776 388 external_mini_cluster.cc:1428] Setting key ae0cabe2c5dc7bab53129e9008dedfbc
I20260926 23:30:14.909250 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 23:30:14 dist-test-slave-gwfx krb5kdc[1470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465414, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 23:30:14 dist-test-slave-gwfx krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465414, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:14.962237 1890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:14.945340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43964 (local address 127.0.97.62:42647)
0926 23:30:14.945515 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:14.945519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:14.945875 (+ 356us) server_negotiation.cc:407] Connection header received
0926 23:30:14.946047 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:14.946051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:14.946130 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:14.946246 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:14.948027 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:14.948825 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:14.950500 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:14.951001 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:14.951196 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:14.955838 (+ 4642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:14.955843 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:14.955847 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:14.955891 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:14.958607 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:14.959547 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:14.959552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:14.959554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:14.959724 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:14.960217 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:14.960225 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:14.960229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:14.960746 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:14.961181 (+ 435us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:14.961460 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 23:30:14.961778 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 23:30:14.967252 1514 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:30:14.974920 1514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:30:15.003198 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:14.990657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41071 (local address 127.0.0.1:38006)
0926 23:30:14.991585 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:14.991645 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:14.991846 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:14.993748 (+ 1902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:14.993756 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:14.993787 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:30:14.994679 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:14.994720 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:14.996835 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:14.996840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:15.000043 (+ 3203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:15.000059 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.001879 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.001886 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:15.002135 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:15.002195 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 23:30:15.002350 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 23:30:15.002647 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":98,"thread_start_us":418,"threads_started":1}
I20260926 23:30:15.003754 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:14.991319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57684 (local address 127.0.97.1:45241)
0926 23:30:14.992983 (+ 1664us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:14.992993 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:14.993028 (+ 35us) server_negotiation.cc:407] Connection header received
0926 23:30:14.993144 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:14.993154 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:14.993481 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:14.993704 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:30:14.995254 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:14.996706 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.001019 (+ 4313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.002568 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.002738 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:15.002975 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:15.003313 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 23:30:15.003462 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1458,"thread_start_us":2733,"threads_started":1}
I20260926 23:30:15.003754 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:14.990905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38006 (local address 127.0.97.2:41071)
0926 23:30:14.993098 (+ 2193us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:14.993105 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:14.993128 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:30:14.993214 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:14.993219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:14.993475 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:14.994325 (+ 850us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:30:14.995055 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:14.996661 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.000266 (+ 3605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.001654 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.001794 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:15.002072 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:15.003170 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0926 23:30:15.003416 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2036,"thread_start_us":1880,"threads_started":1}
I20260926 23:30:15.004616 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:14.990847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45241 (local address 127.0.0.1:57684)
0926 23:30:14.992429 (+ 1582us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:14.992446 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:14.992566 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:14.993973 (+ 1407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:14.993977 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:14.993996 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:30:14.995050 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:14.995062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:14.996910 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:14.996914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:15.000795 (+ 3881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:15.000806 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.002865 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.002872 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:15.003081 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:15.003128 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 23:30:15.004003 (+ 875us) client_negotiation.cc:240] Negotiation successful
0926 23:30:15.004184 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1451,"mutex_wait_us":88,"thread_start_us":1284,"threads_started":1}
I20260926 23:30:15.009864 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:14.992970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:32889 (local address 127.0.0.1:38240)
0926 23:30:14.997899 (+ 4929us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:14.997923 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:14.998059 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:15.000236 (+ 2177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:15.000244 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:15.000291 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:30:15.000691 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:15.000705 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.004382 (+ 3677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.004389 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:15.006563 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:15.006587 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.008725 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.008734 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:15.008972 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:15.009017 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 23:30:15.009148 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 23:30:15.009333 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4771,"thread_start_us":4439,"threads_started":1}
I20260926 23:30:15.009397 1652 tablet_service.cc:1467] Processing CreateTablet for tablet 0043ae38cee548318786f4e63d7b9f95 (DEFAULT_TABLE table=test-table [id=9ec0071e8d5e4fb58e3e0a5828665d9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:30:15.008991 1792 tablet_service.cc:1467] Processing CreateTablet for tablet 0043ae38cee548318786f4e63d7b9f95 (DEFAULT_TABLE table=test-table [id=9ec0071e8d5e4fb58e3e0a5828665d9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:30:15.011175 2020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:14.994258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38240 (local address 127.0.97.3:32889)
0926 23:30:14.999227 (+ 4969us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:14.999237 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:14.999278 (+ 41us) server_negotiation.cc:407] Connection header received
0926 23:30:14.999405 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:14.999423 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:14.999775 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:15.000119 (+ 344us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:30:15.001038 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.004210 (+ 3172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.006859 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.008439 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.008615 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:15.008860 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:15.010550 (+ 1690us) server_negotiation.cc:299] Negotiation successful
0926 23:30:15.010789 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4758,"thread_start_us":4523,"threads_started":1}
I20260926 23:30:15.011700 1652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0043ae38cee548318786f4e63d7b9f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:15.011868 1792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0043ae38cee548318786f4e63d7b9f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:15.016268 1932 tablet_service.cc:1467] Processing CreateTablet for tablet 0043ae38cee548318786f4e63d7b9f95 (DEFAULT_TABLE table=test-table [id=9ec0071e8d5e4fb58e3e0a5828665d9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:30:15.018364 1932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0043ae38cee548318786f4e63d7b9f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:15.033154 2021 tablet_bootstrap.cc:492] T 0043ae38cee548318786f4e63d7b9f95 P 762530e791834c2cb07652dbde301eb1: Bootstrap starting.
I20260926 23:30:15.034134 2022 tablet_bootstrap.cc:492] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d: Bootstrap starting.
I20260926 23:30:15.038955 2022 tablet_bootstrap.cc:654] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:15.038954 2021 tablet_bootstrap.cc:654] T 0043ae38cee548318786f4e63d7b9f95 P 762530e791834c2cb07652dbde301eb1: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:15.039624 2023 tablet_bootstrap.cc:492] T 0043ae38cee548318786f4e63d7b9f95 P c7a26f6aa4ca4af587543e3d7ad5670f: Bootstrap starting.
I20260926 23:30:15.041649 2022 log.cc:824] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d: Log is configured to *not* fsync() on all Append() calls
I20260926 23:30:15.041658 2021 log.cc:824] T 0043ae38cee548318786f4e63d7b9f95 P 762530e791834c2cb07652dbde301eb1: Log is configured to *not* fsync() on all Append() calls
I20260926 23:30:15.044916 2023 tablet_bootstrap.cc:654] T 0043ae38cee548318786f4e63d7b9f95 P c7a26f6aa4ca4af587543e3d7ad5670f: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:15.045996 2022 tablet_bootstrap.cc:492] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d: No bootstrap required, opened a new log
I20260926 23:30:15.045994 2021 tablet_bootstrap.cc:492] T 0043ae38cee548318786f4e63d7b9f95 P 762530e791834c2cb07652dbde301eb1: No bootstrap required, opened a new log
I20260926 23:30:15.046172 2023 log.cc:824] T 0043ae38cee548318786f4e63d7b9f95 P c7a26f6aa4ca4af587543e3d7ad5670f: Log is configured to *not* fsync() on all Append() calls
I20260926 23:30:15.046386 2022 ts_tablet_manager.cc:1397] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.008s
I20260926 23:30:15.046386 2021 ts_tablet_manager.cc:1397] T 0043ae38cee548318786f4e63d7b9f95 P 762530e791834c2cb07652dbde301eb1: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.008s
I20260926 23:30:15.049103 2023 tablet_bootstrap.cc:492] T 0043ae38cee548318786f4e63d7b9f95 P c7a26f6aa4ca4af587543e3d7ad5670f: No bootstrap required, opened a new log
I20260926 23:30:15.049417 2023 ts_tablet_manager.cc:1397] T 0043ae38cee548318786f4e63d7b9f95 P c7a26f6aa4ca4af587543e3d7ad5670f: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260926 23:30:15.057379 2023 raft_consensus.cc:348] T 0043ae38cee548318786f4e63d7b9f95 P c7a26f6aa4ca4af587543e3d7ad5670f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } }
I20260926 23:30:15.057834 2023 raft_consensus.cc:374] T 0043ae38cee548318786f4e63d7b9f95 P c7a26f6aa4ca4af587543e3d7ad5670f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:15.057935 2023 raft_consensus.cc:729] T 0043ae38cee548318786f4e63d7b9f95 P c7a26f6aa4ca4af587543e3d7ad5670f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7a26f6aa4ca4af587543e3d7ad5670f, State: Initialized, Role: FOLLOWER
I20260926 23:30:15.058606 2023 consensus_queue.cc:260] T 0043ae38cee548318786f4e63d7b9f95 P c7a26f6aa4ca4af587543e3d7ad5670f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } }
I20260926 23:30:15.059690 2022 raft_consensus.cc:348] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } }
I20260926 23:30:15.060003 1862 heartbeater.cc:503] Master 127.0.97.62:42647 was elected leader, sending a full tablet report...
I20260926 23:30:15.060199 2022 raft_consensus.cc:374] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:15.060316 2022 raft_consensus.cc:729] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596576a3490444db809867705df4339d, State: Initialized, Role: FOLLOWER
I20260926 23:30:15.059170 2021 raft_consensus.cc:348] T 0043ae38cee548318786f4e63d7b9f95 P 762530e791834c2cb07652dbde301eb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } }
I20260926 23:30:15.060600 2023 ts_tablet_manager.cc:1428] T 0043ae38cee548318786f4e63d7b9f95 P c7a26f6aa4ca4af587543e3d7ad5670f: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260926 23:30:15.060781 2021 raft_consensus.cc:374] T 0043ae38cee548318786f4e63d7b9f95 P 762530e791834c2cb07652dbde301eb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:15.060961 2021 raft_consensus.cc:729] T 0043ae38cee548318786f4e63d7b9f95 P 762530e791834c2cb07652dbde301eb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 762530e791834c2cb07652dbde301eb1, State: Initialized, Role: FOLLOWER
I20260926 23:30:15.061268 2022 consensus_queue.cc:260] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } }
I20260926 23:30:15.061753 2021 consensus_queue.cc:260] T 0043ae38cee548318786f4e63d7b9f95 P 762530e791834c2cb07652dbde301eb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } }
I20260926 23:30:15.062620 2002 heartbeater.cc:503] Master 127.0.97.62:42647 was elected leader, sending a full tablet report...
I20260926 23:30:15.063532 2022 ts_tablet_manager.cc:1428] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d: Time spent starting tablet: real 0.017s user 0.009s sys 0.010s
I20260926 23:30:15.067394 2021 ts_tablet_manager.cc:1428] T 0043ae38cee548318786f4e63d7b9f95 P 762530e791834c2cb07652dbde301eb1: Time spent starting tablet: real 0.021s user 0.012s sys 0.008s
I20260926 23:30:15.073007 2027 raft_consensus.cc:482] T 0043ae38cee548318786f4e63d7b9f95 P c7a26f6aa4ca4af587543e3d7ad5670f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:30:15.073307 2028 raft_consensus.cc:482] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:30:15.073446 2027 raft_consensus.cc:504] T 0043ae38cee548318786f4e63d7b9f95 P c7a26f6aa4ca4af587543e3d7ad5670f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } }
I20260926 23:30:15.073597 2028 raft_consensus.cc:504] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } }
I20260926 23:30:15.076330 2028 leader_election.cc:288] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7a26f6aa4ca4af587543e3d7ad5670f (127.0.97.2:41071), 762530e791834c2cb07652dbde301eb1 (127.0.97.1:45241)
I20260926 23:30:15.081682 2027 leader_election.cc:288] T 0043ae38cee548318786f4e63d7b9f95 P c7a26f6aa4ca4af587543e3d7ad5670f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 762530e791834c2cb07652dbde301eb1 (127.0.97.1:45241), 596576a3490444db809867705df4339d (127.0.97.3:32889)
I20260926 23:30:15.086840 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:15.076365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41071 (local address 127.0.97.3:58967)
0926 23:30:15.076810 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:15.076838 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:15.077015 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:15.077812 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:15.077819 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:15.077858 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:30:15.078132 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:15.078146 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.079930 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.079937 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:15.082146 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:15.082167 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.085740 (+ 3573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.085749 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:15.086027 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:15.086073 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 23:30:15.086187 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 23:30:15.086398 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284}
I20260926 23:30:15.087790 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:15.076881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58967 (local address 127.0.97.2:41071)
0926 23:30:15.077304 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:15.077310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:15.077328 (+ 18us) server_negotiation.cc:407] Connection header received
0926 23:30:15.077414 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:15.077420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:15.077503 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:15.077598 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:30:15.078335 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.079722 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.082518 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.084006 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.084177 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:15.084342 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:15.087244 (+ 2902us) server_negotiation.cc:299] Negotiation successful
0926 23:30:15.087472 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261}
I20260926 23:30:15.087783 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:15.076931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40237 (local address 127.0.97.1:45241)
0926 23:30:15.077132 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:15.077139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:15.078044 (+ 905us) server_negotiation.cc:407] Connection header received
0926 23:30:15.078392 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:15.078399 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:15.078508 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:15.078581 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:30:15.079549 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.080910 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.084739 (+ 3829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.086064 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.086189 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:15.086308 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:15.087291 (+ 983us) server_negotiation.cc:299] Negotiation successful
0926 23:30:15.087462 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 23:30:15.088966 2030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:15.076524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45241 (local address 127.0.97.3:40237)
0926 23:30:15.077876 (+ 1352us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:15.077901 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:15.078206 (+ 305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:15.078874 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:15.078880 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:15.078914 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:30:15.079348 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:15.079363 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.082160 (+ 2797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.082166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:15.084487 (+ 2321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:15.084508 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.086561 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.086568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:15.086830 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:15.086872 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 23:30:15.088437 (+ 1565us) client_negotiation.cc:240] Negotiation successful
0926 23:30:15.088619 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1136,"thread_start_us":846,"threads_started":1}
I20260926 23:30:15.088966 1817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0043ae38cee548318786f4e63d7b9f95" candidate_uuid: "596576a3490444db809867705df4339d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" is_pre_election: true
I20260926 23:30:15.089550 1817 raft_consensus.cc:2454] T 0043ae38cee548318786f4e63d7b9f95 P c7a26f6aa4ca4af587543e3d7ad5670f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 596576a3490444db809867705df4339d in term 0.
I20260926 23:30:15.091045 1884 leader_election.cc:302] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 596576a3490444db809867705df4339d, c7a26f6aa4ca4af587543e3d7ad5670f; no voters:
I20260926 23:30:15.091624 2028 raft_consensus.cc:2781] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:30:15.091826 2028 raft_consensus.cc:482] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:30:15.091933 2028 raft_consensus.cc:3037] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:15.091925 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0043ae38cee548318786f4e63d7b9f95" candidate_uuid: "596576a3490444db809867705df4339d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762530e791834c2cb07652dbde301eb1" is_pre_election: true
I20260926 23:30:15.092568 1672 raft_consensus.cc:2454] T 0043ae38cee548318786f4e63d7b9f95 P 762530e791834c2cb07652dbde301eb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 596576a3490444db809867705df4339d in term 0.
I20260926 23:30:15.096081 2028 raft_consensus.cc:504] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } }
I20260926 23:30:15.097172 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:15.077192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50429 (local address 127.0.97.1:45241)
0926 23:30:15.088075 (+ 10883us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:15.088081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:15.088530 (+ 449us) server_negotiation.cc:407] Connection header received
0926 23:30:15.088855 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:15.088861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:15.088952 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:15.089014 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:30:15.089974 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.092044 (+ 2070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.094825 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.095887 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.096024 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:15.096136 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:15.096724 (+ 588us) server_negotiation.cc:299] Negotiation successful
0926 23:30:15.096859 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10771,"thread_start_us":13426,"threads_started":1}
I20260926 23:30:15.097221 2028 leader_election.cc:288] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d [CANDIDATE]: Term 1 election: Requested vote from peers c7a26f6aa4ca4af587543e3d7ad5670f (127.0.97.2:41071), 762530e791834c2cb07652dbde301eb1 (127.0.97.1:45241)
I20260926 23:30:15.097757 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0043ae38cee548318786f4e63d7b9f95" candidate_uuid: "596576a3490444db809867705df4339d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762530e791834c2cb07652dbde301eb1"
I20260926 23:30:15.097772 1817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0043ae38cee548318786f4e63d7b9f95" candidate_uuid: "596576a3490444db809867705df4339d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f"
I20260926 23:30:15.097981 1672 raft_consensus.cc:3037] T 0043ae38cee548318786f4e63d7b9f95 P 762530e791834c2cb07652dbde301eb1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:15.098026 1817 raft_consensus.cc:3037] T 0043ae38cee548318786f4e63d7b9f95 P c7a26f6aa4ca4af587543e3d7ad5670f [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:15.098778 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:15.077291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45241 (local address 127.0.97.2:50429)
0926 23:30:15.088402 (+ 11111us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:15.088423 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:15.088667 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:15.089250 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:15.089255 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:15.089284 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:30:15.089779 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:15.089795 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.092254 (+ 2459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.092260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:15.094585 (+ 2325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:15.094604 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.096169 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.096177 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:15.096438 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:15.096513 (+ 75us) client_negotiation.cc:769] Sending connection context
0926 23:30:15.097421 (+ 908us) client_negotiation.cc:240] Negotiation successful
0926 23:30:15.097603 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10957,"mutex_wait_us":51,"thread_start_us":11225,"threads_started":1}
I20260926 23:30:15.100203 2020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:15.089298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44605 (local address 127.0.97.3:32889)
0926 23:30:15.089499 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:15.089506 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:15.092262 (+ 2756us) server_negotiation.cc:407] Connection header received
0926 23:30:15.092675 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:15.092681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:15.092765 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:15.092825 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:30:15.093954 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.095246 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.097711 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.098967 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.099094 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:15.099216 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:15.099773 (+ 557us) server_negotiation.cc:299] Negotiation successful
0926 23:30:15.099930 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 23:30:15.100289 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0043ae38cee548318786f4e63d7b9f95" candidate_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762530e791834c2cb07652dbde301eb1" is_pre_election: true
I20260926 23:30:15.101306 2033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:15.089514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:32889 (local address 127.0.97.2:44605)
0926 23:30:15.092153 (+ 2639us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:15.092175 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:15.092468 (+ 293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:15.093074 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:15.093079 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:15.093106 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:30:15.093706 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:15.093724 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.095424 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.095431 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:15.097520 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:15.097534 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.099360 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.099367 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:15.099551 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:15.099588 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 23:30:15.100723 (+ 1135us) client_negotiation.cc:240] Negotiation successful
0926 23:30:15.100909 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2365,"mutex_wait_us":36,"thread_start_us":4134,"threads_started":1}
I20260926 23:30:15.102042 1817 raft_consensus.cc:2454] T 0043ae38cee548318786f4e63d7b9f95 P c7a26f6aa4ca4af587543e3d7ad5670f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 596576a3490444db809867705df4339d in term 1.
I20260926 23:30:15.102741 1952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0043ae38cee548318786f4e63d7b9f95" candidate_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596576a3490444db809867705df4339d" is_pre_election: true
I20260926 23:30:15.102042 1672 raft_consensus.cc:2454] T 0043ae38cee548318786f4e63d7b9f95 P 762530e791834c2cb07652dbde301eb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 596576a3490444db809867705df4339d in term 1.
I20260926 23:30:15.103219 1884 leader_election.cc:302] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 596576a3490444db809867705df4339d, c7a26f6aa4ca4af587543e3d7ad5670f; no voters:
W20260926 23:30:15.103214 1723 tablet.cc:2369] T 0043ae38cee548318786f4e63d7b9f95 P 762530e791834c2cb07652dbde301eb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:30:15.103515 1952 raft_consensus.cc:2379] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c7a26f6aa4ca4af587543e3d7ad5670f in current term 1: Already voted for candidate 596576a3490444db809867705df4339d in this term.
I20260926 23:30:15.104436 2028 raft_consensus.cc:2781] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:30:15.104503 1743 leader_election.cc:302] T 0043ae38cee548318786f4e63d7b9f95 P c7a26f6aa4ca4af587543e3d7ad5670f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c7a26f6aa4ca4af587543e3d7ad5670f; no voters: 596576a3490444db809867705df4339d, 762530e791834c2cb07652dbde301eb1
I20260926 23:30:15.105103 2027 raft_consensus.cc:2726] T 0043ae38cee548318786f4e63d7b9f95 P c7a26f6aa4ca4af587543e3d7ad5670f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 23:30:15.105692 2028 raft_consensus.cc:686] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d [term 1 LEADER]: Becoming Leader. State: Replica: 596576a3490444db809867705df4339d, State: Running, Role: LEADER
I20260926 23:30:15.106297 2028 consensus_queue.cc:237] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } }
I20260926 23:30:15.112182 1513 catalog_manager.cc:5495] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d reported cstate change: term changed from 0 to 1, leader changed from <none> to 596576a3490444db809867705df4339d (127.0.97.3). New cstate: current_term: 1 leader_uuid: "596576a3490444db809867705df4339d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } health_report { overall_health: HEALTHY } } }
W20260926 23:30:15.147292 2003 tablet.cc:2369] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:30:15.154300 2020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:15.147719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38252 (local address 127.0.97.3:32889)
0926 23:30:15.148034 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:15.148040 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:15.148067 (+ 27us) server_negotiation.cc:407] Connection header received
0926 23:30:15.148435 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:15.148441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:15.148551 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:15.148708 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:30:15.149114 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.149993 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.151483 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.152000 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.152116 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:15.152793 (+ 677us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:30:15.153563 (+ 770us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:30:15.153701 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:15.153879 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 23:30:15.153972 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260926 23:30:15.164620 1672 raft_consensus.cc:1261] T 0043ae38cee548318786f4e63d7b9f95 P 762530e791834c2cb07652dbde301eb1 [term 1 FOLLOWER]: Refusing update from remote peer 596576a3490444db809867705df4339d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:30:15.164712 1817 raft_consensus.cc:1261] T 0043ae38cee548318786f4e63d7b9f95 P c7a26f6aa4ca4af587543e3d7ad5670f [term 1 FOLLOWER]: Refusing update from remote peer 596576a3490444db809867705df4339d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:30:15.165709 2034 consensus_queue.cc:1035] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d [LEADER]: Connected to new peer: Peer: permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:30:15.166188 2028 consensus_queue.cc:1035] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:30:15.180717 2037 mvcc.cc:205] Tried to move back new op lower bound from 7333746340501057536 to 7333746340280635392. Current Snapshot: MvccSnapshot[applied={T|T < 7333746340501057536}]
I20260926 23:30:15.206727 1515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:30:15.207211 1515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:30:15.212251 1515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P beae4f80ff99489785e3c26ed7c186cb: Sending DeleteTablet for 3 replicas of tablet 0043ae38cee548318786f4e63d7b9f95
I20260926 23:30:15.213728 1652 tablet_service.cc:1514] Processing DeleteTablet for tablet 0043ae38cee548318786f4e63d7b9f95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:30:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57684
I20260926 23:30:15.213953 1792 tablet_service.cc:1514] Processing DeleteTablet for tablet 0043ae38cee548318786f4e63d7b9f95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:30:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38006
I20260926 23:30:15.214223 1932 tablet_service.cc:1514] Processing DeleteTablet for tablet 0043ae38cee548318786f4e63d7b9f95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:30:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38240
I20260926 23:30:15.219154 1515 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:30:15.220091 1515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:30:15.226192 2049 tablet_replica.cc:330] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d: stopping tablet replica
I20260926 23:30:15.227020 2049 raft_consensus.cc:2229] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:30:15.227962 2049 raft_consensus.cc:2258] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:30:15.230602 2049 ts_tablet_manager.cc:1905] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:30:15.233641 1791 tablet_service.cc:1467] Processing CreateTablet for tablet fff6b1e7a60646f3a9481265388ce11e (DEFAULT_TABLE table=test-table [id=45cbf0aaf40c4bf788e2cde9f50386fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:30:15.234860 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fff6b1e7a60646f3a9481265388ce11e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:15.235457 1932 tablet_service.cc:1467] Processing CreateTablet for tablet fff6b1e7a60646f3a9481265388ce11e (DEFAULT_TABLE table=test-table [id=45cbf0aaf40c4bf788e2cde9f50386fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:30:15.236064 1932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fff6b1e7a60646f3a9481265388ce11e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:15.235903 1651 tablet_service.cc:1467] Processing CreateTablet for tablet fff6b1e7a60646f3a9481265388ce11e (DEFAULT_TABLE table=test-table [id=45cbf0aaf40c4bf788e2cde9f50386fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:30:15.236598 1651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fff6b1e7a60646f3a9481265388ce11e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:15.239634 2049 ts_tablet_manager.cc:1918] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:30:15.239810 2049 log.cc:1196] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-2/wal/wals/0043ae38cee548318786f4e63d7b9f95
I20260926 23:30:15.240367 2049 ts_tablet_manager.cc:1939] T 0043ae38cee548318786f4e63d7b9f95 P 596576a3490444db809867705df4339d: Deleting consensus metadata
I20260926 23:30:15.242588 1500 catalog_manager.cc:4841] TS 596576a3490444db809867705df4339d (127.0.97.3:32889): tablet 0043ae38cee548318786f4e63d7b9f95 (table test-table [id=9ec0071e8d5e4fb58e3e0a5828665d9d]) successfully deleted
I20260926 23:30:15.242977 2022 tablet_bootstrap.cc:492] T fff6b1e7a60646f3a9481265388ce11e P 596576a3490444db809867705df4339d: Bootstrap starting.
I20260926 23:30:15.243556 2048 tablet_replica.cc:330] T 0043ae38cee548318786f4e63d7b9f95 P c7a26f6aa4ca4af587543e3d7ad5670f: stopping tablet replica
I20260926 23:30:15.244251 2048 raft_consensus.cc:2229] T 0043ae38cee548318786f4e63d7b9f95 P c7a26f6aa4ca4af587543e3d7ad5670f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:30:15.244848 2023 tablet_bootstrap.cc:492] T fff6b1e7a60646f3a9481265388ce11e P c7a26f6aa4ca4af587543e3d7ad5670f: Bootstrap starting.
I20260926 23:30:15.244977 2022 tablet_bootstrap.cc:654] T fff6b1e7a60646f3a9481265388ce11e P 596576a3490444db809867705df4339d: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:15.245398 2048 raft_consensus.cc:2258] T 0043ae38cee548318786f4e63d7b9f95 P c7a26f6aa4ca4af587543e3d7ad5670f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:30:15.247542 2023 tablet_bootstrap.cc:654] T fff6b1e7a60646f3a9481265388ce11e P c7a26f6aa4ca4af587543e3d7ad5670f: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:15.251540 2047 tablet_replica.cc:330] T 0043ae38cee548318786f4e63d7b9f95 P 762530e791834c2cb07652dbde301eb1: stopping tablet replica
I20260926 23:30:15.252172 2047 raft_consensus.cc:2229] T 0043ae38cee548318786f4e63d7b9f95 P 762530e791834c2cb07652dbde301eb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:30:15.252862 2047 raft_consensus.cc:2258] T 0043ae38cee548318786f4e63d7b9f95 P 762530e791834c2cb07652dbde301eb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:30:15.253698 2048 ts_tablet_manager.cc:1905] T 0043ae38cee548318786f4e63d7b9f95 P c7a26f6aa4ca4af587543e3d7ad5670f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:30:15.255445 2047 ts_tablet_manager.cc:1905] T 0043ae38cee548318786f4e63d7b9f95 P 762530e791834c2cb07652dbde301eb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:30:15.256393 2023 tablet_bootstrap.cc:492] T fff6b1e7a60646f3a9481265388ce11e P c7a26f6aa4ca4af587543e3d7ad5670f: No bootstrap required, opened a new log
I20260926 23:30:15.256577 2023 ts_tablet_manager.cc:1397] T fff6b1e7a60646f3a9481265388ce11e P c7a26f6aa4ca4af587543e3d7ad5670f: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.007s
I20260926 23:30:15.257045 2022 tablet_bootstrap.cc:492] T fff6b1e7a60646f3a9481265388ce11e P 596576a3490444db809867705df4339d: No bootstrap required, opened a new log
I20260926 23:30:15.257314 2022 ts_tablet_manager.cc:1397] T fff6b1e7a60646f3a9481265388ce11e P 596576a3490444db809867705df4339d: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.002s
I20260926 23:30:15.258275 2022 raft_consensus.cc:348] T fff6b1e7a60646f3a9481265388ce11e P 596576a3490444db809867705df4339d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } }
I20260926 23:30:15.258636 2022 raft_consensus.cc:374] T fff6b1e7a60646f3a9481265388ce11e P 596576a3490444db809867705df4339d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:15.258711 2021 tablet_bootstrap.cc:492] T fff6b1e7a60646f3a9481265388ce11e P 762530e791834c2cb07652dbde301eb1: Bootstrap starting.
I20260926 23:30:15.258846 2022 raft_consensus.cc:729] T fff6b1e7a60646f3a9481265388ce11e P 596576a3490444db809867705df4339d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596576a3490444db809867705df4339d, State: Initialized, Role: FOLLOWER
I20260926 23:30:15.259200 2022 consensus_queue.cc:260] T fff6b1e7a60646f3a9481265388ce11e P 596576a3490444db809867705df4339d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } }
I20260926 23:30:15.260035 2022 ts_tablet_manager.cc:1428] T fff6b1e7a60646f3a9481265388ce11e P 596576a3490444db809867705df4339d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 23:30:15.260581 2021 tablet_bootstrap.cc:654] T fff6b1e7a60646f3a9481265388ce11e P 762530e791834c2cb07652dbde301eb1: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:15.263203 2047 ts_tablet_manager.cc:1918] T 0043ae38cee548318786f4e63d7b9f95 P 762530e791834c2cb07652dbde301eb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:30:15.263418 2047 log.cc:1196] T 0043ae38cee548318786f4e63d7b9f95 P 762530e791834c2cb07652dbde301eb1: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-0/wal/wals/0043ae38cee548318786f4e63d7b9f95
I20260926 23:30:15.263993 2047 ts_tablet_manager.cc:1939] T 0043ae38cee548318786f4e63d7b9f95 P 762530e791834c2cb07652dbde301eb1: Deleting consensus metadata
I20260926 23:30:15.264084 2023 raft_consensus.cc:348] T fff6b1e7a60646f3a9481265388ce11e P c7a26f6aa4ca4af587543e3d7ad5670f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } }
I20260926 23:30:15.264434 2023 raft_consensus.cc:374] T fff6b1e7a60646f3a9481265388ce11e P c7a26f6aa4ca4af587543e3d7ad5670f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:15.264688 2023 raft_consensus.cc:729] T fff6b1e7a60646f3a9481265388ce11e P c7a26f6aa4ca4af587543e3d7ad5670f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7a26f6aa4ca4af587543e3d7ad5670f, State: Initialized, Role: FOLLOWER
I20260926 23:30:15.266117 1500 catalog_manager.cc:4841] TS 762530e791834c2cb07652dbde301eb1 (127.0.97.1:45241): tablet 0043ae38cee548318786f4e63d7b9f95 (table test-table [id=9ec0071e8d5e4fb58e3e0a5828665d9d]) successfully deleted
I20260926 23:30:15.266757 2023 consensus_queue.cc:260] T fff6b1e7a60646f3a9481265388ce11e P c7a26f6aa4ca4af587543e3d7ad5670f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } }
I20260926 23:30:15.267620 2023 ts_tablet_manager.cc:1428] T fff6b1e7a60646f3a9481265388ce11e P c7a26f6aa4ca4af587543e3d7ad5670f: Time spent starting tablet: real 0.011s user 0.000s sys 0.003s
I20260926 23:30:15.275305 2021 tablet_bootstrap.cc:492] T fff6b1e7a60646f3a9481265388ce11e P 762530e791834c2cb07652dbde301eb1: No bootstrap required, opened a new log
I20260926 23:30:15.275501 2021 ts_tablet_manager.cc:1397] T fff6b1e7a60646f3a9481265388ce11e P 762530e791834c2cb07652dbde301eb1: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.003s
I20260926 23:30:15.276394 2021 raft_consensus.cc:348] T fff6b1e7a60646f3a9481265388ce11e P 762530e791834c2cb07652dbde301eb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } }
I20260926 23:30:15.276693 2021 raft_consensus.cc:374] T fff6b1e7a60646f3a9481265388ce11e P 762530e791834c2cb07652dbde301eb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:15.276789 2021 raft_consensus.cc:729] T fff6b1e7a60646f3a9481265388ce11e P 762530e791834c2cb07652dbde301eb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 762530e791834c2cb07652dbde301eb1, State: Initialized, Role: FOLLOWER
I20260926 23:30:15.277078 2021 consensus_queue.cc:260] T fff6b1e7a60646f3a9481265388ce11e P 762530e791834c2cb07652dbde301eb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } }
I20260926 23:30:15.279583 2048 ts_tablet_manager.cc:1918] T 0043ae38cee548318786f4e63d7b9f95 P c7a26f6aa4ca4af587543e3d7ad5670f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:30:15.279881 2048 log.cc:1196] T 0043ae38cee548318786f4e63d7b9f95 P c7a26f6aa4ca4af587543e3d7ad5670f: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-1/wal/wals/0043ae38cee548318786f4e63d7b9f95
I20260926 23:30:15.280581 2048 ts_tablet_manager.cc:1939] T 0043ae38cee548318786f4e63d7b9f95 P c7a26f6aa4ca4af587543e3d7ad5670f: Deleting consensus metadata
I20260926 23:30:15.281050 2021 ts_tablet_manager.cc:1428] T fff6b1e7a60646f3a9481265388ce11e P 762530e791834c2cb07652dbde301eb1: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260926 23:30:15.283812 1501 catalog_manager.cc:4841] TS c7a26f6aa4ca4af587543e3d7ad5670f (127.0.97.2:41071): tablet 0043ae38cee548318786f4e63d7b9f95 (table test-table [id=9ec0071e8d5e4fb58e3e0a5828665d9d]) successfully deleted
W20260926 23:30:15.505252 1863 tablet.cc:2369] T fff6b1e7a60646f3a9481265388ce11e P c7a26f6aa4ca4af587543e3d7ad5670f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:30:15.668776 2034 raft_consensus.cc:482] T fff6b1e7a60646f3a9481265388ce11e P 596576a3490444db809867705df4339d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:30:15.669049 2034 raft_consensus.cc:504] T fff6b1e7a60646f3a9481265388ce11e P 596576a3490444db809867705df4339d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } }
I20260926 23:30:15.669772 2034 leader_election.cc:288] T fff6b1e7a60646f3a9481265388ce11e P 596576a3490444db809867705df4339d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 762530e791834c2cb07652dbde301eb1 (127.0.97.1:45241), c7a26f6aa4ca4af587543e3d7ad5670f (127.0.97.2:41071)
I20260926 23:30:15.670480 1817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff6b1e7a60646f3a9481265388ce11e" candidate_uuid: "596576a3490444db809867705df4339d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" is_pre_election: true
I20260926 23:30:15.670743 1817 raft_consensus.cc:2454] T fff6b1e7a60646f3a9481265388ce11e P c7a26f6aa4ca4af587543e3d7ad5670f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 596576a3490444db809867705df4339d in term 0.
I20260926 23:30:15.670560 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff6b1e7a60646f3a9481265388ce11e" candidate_uuid: "596576a3490444db809867705df4339d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762530e791834c2cb07652dbde301eb1" is_pre_election: true
I20260926 23:30:15.670980 1672 raft_consensus.cc:2454] T fff6b1e7a60646f3a9481265388ce11e P 762530e791834c2cb07652dbde301eb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 596576a3490444db809867705df4339d in term 0.
I20260926 23:30:15.671226 1884 leader_election.cc:302] T fff6b1e7a60646f3a9481265388ce11e P 596576a3490444db809867705df4339d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 596576a3490444db809867705df4339d, c7a26f6aa4ca4af587543e3d7ad5670f; no voters:
I20260926 23:30:15.671576 2034 raft_consensus.cc:2781] T fff6b1e7a60646f3a9481265388ce11e P 596576a3490444db809867705df4339d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:30:15.671674 2034 raft_consensus.cc:482] T fff6b1e7a60646f3a9481265388ce11e P 596576a3490444db809867705df4339d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:30:15.671783 2034 raft_consensus.cc:3037] T fff6b1e7a60646f3a9481265388ce11e P 596576a3490444db809867705df4339d [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:15.673907 2034 raft_consensus.cc:504] T fff6b1e7a60646f3a9481265388ce11e P 596576a3490444db809867705df4339d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } }
I20260926 23:30:15.674508 2034 leader_election.cc:288] T fff6b1e7a60646f3a9481265388ce11e P 596576a3490444db809867705df4339d [CANDIDATE]: Term 1 election: Requested vote from peers 762530e791834c2cb07652dbde301eb1 (127.0.97.1:45241), c7a26f6aa4ca4af587543e3d7ad5670f (127.0.97.2:41071)
I20260926 23:30:15.675312 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff6b1e7a60646f3a9481265388ce11e" candidate_uuid: "596576a3490444db809867705df4339d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762530e791834c2cb07652dbde301eb1"
I20260926 23:30:15.675623 1672 raft_consensus.cc:3037] T fff6b1e7a60646f3a9481265388ce11e P 762530e791834c2cb07652dbde301eb1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:15.675704 1817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff6b1e7a60646f3a9481265388ce11e" candidate_uuid: "596576a3490444db809867705df4339d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f"
I20260926 23:30:15.675909 1817 raft_consensus.cc:3037] T fff6b1e7a60646f3a9481265388ce11e P c7a26f6aa4ca4af587543e3d7ad5670f [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:15.678880 1672 raft_consensus.cc:2454] T fff6b1e7a60646f3a9481265388ce11e P 762530e791834c2cb07652dbde301eb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 596576a3490444db809867705df4339d in term 1.
I20260926 23:30:15.678880 1817 raft_consensus.cc:2454] T fff6b1e7a60646f3a9481265388ce11e P c7a26f6aa4ca4af587543e3d7ad5670f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 596576a3490444db809867705df4339d in term 1.
I20260926 23:30:15.679402 1884 leader_election.cc:302] T fff6b1e7a60646f3a9481265388ce11e P 596576a3490444db809867705df4339d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 596576a3490444db809867705df4339d, c7a26f6aa4ca4af587543e3d7ad5670f; no voters:
I20260926 23:30:15.679824 2034 raft_consensus.cc:2781] T fff6b1e7a60646f3a9481265388ce11e P 596576a3490444db809867705df4339d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:30:15.679992 2034 raft_consensus.cc:686] T fff6b1e7a60646f3a9481265388ce11e P 596576a3490444db809867705df4339d [term 1 LEADER]: Becoming Leader. State: Replica: 596576a3490444db809867705df4339d, State: Running, Role: LEADER
I20260926 23:30:15.680243 2034 consensus_queue.cc:237] T fff6b1e7a60646f3a9481265388ce11e P 596576a3490444db809867705df4339d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } }
I20260926 23:30:15.688638 1513 catalog_manager.cc:5495] T fff6b1e7a60646f3a9481265388ce11e P 596576a3490444db809867705df4339d reported cstate change: term changed from 0 to 1, leader changed from <none> to 596576a3490444db809867705df4339d (127.0.97.3). New cstate: current_term: 1 leader_uuid: "596576a3490444db809867705df4339d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } health_report { overall_health: HEALTHY } } }
Sep 26 23:30:15 dist-test-slave-gwfx krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:15.750541 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:15.733906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42647 (local address 127.0.0.1:43968)
0926 23:30:15.734951 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:15.734973 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:15.735116 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:15.735510 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:15.735515 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:15.736210 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:15.736441 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:15.736451 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.737715 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.737719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:15.738822 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:15.738837 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.739750 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.739756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:15.739916 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:15.740702 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:15.740721 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:15.743548 (+ 2827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:15.747465 (+ 3917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:15.747480 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:15.747488 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:15.748087 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:15.748614 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:15.748618 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:15.748621 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:15.748804 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:15.749509 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:15.749519 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:15.749728 (+ 209us) client_negotiation.cc:769] Sending connection context
0926 23:30:15.749900 (+ 172us) client_negotiation.cc:240] Negotiation successful
0926 23:30:15.750211 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"thread_start_us":532,"threads_started":1}
I20260926 23:30:15.751005 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:15.733893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43968 (local address 127.0.97.62:42647)
0926 23:30:15.734986 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:15.734991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:15.735050 (+ 59us) server_negotiation.cc:407] Connection header received
0926 23:30:15.735232 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:15.735237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:15.735330 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:15.735465 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:15.736632 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.737521 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.739029 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.739518 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.739641 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:15.743833 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:15.743840 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:15.743845 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:15.743888 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:15.747219 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:15.748273 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:15.748278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:15.748281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:15.748457 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:15.748960 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:15.748965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:15.748967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:15.749322 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:15.749470 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:15.750063 (+ 593us) server_negotiation.cc:299] Negotiation successful
0926 23:30:15.750443 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":970,"thread_start_us":793,"threads_started":1}
I20260926 23:30:15.759877 1513 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43968:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 23:30:15.761304 1513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:30:15.769663 1791 tablet_service.cc:1467] Processing CreateTablet for tablet 81c7ebfa725c4337a8149fa7bb5cb55f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bbdd2ab2495d41cc8b207490a648f09a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 23:30:15.769676 1651 tablet_service.cc:1467] Processing CreateTablet for tablet 81c7ebfa725c4337a8149fa7bb5cb55f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bbdd2ab2495d41cc8b207490a648f09a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 23:30:15.769682 1932 tablet_service.cc:1467] Processing CreateTablet for tablet 81c7ebfa725c4337a8149fa7bb5cb55f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bbdd2ab2495d41cc8b207490a648f09a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 23:30:15.770238 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81c7ebfa725c4337a8149fa7bb5cb55f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:15.770370 1651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81c7ebfa725c4337a8149fa7bb5cb55f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:15.770370 1932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81c7ebfa725c4337a8149fa7bb5cb55f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:15.777458 2061 tablet_bootstrap.cc:492] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f: Bootstrap starting.
I20260926 23:30:15.778502 2021 tablet_bootstrap.cc:492] T 81c7ebfa725c4337a8149fa7bb5cb55f P 762530e791834c2cb07652dbde301eb1: Bootstrap starting.
I20260926 23:30:15.779497 2062 tablet_bootstrap.cc:492] T 81c7ebfa725c4337a8149fa7bb5cb55f P 596576a3490444db809867705df4339d: Bootstrap starting.
I20260926 23:30:15.780385 2021 tablet_bootstrap.cc:654] T 81c7ebfa725c4337a8149fa7bb5cb55f P 762530e791834c2cb07652dbde301eb1: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:15.780412 2061 tablet_bootstrap.cc:654] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:15.782210 2062 tablet_bootstrap.cc:654] T 81c7ebfa725c4337a8149fa7bb5cb55f P 596576a3490444db809867705df4339d: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:15.784960 2061 tablet_bootstrap.cc:492] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f: No bootstrap required, opened a new log
I20260926 23:30:15.785111 2061 ts_tablet_manager.cc:1397] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260926 23:30:15.785463 2062 tablet_bootstrap.cc:492] T 81c7ebfa725c4337a8149fa7bb5cb55f P 596576a3490444db809867705df4339d: No bootstrap required, opened a new log
I20260926 23:30:15.785658 2062 ts_tablet_manager.cc:1397] T 81c7ebfa725c4337a8149fa7bb5cb55f P 596576a3490444db809867705df4339d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 23:30:15.785853 2061 raft_consensus.cc:348] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } }
I20260926 23:30:15.786053 2061 raft_consensus.cc:374] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:15.786151 2061 raft_consensus.cc:729] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7a26f6aa4ca4af587543e3d7ad5670f, State: Initialized, Role: FOLLOWER
I20260926 23:30:15.786154 2021 tablet_bootstrap.cc:492] T 81c7ebfa725c4337a8149fa7bb5cb55f P 762530e791834c2cb07652dbde301eb1: No bootstrap required, opened a new log
I20260926 23:30:15.786340 2021 ts_tablet_manager.cc:1397] T 81c7ebfa725c4337a8149fa7bb5cb55f P 762530e791834c2cb07652dbde301eb1: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 23:30:15.786415 2061 consensus_queue.cc:260] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } }
I20260926 23:30:15.786532 2062 raft_consensus.cc:348] T 81c7ebfa725c4337a8149fa7bb5cb55f P 596576a3490444db809867705df4339d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } }
I20260926 23:30:15.786829 2062 raft_consensus.cc:374] T 81c7ebfa725c4337a8149fa7bb5cb55f P 596576a3490444db809867705df4339d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:15.786923 2062 raft_consensus.cc:729] T 81c7ebfa725c4337a8149fa7bb5cb55f P 596576a3490444db809867705df4339d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596576a3490444db809867705df4339d, State: Initialized, Role: FOLLOWER
I20260926 23:30:15.787134 2021 raft_consensus.cc:348] T 81c7ebfa725c4337a8149fa7bb5cb55f P 762530e791834c2cb07652dbde301eb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } }
I20260926 23:30:15.787220 2062 consensus_queue.cc:260] T 81c7ebfa725c4337a8149fa7bb5cb55f P 596576a3490444db809867705df4339d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } }
I20260926 23:30:15.787391 2021 raft_consensus.cc:374] T 81c7ebfa725c4337a8149fa7bb5cb55f P 762530e791834c2cb07652dbde301eb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:15.787457 2021 raft_consensus.cc:729] T 81c7ebfa725c4337a8149fa7bb5cb55f P 762530e791834c2cb07652dbde301eb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 762530e791834c2cb07652dbde301eb1, State: Initialized, Role: FOLLOWER
I20260926 23:30:15.787688 2021 consensus_queue.cc:260] T 81c7ebfa725c4337a8149fa7bb5cb55f P 762530e791834c2cb07652dbde301eb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } }
I20260926 23:30:15.787700 2034 tablet_replica.cc:439] T 81c7ebfa725c4337a8149fa7bb5cb55f P 596576a3490444db809867705df4339d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } } }
I20260926 23:30:15.787983 2034 tablet_replica.cc:442] T 81c7ebfa725c4337a8149fa7bb5cb55f P 596576a3490444db809867705df4339d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:30:15.789276 2062 ts_tablet_manager.cc:1428] T 81c7ebfa725c4337a8149fa7bb5cb55f P 596576a3490444db809867705df4339d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 23:30:15.789978 2021 ts_tablet_manager.cc:1428] T 81c7ebfa725c4337a8149fa7bb5cb55f P 762530e791834c2cb07652dbde301eb1: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 23:30:15.790529 2067 tablet_replica.cc:439] T 81c7ebfa725c4337a8149fa7bb5cb55f P 762530e791834c2cb07652dbde301eb1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } } }
I20260926 23:30:15.790898 2067 tablet_replica.cc:442] T 81c7ebfa725c4337a8149fa7bb5cb55f P 762530e791834c2cb07652dbde301eb1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:30:15.791651 2061 ts_tablet_manager.cc:1428] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260926 23:30:15.792179 2066 tablet_replica.cc:439] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } } }
I20260926 23:30:15.792563 2066 tablet_replica.cc:442] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:30:15.842730 2066 raft_consensus.cc:482] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:30:15.842993 2066 raft_consensus.cc:504] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } }
I20260926 23:30:15.843706 2066 leader_election.cc:288] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 762530e791834c2cb07652dbde301eb1 (127.0.97.1:45241), 596576a3490444db809867705df4339d (127.0.97.3:32889)
I20260926 23:30:15.844385 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81c7ebfa725c4337a8149fa7bb5cb55f" candidate_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762530e791834c2cb07652dbde301eb1" is_pre_election: true
I20260926 23:30:15.844539 1952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81c7ebfa725c4337a8149fa7bb5cb55f" candidate_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596576a3490444db809867705df4339d" is_pre_election: true
I20260926 23:30:15.844686 1672 raft_consensus.cc:2454] T 81c7ebfa725c4337a8149fa7bb5cb55f P 762530e791834c2cb07652dbde301eb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7a26f6aa4ca4af587543e3d7ad5670f in term 0.
I20260926 23:30:15.844867 1952 raft_consensus.cc:2454] T 81c7ebfa725c4337a8149fa7bb5cb55f P 596576a3490444db809867705df4339d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7a26f6aa4ca4af587543e3d7ad5670f in term 0.
I20260926 23:30:15.845165 1743 leader_election.cc:302] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 762530e791834c2cb07652dbde301eb1, c7a26f6aa4ca4af587543e3d7ad5670f; no voters:
I20260926 23:30:15.845573 2066 raft_consensus.cc:2781] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:30:15.845726 2066 raft_consensus.cc:482] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:30:15.845793 2066 raft_consensus.cc:3037] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:15.848098 2066 raft_consensus.cc:504] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } }
I20260926 23:30:15.849198 2066 leader_election.cc:288] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f [CANDIDATE]: Term 1 election: Requested vote from peers 762530e791834c2cb07652dbde301eb1 (127.0.97.1:45241), 596576a3490444db809867705df4339d (127.0.97.3:32889)
I20260926 23:30:15.849449 1952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81c7ebfa725c4337a8149fa7bb5cb55f" candidate_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596576a3490444db809867705df4339d"
I20260926 23:30:15.849555 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81c7ebfa725c4337a8149fa7bb5cb55f" candidate_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "762530e791834c2cb07652dbde301eb1"
I20260926 23:30:15.849711 1952 raft_consensus.cc:3037] T 81c7ebfa725c4337a8149fa7bb5cb55f P 596576a3490444db809867705df4339d [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:15.849802 1672 raft_consensus.cc:3037] T 81c7ebfa725c4337a8149fa7bb5cb55f P 762530e791834c2cb07652dbde301eb1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:15.853036 1952 raft_consensus.cc:2454] T 81c7ebfa725c4337a8149fa7bb5cb55f P 596576a3490444db809867705df4339d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7a26f6aa4ca4af587543e3d7ad5670f in term 1.
I20260926 23:30:15.853024 1672 raft_consensus.cc:2454] T 81c7ebfa725c4337a8149fa7bb5cb55f P 762530e791834c2cb07652dbde301eb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7a26f6aa4ca4af587543e3d7ad5670f in term 1.
I20260926 23:30:15.853643 1743 leader_election.cc:302] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 596576a3490444db809867705df4339d, c7a26f6aa4ca4af587543e3d7ad5670f; no voters:
I20260926 23:30:15.854032 2066 raft_consensus.cc:2781] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:30:15.854507 2066 raft_consensus.cc:686] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f [term 1 LEADER]: Becoming Leader. State: Replica: c7a26f6aa4ca4af587543e3d7ad5670f, State: Running, Role: LEADER
I20260926 23:30:15.854936 2066 consensus_queue.cc:237] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } }
I20260926 23:30:15.857035 2068 tablet_replica.cc:439] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f: TxnStatusTablet state changed. Reason: New leader c7a26f6aa4ca4af587543e3d7ad5670f. Latest consensus state: current_term: 1 leader_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } } }
I20260926 23:30:15.857432 2068 tablet_replica.cc:442] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f: This TxnStatusTablet replica's current role is: LEADER
I20260926 23:30:15.858279 2070 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 23:30:15.858539 2070 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 23:30:15.859231 1512 catalog_manager.cc:5495] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f reported cstate change: term changed from 0 to 1, leader changed from <none> to c7a26f6aa4ca4af587543e3d7ad5670f (127.0.97.2). New cstate: current_term: 1 leader_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } health_report { overall_health: UNKNOWN } } }
I20260926 23:30:15.884735 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:15.875192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41071 (local address 127.0.0.1:38008)
0926 23:30:15.875447 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:15.875469 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:15.875616 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:15.877040 (+ 1424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:15.877045 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:15.877059 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:30:15.877263 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:15.877273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.879167 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.879173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:15.880799 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:15.880815 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.882157 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.882163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:15.882405 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:15.882511 (+ 106us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:30:15.884078 (+ 1567us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:30:15.884106 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 23:30:15.884206 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 23:30:15.884388 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 23:30:15.884847 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:15.875158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38008 (local address 127.0.97.2:41071)
0926 23:30:15.876509 (+ 1351us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:15.876516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:15.876541 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:30:15.876653 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:15.876658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:15.876773 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:15.877402 (+ 629us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:30:15.877673 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.878965 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.880993 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.881912 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.882096 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:15.882804 (+ 708us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:30:15.883903 (+ 1099us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:30:15.884109 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:15.884258 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 23:30:15.884377 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1192,"thread_start_us":990,"threads_started":1}
I20260926 23:30:15.891570 1672 raft_consensus.cc:1261] T 81c7ebfa725c4337a8149fa7bb5cb55f P 762530e791834c2cb07652dbde301eb1 [term 1 FOLLOWER]: Refusing update from remote peer c7a26f6aa4ca4af587543e3d7ad5670f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:30:15.891713 1952 raft_consensus.cc:1261] T 81c7ebfa725c4337a8149fa7bb5cb55f P 596576a3490444db809867705df4339d [term 1 FOLLOWER]: Refusing update from remote peer c7a26f6aa4ca4af587543e3d7ad5670f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:30:15.892596 2066 consensus_queue.cc:1035] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f [LEADER]: Connected to new peer: Peer: permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:30:15.894361 2066 consensus_queue.cc:1035] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f [LEADER]: Connected to new peer: Peer: permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:30:15.916428 2067 tablet_replica.cc:439] T 81c7ebfa725c4337a8149fa7bb5cb55f P 762530e791834c2cb07652dbde301eb1: TxnStatusTablet state changed. Reason: New leader c7a26f6aa4ca4af587543e3d7ad5670f. Latest consensus state: current_term: 1 leader_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } } }
I20260926 23:30:15.916770 2067 tablet_replica.cc:442] T 81c7ebfa725c4337a8149fa7bb5cb55f P 762530e791834c2cb07652dbde301eb1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:30:15.922240 2034 tablet_replica.cc:439] T 81c7ebfa725c4337a8149fa7bb5cb55f P 596576a3490444db809867705df4339d: TxnStatusTablet state changed. Reason: New leader c7a26f6aa4ca4af587543e3d7ad5670f. Latest consensus state: current_term: 1 leader_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } } }
I20260926 23:30:15.922665 2034 tablet_replica.cc:442] T 81c7ebfa725c4337a8149fa7bb5cb55f P 596576a3490444db809867705df4339d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:30:15.923727 2074 mvcc.cc:205] Tried to move back new op lower bound from 7333746343481958400 to 7333746343345770496. Current Snapshot: MvccSnapshot[applied={T|T < 7333746343481958400}]
I20260926 23:30:15.933269 2067 tablet_replica.cc:439] T 81c7ebfa725c4337a8149fa7bb5cb55f P 762530e791834c2cb07652dbde301eb1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } } }
I20260926 23:30:15.933643 2067 tablet_replica.cc:442] T 81c7ebfa725c4337a8149fa7bb5cb55f P 762530e791834c2cb07652dbde301eb1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:30:15.939523 2066 tablet_replica.cc:439] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } } }
I20260926 23:30:15.939939 2066 tablet_replica.cc:442] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f: This TxnStatusTablet replica's current role is: LEADER
I20260926 23:30:15.944180 2066 tablet_replica.cc:439] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } } }
I20260926 23:30:15.944646 2066 tablet_replica.cc:442] T 81c7ebfa725c4337a8149fa7bb5cb55f P c7a26f6aa4ca4af587543e3d7ad5670f: This TxnStatusTablet replica's current role is: LEADER
I20260926 23:30:15.951130 2034 tablet_replica.cc:439] T 81c7ebfa725c4337a8149fa7bb5cb55f P 596576a3490444db809867705df4339d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 } } peers { permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 } } peers { permanent_uuid: "596576a3490444db809867705df4339d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32889 } } }
I20260926 23:30:15.951473 2034 tablet_replica.cc:442] T 81c7ebfa725c4337a8149fa7bb5cb55f P 596576a3490444db809867705df4339d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:30:15.968086 2084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:15.961048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41071 (local address 127.0.97.3:36521)
0926 23:30:15.961797 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:15.961815 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:15.961976 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:15.962490 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:15.962495 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:15.962511 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:30:15.962825 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:15.962835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.964458 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.964463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:15.965546 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:15.965562 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.966670 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.966676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:15.966859 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:15.966939 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:30:15.967501 (+ 562us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:30:15.967512 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 23:30:15.967605 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 23:30:15.967730 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":435,"threads_started":1}
I20260926 23:30:15.968072 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:15.961163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36521 (local address 127.0.97.2:41071)
0926 23:30:15.961326 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:15.961330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:15.961944 (+ 614us) server_negotiation.cc:407] Connection header received
0926 23:30:15.962146 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:15.962153 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:15.962273 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:15.962411 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:30:15.963009 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.964247 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.965731 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:15.966439 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:15.966588 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:15.967149 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:30:15.967330 (+ 181us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:30:15.967439 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:15.967647 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 23:30:15.967750 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 23:30:15.980458 1672 raft_consensus.cc:1261] T fff6b1e7a60646f3a9481265388ce11e P 762530e791834c2cb07652dbde301eb1 [term 1 FOLLOWER]: Refusing update from remote peer 596576a3490444db809867705df4339d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:30:15.980428 1817 raft_consensus.cc:1261] T fff6b1e7a60646f3a9481265388ce11e P c7a26f6aa4ca4af587543e3d7ad5670f [term 1 FOLLOWER]: Refusing update from remote peer 596576a3490444db809867705df4339d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:30:15.981101 2028 consensus_queue.cc:1035] T fff6b1e7a60646f3a9481265388ce11e P 596576a3490444db809867705df4339d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7a26f6aa4ca4af587543e3d7ad5670f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:30:15.981494 2034 consensus_queue.cc:1035] T fff6b1e7a60646f3a9481265388ce11e P 596576a3490444db809867705df4339d [LEADER]: Connected to new peer: Peer: permanent_uuid: "762530e791834c2cb07652dbde301eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:30:15.995590 2075 mvcc.cc:205] Tried to move back new op lower bound from 7333746343849721856 to 7333746342628651008. Current Snapshot: MvccSnapshot[applied={T|T < 7333746343849721856}]
I20260926 23:30:16.027721 2090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:16.018068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55429 (local address 127.0.97.3:32889)
0926 23:30:16.019229 (+ 1161us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:16.019235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:16.019261 (+ 26us) server_negotiation.cc:407] Connection header received
0926 23:30:16.019332 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:16.019337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:16.019402 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:16.019484 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:30:16.020165 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:16.021223 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:16.024324 (+ 3101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:16.025106 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:16.025299 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:16.026052 (+ 753us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:30:16.026221 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:30:16.026324 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:16.027232 (+ 908us) server_negotiation.cc:299] Negotiation successful
0926 23:30:16.027345 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1020,"thread_start_us":872,"threads_started":1}
I20260926 23:30:16.028352 2089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:16.018074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:32889 (local address 127.0.97.2:55429)
0926 23:30:16.018965 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:16.018989 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:16.019208 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:16.019529 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:16.019535 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:16.019551 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:30:16.019990 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:16.020002 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:16.022144 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:16.022148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:16.024121 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:16.024137 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:16.025393 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:16.025398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:16.025630 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:16.025744 (+ 114us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:30:16.027022 (+ 1278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:30:16.027043 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 23:30:16.027161 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 23:30:16.027387 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":396,"threads_started":1}
I20260926 23:30:16.079416 1512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:30:16.079712 1512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:30:16.083789 1512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P beae4f80ff99489785e3c26ed7c186cb: Sending DeleteTablet for 3 replicas of tablet fff6b1e7a60646f3a9481265388ce11e
I20260926 23:30:16.085067 1651 tablet_service.cc:1514] Processing DeleteTablet for tablet fff6b1e7a60646f3a9481265388ce11e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:30:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57684
I20260926 23:30:16.085148 1932 tablet_service.cc:1514] Processing DeleteTablet for tablet fff6b1e7a60646f3a9481265388ce11e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:30:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38240
I20260926 23:30:16.085683 1791 tablet_service.cc:1514] Processing DeleteTablet for tablet fff6b1e7a60646f3a9481265388ce11e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:30:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38006
I20260926 23:30:16.100548 2094 tablet_replica.cc:330] T fff6b1e7a60646f3a9481265388ce11e P 596576a3490444db809867705df4339d: stopping tablet replica
I20260926 23:30:16.100826 2094 raft_consensus.cc:2229] T fff6b1e7a60646f3a9481265388ce11e P 596576a3490444db809867705df4339d [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:30:16.101302 2094 raft_consensus.cc:2258] T fff6b1e7a60646f3a9481265388ce11e P 596576a3490444db809867705df4339d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:30:16.102751 2094 ts_tablet_manager.cc:1905] T fff6b1e7a60646f3a9481265388ce11e P 596576a3490444db809867705df4339d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:30:16.110809 2096 tablet_replica.cc:330] T fff6b1e7a60646f3a9481265388ce11e P c7a26f6aa4ca4af587543e3d7ad5670f: stopping tablet replica
I20260926 23:30:16.111122 2096 raft_consensus.cc:2229] T fff6b1e7a60646f3a9481265388ce11e P c7a26f6aa4ca4af587543e3d7ad5670f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:30:16.111548 2096 raft_consensus.cc:2258] T fff6b1e7a60646f3a9481265388ce11e P c7a26f6aa4ca4af587543e3d7ad5670f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:30:16.112825 2095 tablet_replica.cc:330] T fff6b1e7a60646f3a9481265388ce11e P 762530e791834c2cb07652dbde301eb1: stopping tablet replica
I20260926 23:30:16.113117 2095 raft_consensus.cc:2229] T fff6b1e7a60646f3a9481265388ce11e P 762530e791834c2cb07652dbde301eb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:30:16.113485 2094 ts_tablet_manager.cc:1918] T fff6b1e7a60646f3a9481265388ce11e P 596576a3490444db809867705df4339d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:30:16.113660 2094 log.cc:1196] T fff6b1e7a60646f3a9481265388ce11e P 596576a3490444db809867705df4339d: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-2/wal/wals/fff6b1e7a60646f3a9481265388ce11e
I20260926 23:30:16.113657 2095 raft_consensus.cc:2258] T fff6b1e7a60646f3a9481265388ce11e P 762530e791834c2cb07652dbde301eb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:30:16.115389 2094 ts_tablet_manager.cc:1939] T fff6b1e7a60646f3a9481265388ce11e P 596576a3490444db809867705df4339d: Deleting consensus metadata
I20260926 23:30:16.115924 2096 ts_tablet_manager.cc:1905] T fff6b1e7a60646f3a9481265388ce11e P c7a26f6aa4ca4af587543e3d7ad5670f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:30:16.117004 1500 catalog_manager.cc:4841] TS 596576a3490444db809867705df4339d (127.0.97.3:32889): tablet fff6b1e7a60646f3a9481265388ce11e (table test-table [id=45cbf0aaf40c4bf788e2cde9f50386fb]) successfully deleted
I20260926 23:30:16.118572 2095 ts_tablet_manager.cc:1905] T fff6b1e7a60646f3a9481265388ce11e P 762530e791834c2cb07652dbde301eb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:30:16.126794 2096 ts_tablet_manager.cc:1918] T fff6b1e7a60646f3a9481265388ce11e P c7a26f6aa4ca4af587543e3d7ad5670f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:30:16.127017 2096 log.cc:1196] T fff6b1e7a60646f3a9481265388ce11e P c7a26f6aa4ca4af587543e3d7ad5670f: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-1/wal/wals/fff6b1e7a60646f3a9481265388ce11e
I20260926 23:30:16.127473 2095 ts_tablet_manager.cc:1918] T fff6b1e7a60646f3a9481265388ce11e P 762530e791834c2cb07652dbde301eb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
Sep 26 23:30:16 dist-test-slave-gwfx krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465414, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:30:16.127524 2096 ts_tablet_manager.cc:1939] T fff6b1e7a60646f3a9481265388ce11e P c7a26f6aa4ca4af587543e3d7ad5670f: Deleting consensus metadata
I20260926 23:30:16.127635 2095 log.cc:1196] T fff6b1e7a60646f3a9481265388ce11e P 762530e791834c2cb07652dbde301eb1: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790465400694755-388-0/minicluster-data/ts-0/wal/wals/fff6b1e7a60646f3a9481265388ce11e
I20260926 23:30:16.128094 2095 ts_tablet_manager.cc:1939] T fff6b1e7a60646f3a9481265388ce11e P 762530e791834c2cb07652dbde301eb1: Deleting consensus metadata
I20260926 23:30:16.129001 1501 catalog_manager.cc:4841] TS c7a26f6aa4ca4af587543e3d7ad5670f (127.0.97.2:41071): tablet fff6b1e7a60646f3a9481265388ce11e (table test-table [id=45cbf0aaf40c4bf788e2cde9f50386fb]) successfully deleted
I20260926 23:30:16.129311 1500 catalog_manager.cc:4841] TS 762530e791834c2cb07652dbde301eb1 (127.0.97.1:45241): tablet fff6b1e7a60646f3a9481265388ce11e (table test-table [id=45cbf0aaf40c4bf788e2cde9f50386fb]) successfully deleted
I20260926 23:30:16.135336 2098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:16.096954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57700 (local address 127.0.97.1:45241)
0926 23:30:16.113989 (+ 17035us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:16.114001 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:16.114064 (+ 63us) server_negotiation.cc:407] Connection header received
0926 23:30:16.114185 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:16.114194 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:16.114528 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:16.114760 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:16.117479 (+ 2719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:16.118906 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:16.121135 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:16.122072 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:16.122215 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:16.129113 (+ 6898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:16.129132 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:16.129223 (+ 91us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:16.129278 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:16.132416 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:16.133239 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:16.133250 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:16.133253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:16.133384 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:16.133751 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:16.133757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:16.133760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:16.134144 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:16.134429 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:16.134766 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 23:30:16.134987 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16526,"thread_start_us":16335,"threads_started":1}
W20260926 23:30:16.136065 1717 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57700
I20260926 23:30:16.156062 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:16.141159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43972 (local address 127.0.97.62:42647)
0926 23:30:16.141502 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:16.141510 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:16.141791 (+ 281us) server_negotiation.cc:407] Connection header received
0926 23:30:16.142011 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:16.142018 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:16.142163 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:16.142297 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:16.143776 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:16.144945 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:16.146325 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:16.147075 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:16.147246 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:16.149789 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:16.149796 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:16.149800 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:16.149844 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:16.153127 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:16.153903 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:16.153907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:16.153910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:16.154031 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:16.154381 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:16.154385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:16.154387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:16.154742 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:16.154997 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:16.155594 (+ 597us) server_negotiation.cc:299] Negotiation successful
0926 23:30:16.155820 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
W20260926 23:30:16.156647 1512 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43972
I20260926 23:30:16.160442 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 1586
I20260926 23:30:16.174842 2098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:16.161473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57706 (local address 127.0.97.1:45241)
0926 23:30:16.161738 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:16.161746 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:16.162202 (+ 456us) server_negotiation.cc:407] Connection header received
0926 23:30:16.162377 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:16.162381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:16.162479 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:16.162562 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:16.163769 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:16.164460 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:16.165649 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:16.166071 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:16.166184 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:16.168732 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:16.168736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:16.168740 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:16.168777 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:16.171649 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:16.172391 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:16.172395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:16.172398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:16.172528 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:16.172869 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:16.172873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:16.172875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:16.173139 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:16.173346 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:16.174044 (+ 698us) server_negotiation.cc:299] Negotiation successful
0926 23:30:16.174408 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260926 23:30:16.175583 1717 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57706
W20260926 23:30:16.176254 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 1586: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:30:16.176453 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 1586
I20260926 23:30:16.192128 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 1726
Sep 26 23:30:16 dist-test-slave-gwfx krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465414, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:30:16.212771 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:16.193039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38024 (local address 127.0.97.2:41071)
0926 23:30:16.193266 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:16.193274 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:16.193290 (+ 16us) server_negotiation.cc:407] Connection header received
0926 23:30:16.193374 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:16.193378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:16.193469 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:16.193571 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:16.194754 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:16.195610 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:16.196943 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:16.197426 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:16.197549 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:16.206141 (+ 8592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:16.206158 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:16.206174 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:16.206218 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:16.209822 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:16.210559 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:16.210567 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:16.210569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:16.210741 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:16.211111 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:16.211124 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:16.211127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:16.211517 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:16.211802 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:16.212171 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 23:30:16.212410 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260926 23:30:16.213510 1857 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38024
W20260926 23:30:16.214357 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 1726: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:30:16.214545 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 1726
I20260926 23:30:16.233784 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 1866
Sep 26 23:30:16 dist-test-slave-gwfx krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465414, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:30:16.249676 2090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:16.234846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38268 (local address 127.0.97.3:32889)
0926 23:30:16.235114 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:16.235120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:16.235135 (+ 15us) server_negotiation.cc:407] Connection header received
0926 23:30:16.235186 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:16.235190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:16.235250 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:16.235343 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:16.236638 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:16.237354 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:16.238526 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:16.239097 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:16.239215 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:16.243661 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:16.243678 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:16.243696 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:16.243739 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:16.246896 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:16.247679 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:16.247685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:16.247688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:16.247836 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:16.248200 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:16.248204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:16.248206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:16.248555 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:16.248867 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:16.249118 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 23:30:16.249339 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260926 23:30:16.250633 1997 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38268
W20260926 23:30:16.251582 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 1866: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:30:16.251770 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 1866
I20260926 23:30:16.271719 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 1483
I20260926 23:30:16.273088 1555 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:30:16.404033 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 1483
2026-09-26T23:30:16Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5301 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:30:16 dist-test-slave-gwfx krb5kdc[2106](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:30:16 dist-test-slave-gwfx krb5kdc[2106](info): set up 2 sockets
Sep 26 23:30:16 dist-test-slave-gwfx krb5kdc[2106](info): commencing operation
krb5kdc: starting...
W20260926 23:30:17.419193 388 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:30:17 dist-test-slave-gwfx krb5kdc[2106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:30:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:30:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:30:17.631055 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37755
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:30:18.023594 2119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:18.024045 2119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:18.024153 2119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:18.032689 2119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:30:18.032843 2119 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 23:30:18.032894 2119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:18.032974 2119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:18.033044 2119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:30:18.033133 2119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:30:18.044703 2119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37755
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:18.046521 2119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:18.048218 2119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:18.062183 2127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:18.062183 2125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:18.062372 2124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:18.063172 2119 server_base.cc:1034] running on GCE node
I20260926 23:30:18.064162 2119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:18.065956 2119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:18.067233 2119 hybrid_clock.cc:648] HybridClock initialized: now 1790465418067137 us; error 54 us; skew 500 ppm
I20260926 23:30:18.067718 2119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:18 dist-test-slave-gwfx krb5kdc[2106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:18.071658 2119 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:30:18.073662 2119 webserver.cc:460] Webserver started at http://127.0.97.62:39241/ using document root <none> and password file <none>
I20260926 23:30:18.074568 2119 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:18.074677 2119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:18.075075 2119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:18.078438 2119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "bd9f94aebcc1490c842e1a85f4df9918"
format_stamp: "Formatted at 2026-09-26 23:30:18 on dist-test-slave-gwfx"
server_key: "12b6c0592e627c232677d723a72c04f7"
server_key_iv: "a74fa598d1b129c4f352d61da0cc31eb"
server_key_version: "encryptionkey@0"
I20260926 23:30:18.079361 2119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "bd9f94aebcc1490c842e1a85f4df9918"
format_stamp: "Formatted at 2026-09-26 23:30:18 on dist-test-slave-gwfx"
server_key: "12b6c0592e627c232677d723a72c04f7"
server_key_iv: "a74fa598d1b129c4f352d61da0cc31eb"
server_key_version: "encryptionkey@0"
I20260926 23:30:18.086674 2119 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260926 23:30:18.091811 2134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:18.094200 2119 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260926 23:30:18.094465 2119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "bd9f94aebcc1490c842e1a85f4df9918"
format_stamp: "Formatted at 2026-09-26 23:30:18 on dist-test-slave-gwfx"
server_key: "12b6c0592e627c232677d723a72c04f7"
server_key_iv: "a74fa598d1b129c4f352d61da0cc31eb"
server_key_version: "encryptionkey@0"
I20260926 23:30:18.094767 2119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:18.109656 2119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:18.114835 2119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:18.115149 2119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:18.140617 2119 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37755
I20260926 23:30:18.140694 2195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37755 every 8 connection(s)
I20260926 23:30:18.142120 2119 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:30:18.143981 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 2119
I20260926 23:30:18.144188 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:30:18.144901 388 external_mini_cluster.cc:1428] Setting key 389cea73044856090c5dfd098d062edd
I20260926 23:30:18.148239 2196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 23:30:18 dist-test-slave-gwfx krb5kdc[2106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:18.164580 2196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd9f94aebcc1490c842e1a85f4df9918: Bootstrap starting.
I20260926 23:30:18.169495 2196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd9f94aebcc1490c842e1a85f4df9918: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:18.170923 2196 log.cc:824] T 00000000000000000000000000000000 P bd9f94aebcc1490c842e1a85f4df9918: Log is configured to *not* fsync() on all Append() calls
I20260926 23:30:18.175184 2196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd9f94aebcc1490c842e1a85f4df9918: No bootstrap required, opened a new log
I20260926 23:30:18.175230 2199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:18.148697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56608 (local address 127.0.97.62:37755)
0926 23:30:18.149789 (+ 1092us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:18.149817 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:18.149881 (+ 64us) server_negotiation.cc:407] Connection header received
0926 23:30:18.152308 (+ 2427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:18.152347 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:18.153087 (+ 740us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:18.153719 (+ 632us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:18.155272 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:18.157449 (+ 2177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:18.159125 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:18.160265 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:18.160514 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:18.165152 (+ 4638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:18.165167 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:18.165203 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:18.165270 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:18.169540 (+ 4270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:18.170571 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:18.170581 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:18.170592 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:18.170866 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:18.171412 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:18.171418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:18.171422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:18.172228 (+ 806us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:18.172503 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:18.173104 (+ 601us) server_negotiation.cc:299] Negotiation successful
0926 23:30:18.173636 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":522,"threads_started":1}
I20260926 23:30:18.184098 2196 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd9f94aebcc1490c842e1a85f4df9918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9f94aebcc1490c842e1a85f4df9918" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37755 } }
I20260926 23:30:18.184592 2196 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd9f94aebcc1490c842e1a85f4df9918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:18.184675 2196 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd9f94aebcc1490c842e1a85f4df9918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd9f94aebcc1490c842e1a85f4df9918, State: Initialized, Role: FOLLOWER
I20260926 23:30:18.185465 2196 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd9f94aebcc1490c842e1a85f4df9918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9f94aebcc1490c842e1a85f4df9918" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37755 } }
I20260926 23:30:18.185751 2196 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd9f94aebcc1490c842e1a85f4df9918 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:30:18.185926 2196 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd9f94aebcc1490c842e1a85f4df9918 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:30:18.186107 2196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd9f94aebcc1490c842e1a85f4df9918 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:18.188616 2196 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd9f94aebcc1490c842e1a85f4df9918 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9f94aebcc1490c842e1a85f4df9918" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37755 } }
I20260926 23:30:18.189325 2196 leader_election.cc:302] T 00000000000000000000000000000000 P bd9f94aebcc1490c842e1a85f4df9918 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd9f94aebcc1490c842e1a85f4df9918; no voters:
I20260926 23:30:18.190071 2196 leader_election.cc:288] T 00000000000000000000000000000000 P bd9f94aebcc1490c842e1a85f4df9918 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:30:18.190332 2201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd9f94aebcc1490c842e1a85f4df9918 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:30:18.191407 2201 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd9f94aebcc1490c842e1a85f4df9918 [term 1 LEADER]: Becoming Leader. State: Replica: bd9f94aebcc1490c842e1a85f4df9918, State: Running, Role: LEADER
I20260926 23:30:18.192121 2201 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd9f94aebcc1490c842e1a85f4df9918 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9f94aebcc1490c842e1a85f4df9918" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37755 } }
I20260926 23:30:18.192866 2196 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd9f94aebcc1490c842e1a85f4df9918 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:30:18.196161 2203 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd9f94aebcc1490c842e1a85f4df9918 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd9f94aebcc1490c842e1a85f4df9918. Latest consensus state: current_term: 1 leader_uuid: "bd9f94aebcc1490c842e1a85f4df9918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9f94aebcc1490c842e1a85f4df9918" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37755 } } }
I20260926 23:30:18.196743 2203 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd9f94aebcc1490c842e1a85f4df9918 [sys.catalog]: This master's current role is: LEADER
I20260926 23:30:18.196074 2202 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd9f94aebcc1490c842e1a85f4df9918 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd9f94aebcc1490c842e1a85f4df9918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9f94aebcc1490c842e1a85f4df9918" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37755 } } }
I20260926 23:30:18.196955 2202 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd9f94aebcc1490c842e1a85f4df9918 [sys.catalog]: This master's current role is: LEADER
I20260926 23:30:18.199235 2209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:30:18.206521 2209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:30:18.219921 2209 catalog_manager.cc:1347] Generated new cluster ID: 6c1a15a694de42fca906176267ae49e2
I20260926 23:30:18.220178 2209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:30:18.236227 2209 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:30:18.238183 2209 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:30:18.249198 2209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd9f94aebcc1490c842e1a85f4df9918: Generated new TSK 0
I20260926 23:30:18.250383 2209 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:30:18.321206 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37755
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:30:18.728945 2222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:18.729331 2222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:18.729446 2222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:18.737641 2222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:18.737802 2222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:18.737897 2222 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 23:30:18.737955 2222 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 23:30:18.738034 2222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:30:18.749346 2222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:18.751084 2222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:18.752683 2222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:18.768236 2230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:18.768291 2227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:18.768291 2228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:18.769173 2222 server_base.cc:1034] running on GCE node
I20260926 23:30:18.770067 2222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:18.771314 2222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:18.772547 2222 hybrid_clock.cc:648] HybridClock initialized: now 1790465418772484 us; error 63 us; skew 500 ppm
I20260926 23:30:18.772955 2222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:18 dist-test-slave-gwfx krb5kdc[2106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:18.776634 2222 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:30:18.778532 2222 webserver.cc:460] Webserver started at http://127.0.97.1:35619/ using document root <none> and password file <none>
I20260926 23:30:18.779464 2222 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:18.779583 2222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:18.779896 2222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:18.783483 2222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-0/data/instance:
uuid: "58372f3855154f02b1652297c709ac56"
format_stamp: "Formatted at 2026-09-26 23:30:18 on dist-test-slave-gwfx"
server_key: "5724424ed71ec6323cd45f191593caf1"
server_key_iv: "7d9f5713799b5b35d914b351c91d829c"
server_key_version: "encryptionkey@0"
I20260926 23:30:18.784382 2222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-0/wal/instance:
uuid: "58372f3855154f02b1652297c709ac56"
format_stamp: "Formatted at 2026-09-26 23:30:18 on dist-test-slave-gwfx"
server_key: "5724424ed71ec6323cd45f191593caf1"
server_key_iv: "7d9f5713799b5b35d914b351c91d829c"
server_key_version: "encryptionkey@0"
I20260926 23:30:18.791257 2222 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 23:30:18.796173 2237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:18.798272 2222 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 23:30:18.798525 2222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "58372f3855154f02b1652297c709ac56"
format_stamp: "Formatted at 2026-09-26 23:30:18 on dist-test-slave-gwfx"
server_key: "5724424ed71ec6323cd45f191593caf1"
server_key_iv: "7d9f5713799b5b35d914b351c91d829c"
server_key_version: "encryptionkey@0"
I20260926 23:30:18.798906 2222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:18.824266 2222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:18.830026 2222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:18.830367 2222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:18.833712 2222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:30:18.833873 2222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:18.833994 2222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:30:18.834079 2222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 23:30:18 dist-test-slave-gwfx krb5kdc[2106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:18.860807 2246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:18.834582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35685 (local address 127.0.97.62:37755)
0926 23:30:18.835147 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:18.835152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:18.836750 (+ 1598us) server_negotiation.cc:407] Connection header received
0926 23:30:18.839136 (+ 2386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:18.839143 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:18.839257 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:18.839423 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:18.842101 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:18.843123 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:18.846604 (+ 3481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:18.847188 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:18.847386 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:18.853007 (+ 5621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:18.853018 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:18.853025 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:18.853084 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:18.856744 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:18.858038 (+ 1294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:18.858046 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:18.858050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:18.858199 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:18.858742 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:18.858748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:18.858752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:18.859077 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:18.859218 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:18.860139 (+ 921us) server_negotiation.cc:299] Negotiation successful
0926 23:30:18.860410 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":303,"threads_started":1}
I20260926 23:30:18.862389 2247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:18.835472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37755 (local address 127.0.97.1:35685)
0926 23:30:18.836547 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:18.836605 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:18.838730 (+ 2125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:18.839801 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:18.839810 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:18.840717 (+ 907us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:18.841783 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:18.841823 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:18.843356 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:18.843363 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:18.846343 (+ 2980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:18.846373 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:18.847594 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:18.847605 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:18.848003 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:18.848971 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:18.849005 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:18.852615 (+ 3610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:18.857005 (+ 4390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:18.857019 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:18.857054 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:18.857886 (+ 832us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:18.858333 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:18.858337 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:18.858339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:18.858587 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:18.859222 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:18.859233 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:18.859713 (+ 480us) client_negotiation.cc:769] Sending connection context
0926 23:30:18.860015 (+ 302us) client_negotiation.cc:240] Negotiation successful
0926 23:30:18.860690 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"thread_start_us":463,"threads_started":1}
I20260926 23:30:18.885003 2222 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33921
I20260926 23:30:18.885015 2357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33921 every 8 connection(s)
I20260926 23:30:18.886510 2246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:18.870104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43807 (local address 127.0.97.62:37755)
0926 23:30:18.870334 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:18.870341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:18.871423 (+ 1082us) server_negotiation.cc:407] Connection header received
0926 23:30:18.871649 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:18.871653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:18.871740 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:18.871913 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:18.873382 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:18.874178 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:18.876217 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:18.876703 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:18.876838 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:18.879414 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:18.879425 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:18.879430 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:18.879484 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:18.883324 (+ 3840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:18.884289 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:18.884296 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:18.884300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:18.884444 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:18.884855 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:18.884859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:18.884861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:18.885224 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:18.885372 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:18.885814 (+ 442us) server_negotiation.cc:299] Negotiation successful
0926 23:30:18.886068 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 23:30:18.886575 2328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:18.870073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37755 (local address 127.0.97.1:43807)
0926 23:30:18.871248 (+ 1175us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:18.871275 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:18.871518 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:18.871970 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:18.871975 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:18.872791 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:18.873165 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:18.873177 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:18.874359 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:18.874365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:18.876049 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:18.876066 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:18.876915 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:18.876924 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:18.877095 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:18.877945 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:18.877967 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:18.879040 (+ 1073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:18.883514 (+ 4474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:18.883519 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:18.883523 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:18.884048 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:18.884570 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:18.884577 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:18.884580 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:18.884735 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:18.885344 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:18.885349 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:18.885546 (+ 197us) client_negotiation.cc:769] Sending connection context
0926 23:30:18.885677 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 23:30:18.886091 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"thread_start_us":776,"threads_started":1}
I20260926 23:30:18.886678 2222 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:30:18.887584 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 2222
I20260926 23:30:18.887811 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:30:18.888645 388 external_mini_cluster.cc:1428] Setting key 7d0e6864fd34ec1816fe75333fb9e0db
I20260926 23:30:18.893268 2358 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37755
I20260926 23:30:18.893692 2358 heartbeater.cc:464] Registering TS with master...
I20260926 23:30:18.894930 2358 heartbeater.cc:511] Master 127.0.97.62:37755 requested a full tablet report, sending...
I20260926 23:30:18.898092 2151 ts_manager.cc:194] Registered new tserver with Master: 58372f3855154f02b1652297c709ac56 (127.0.97.1:33921)
I20260926 23:30:18.900019 2151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35685
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:30:18.939158 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37755
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:30:19.344075 2362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:19.344461 2362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:19.344607 2362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:19.352743 2362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:19.352898 2362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:19.353018 2362 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 23:30:19.353089 2362 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 23:30:19.353176 2362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:30:19.364137 2362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:19.365887 2362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:19.367570 2362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:19.384078 2368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:19.384125 2367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:19.384042 2370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:19.388831 2362 server_base.cc:1034] running on GCE node
I20260926 23:30:19.389794 2362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:19.391209 2362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:19.392449 2362 hybrid_clock.cc:648] HybridClock initialized: now 1790465419392370 us; error 81 us; skew 500 ppm
I20260926 23:30:19.392905 2362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:19 dist-test-slave-gwfx krb5kdc[2106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:19.397045 2362 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:30:19.399259 2362 webserver.cc:460] Webserver started at http://127.0.97.2:45099/ using document root <none> and password file <none>
I20260926 23:30:19.400372 2362 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:19.400559 2362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:19.400946 2362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:19.405544 2362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-1/data/instance:
uuid: "8cd8ead90d5c4ae6bb92ab20cc986875"
format_stamp: "Formatted at 2026-09-26 23:30:19 on dist-test-slave-gwfx"
server_key: "54c97bc212a1266b0864b1de13db59a2"
server_key_iv: "54520b61d672111b56a464ec0fbe1793"
server_key_version: "encryptionkey@0"
I20260926 23:30:19.406777 2362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8cd8ead90d5c4ae6bb92ab20cc986875"
format_stamp: "Formatted at 2026-09-26 23:30:19 on dist-test-slave-gwfx"
server_key: "54c97bc212a1266b0864b1de13db59a2"
server_key_iv: "54520b61d672111b56a464ec0fbe1793"
server_key_version: "encryptionkey@0"
I20260926 23:30:19.414264 2362 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260926 23:30:19.419337 2377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:19.421391 2362 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 23:30:19.421634 2362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "8cd8ead90d5c4ae6bb92ab20cc986875"
format_stamp: "Formatted at 2026-09-26 23:30:19 on dist-test-slave-gwfx"
server_key: "54c97bc212a1266b0864b1de13db59a2"
server_key_iv: "54520b61d672111b56a464ec0fbe1793"
server_key_version: "encryptionkey@0"
I20260926 23:30:19.421922 2362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:19.452013 2362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:19.457558 2362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:19.457870 2362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:19.461246 2362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:30:19.461380 2362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:19.461508 2362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:30:19.461588 2362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 23:30:19 dist-test-slave-gwfx krb5kdc[2106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:19.494378 2386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:19.462272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38477 (local address 127.0.97.62:37755)
0926 23:30:19.462856 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:19.462864 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:19.464513 (+ 1649us) server_negotiation.cc:407] Connection header received
0926 23:30:19.467730 (+ 3217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:19.467739 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:19.467865 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:19.468022 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:19.470963 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:19.472135 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:19.474977 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:19.475579 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:19.475721 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:19.482514 (+ 6793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:19.482524 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:19.482529 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:19.482593 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:19.486397 (+ 3804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:19.488117 (+ 1720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:19.488127 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:19.488131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:19.488372 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:19.489183 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:19.489191 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:19.489194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:19.489621 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:19.489817 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:19.493492 (+ 3675us) server_negotiation.cc:299] Negotiation successful
0926 23:30:19.493861 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":271,"threads_started":1}
I20260926 23:30:19.496271 2387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:19.463130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37755 (local address 127.0.97.2:38477)
0926 23:30:19.464265 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:19.464333 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:19.467284 (+ 2951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:19.468403 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:19.468414 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:19.469468 (+ 1054us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:19.470632 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:19.470678 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:19.472342 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:19.472346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:19.474774 (+ 2428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:19.474800 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:19.475935 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:19.475950 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:19.476385 (+ 435us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:19.477447 (+ 1062us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:19.477494 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:19.482010 (+ 4516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:19.486672 (+ 4662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:19.486709 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:19.486741 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:19.487833 (+ 1092us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:19.488560 (+ 727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:19.488565 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:19.488572 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:19.489002 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:19.489830 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:19.489842 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:19.490721 (+ 879us) client_negotiation.cc:769] Sending connection context
0926 23:30:19.491286 (+ 565us) client_negotiation.cc:240] Negotiation successful
0926 23:30:19.492113 (+ 827us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":443,"threads_started":1}
I20260926 23:30:19.501327 2362 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46071
I20260926 23:30:19.501379 2493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46071 every 8 connection(s)
I20260926 23:30:19.503247 2362 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:30:19.504459 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 2362
I20260926 23:30:19.504657 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:30:19.505494 388 external_mini_cluster.cc:1428] Setting key 7ee351e8388b0c41224e9bf439f17388
I20260926 23:30:19.506577 2497 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37755
I20260926 23:30:19.508046 2497 heartbeater.cc:464] Registering TS with master...
I20260926 23:30:19.509157 2497 heartbeater.cc:511] Master 127.0.97.62:37755 requested a full tablet report, sending...
I20260926 23:30:19.510933 2151 ts_manager.cc:194] Registered new tserver with Master: 8cd8ead90d5c4ae6bb92ab20cc986875 (127.0.97.2:46071)
I20260926 23:30:19.512130 2151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38477
I20260926 23:30:19.522271 2386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:19.507177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34433 (local address 127.0.97.62:37755)
0926 23:30:19.507376 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:19.507382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:19.507978 (+ 596us) server_negotiation.cc:407] Connection header received
0926 23:30:19.508163 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:19.508169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:19.508266 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:19.508374 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:19.509682 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:19.510787 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:19.512747 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:19.513316 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:19.513438 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:19.515916 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:19.515921 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:19.515925 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:19.515970 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:19.519092 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:19.520121 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:19.520126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:19.520128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:19.520280 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:19.520669 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:19.520672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:19.520674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:19.521067 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:19.521193 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:19.521706 (+ 513us) server_negotiation.cc:299] Negotiation successful
0926 23:30:19.522000 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 23:30:19.522480 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:19.507164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37755 (local address 127.0.97.2:34433)
0926 23:30:19.507850 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:19.507871 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:19.508032 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:19.508420 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:19.508425 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:19.509145 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:19.509512 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:19.509529 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:19.510936 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:19.510941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:19.512423 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:19.512438 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:19.513523 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:19.513529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:19.513707 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:19.514488 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:19.514512 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:19.515663 (+ 1151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:19.519312 (+ 3649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:19.519319 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:19.519323 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:19.519942 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:19.520397 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:19.520401 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:19.520404 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:19.520565 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:19.521192 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:19.521197 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:19.521421 (+ 224us) client_negotiation.cc:769] Sending connection context
0926 23:30:19.521578 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 23:30:19.521964 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":357,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:30:19.554569 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37755
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:30:19.904338 2358 heartbeater.cc:503] Master 127.0.97.62:37755 was elected leader, sending a full tablet report...
W20260926 23:30:20.001888 2502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:20.002324 2502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:20.002499 2502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:20.013443 2502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:20.013682 2502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:20.013827 2502 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 23:30:20.013904 2502 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 23:30:20.014017 2502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:30:20.031291 2502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:20.033296 2502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:20.035271 2502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:20.054011 2508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:20.054478 2507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:20.055094 2510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:20.055673 2502 server_base.cc:1034] running on GCE node
I20260926 23:30:20.056490 2502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:20.057768 2502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:20.061369 2502 hybrid_clock.cc:648] HybridClock initialized: now 1790465420061280 us; error 58 us; skew 500 ppm
I20260926 23:30:20.061975 2502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:20 dist-test-slave-gwfx krb5kdc[2106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:20.066811 2502 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:30:20.069559 2502 webserver.cc:460] Webserver started at http://127.0.97.3:34377/ using document root <none> and password file <none>
I20260926 23:30:20.070634 2502 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:20.070880 2502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:20.071274 2502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:20.075556 2502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-2/data/instance:
uuid: "b45c671f77264096a1fbfa9e16df34dc"
format_stamp: "Formatted at 2026-09-26 23:30:20 on dist-test-slave-gwfx"
server_key: "6e889e5af2e0643f59d0b473a5dbb71b"
server_key_iv: "f9def8bcd6eee1b08c8e14a2177f7c38"
server_key_version: "encryptionkey@0"
I20260926 23:30:20.076900 2502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b45c671f77264096a1fbfa9e16df34dc"
format_stamp: "Formatted at 2026-09-26 23:30:20 on dist-test-slave-gwfx"
server_key: "6e889e5af2e0643f59d0b473a5dbb71b"
server_key_iv: "f9def8bcd6eee1b08c8e14a2177f7c38"
server_key_version: "encryptionkey@0"
I20260926 23:30:20.087141 2502 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.007s sys 0.004s
I20260926 23:30:20.094000 2517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:20.097244 2502 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260926 23:30:20.097533 2502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "b45c671f77264096a1fbfa9e16df34dc"
format_stamp: "Formatted at 2026-09-26 23:30:20 on dist-test-slave-gwfx"
server_key: "6e889e5af2e0643f59d0b473a5dbb71b"
server_key_iv: "f9def8bcd6eee1b08c8e14a2177f7c38"
server_key_version: "encryptionkey@0"
I20260926 23:30:20.097844 2502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:20.121771 2502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:20.128568 2502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:20.128964 2502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:20.133622 2502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:30:20.133774 2502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:20.133909 2502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:30:20.134016 2502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 23:30:20 dist-test-slave-gwfx krb5kdc[2106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:20.178480 2527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:20.135634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37755 (local address 127.0.97.3:44627)
0926 23:30:20.136758 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:20.136812 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:20.139109 (+ 2297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:20.141038 (+ 1929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:20.141061 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:20.142970 (+ 1909us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:20.144286 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:20.144329 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.146324 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.146332 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:20.150778 (+ 4446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:20.150825 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.153334 (+ 2509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.153346 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:20.153840 (+ 494us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:20.156239 (+ 2399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:20.156319 (+ 80us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:20.164903 (+ 8584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:20.170551 (+ 5648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:20.170566 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:20.170592 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:20.172008 (+ 1416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:20.172832 (+ 824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:20.172838 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:20.172841 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:20.173200 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:20.174544 (+ 1344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:20.174556 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:20.175189 (+ 633us) client_negotiation.cc:769] Sending connection context
0926 23:30:20.175569 (+ 380us) client_negotiation.cc:240] Negotiation successful
0926 23:30:20.176314 (+ 745us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":462,"threads_started":1}
I20260926 23:30:20.178942 2526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:20.134813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44627 (local address 127.0.97.62:37755)
0926 23:30:20.135435 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:20.135443 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:20.137025 (+ 1582us) server_negotiation.cc:407] Connection header received
0926 23:30:20.139809 (+ 2784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:20.139818 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:20.139973 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:20.140096 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:20.144663 (+ 4567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.145988 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.151184 (+ 5196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.152775 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.153135 (+ 360us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:20.165476 (+ 12341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:20.165485 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:20.165491 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:20.165560 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:20.170034 (+ 4474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:20.172309 (+ 2275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:20.172320 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:20.172325 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:20.172612 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:20.173657 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:20.173666 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:20.173670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:20.174236 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:20.174446 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:20.177934 (+ 3488us) server_negotiation.cc:299] Negotiation successful
0926 23:30:20.178343 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":275,"threads_started":1}
I20260926 23:30:20.208930 2600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:20.187270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37755 (local address 127.0.97.3:52767)
0926 23:30:20.188374 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:20.188396 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:20.188565 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:20.189227 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:20.189233 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:20.189727 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:20.190172 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:20.190185 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.191965 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.191972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:20.193757 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:20.193778 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.195415 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.195424 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:20.195610 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:20.196644 (+ 1034us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:20.196669 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:20.197813 (+ 1144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:20.205034 (+ 7221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:20.205052 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:20.205055 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:20.205770 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:20.206498 (+ 728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:20.206504 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:20.206513 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:20.206810 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:20.207482 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:20.207487 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:20.207691 (+ 204us) client_negotiation.cc:769] Sending connection context
0926 23:30:20.207915 (+ 224us) client_negotiation.cc:240] Negotiation successful
0926 23:30:20.208367 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"thread_start_us":458,"threads_started":1}
I20260926 23:30:20.209271 2526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:20.187031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52767 (local address 127.0.97.62:37755)
0926 23:30:20.187278 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:20.187284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:20.188562 (+ 1278us) server_negotiation.cc:407] Connection header received
0926 23:30:20.188768 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:20.188777 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:20.188918 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:20.189062 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:20.190443 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.191747 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.194007 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.195064 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.195331 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:20.198224 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:20.198234 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:20.198241 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:20.198378 (+ 137us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:20.204781 (+ 6403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:20.205979 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:20.205986 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:20.205990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:20.206191 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:20.206990 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:20.206996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:20.206999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:20.207334 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:20.207469 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:20.208192 (+ 723us) server_negotiation.cc:299] Negotiation successful
0926 23:30:20.208784 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260926 23:30:20.215894 2502 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36383
I20260926 23:30:20.215929 2637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36383 every 8 connection(s)
I20260926 23:30:20.217621 2502 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:30:20.220978 2638 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37755
I20260926 23:30:20.221354 2638 heartbeater.cc:464] Registering TS with master...
I20260926 23:30:20.222285 2638 heartbeater.cc:511] Master 127.0.97.62:37755 requested a full tablet report, sending...
I20260926 23:30:20.224404 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 2502
I20260926 23:30:20.224593 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:30:20.224956 2151 ts_manager.cc:194] Registered new tserver with Master: b45c671f77264096a1fbfa9e16df34dc (127.0.97.3:36383)
I20260926 23:30:20.225075 388 external_mini_cluster.cc:1428] Setting key 44a2b470d8ca4e1573fa9e598ff19d31
I20260926 23:30:20.226094 2151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44627
I20260926 23:30:20.229831 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 23:30:20 dist-test-slave-gwfx krb5kdc[2106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 23:30:20 dist-test-slave-gwfx krb5kdc[2106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:20.284433 2526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:20.268478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56610 (local address 127.0.97.62:37755)
0926 23:30:20.268750 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:20.268757 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:20.269018 (+ 261us) server_negotiation.cc:407] Connection header received
0926 23:30:20.269273 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:20.269279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:20.269358 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:20.269530 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:20.271277 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.272114 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.273599 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.274067 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.274385 (+ 318us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:20.278879 (+ 4494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:20.278886 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:20.278890 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:20.278941 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:20.281539 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:20.282417 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:20.282422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:20.282424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:20.282569 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:20.283017 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:20.283022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:20.283024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:20.283313 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:20.283569 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:20.283861 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 23:30:20.284091 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260926 23:30:20.290370 2150 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:30:20.298511 2150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:30:20.330801 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:20.315591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33052 (local address 127.0.97.2:46071)
0926 23:30:20.319130 (+ 3539us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:20.319140 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:20.319179 (+ 39us) server_negotiation.cc:407] Connection header received
0926 23:30:20.319338 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:20.319348 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:20.319759 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:20.321090 (+ 1331us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:30:20.321448 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.323264 (+ 1816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.327314 (+ 4050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.329148 (+ 1834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.329335 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:20.329583 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:20.330165 (+ 582us) server_negotiation.cc:299] Negotiation successful
0926 23:30:20.330379 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3303,"thread_start_us":3122,"threads_started":1}
I20260926 23:30:20.330909 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:20.315378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46071 (local address 127.0.0.1:33052)
0926 23:30:20.316220 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:20.316283 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:20.316559 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:20.320077 (+ 3518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:20.320089 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:20.320138 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:30:20.321049 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:20.321068 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.323522 (+ 2454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.323529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:20.326970 (+ 3441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:20.326997 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.329444 (+ 2447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.329451 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:20.329817 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:20.329894 (+ 77us) client_negotiation.cc:769] Sending connection context
0926 23:30:20.330093 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 23:30:20.330362 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":79,"thread_start_us":1449,"threads_started":1}
I20260926 23:30:20.332109 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:20.317158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33921 (local address 127.0.0.1:38160)
0926 23:30:20.321508 (+ 4350us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:20.321534 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:20.321674 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:20.322973 (+ 1299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:20.322978 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:20.323011 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:30:20.324413 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:20.324429 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.326172 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.326179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:20.328628 (+ 2449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:20.328645 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.331125 (+ 2480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.331131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:20.331318 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:20.331354 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 23:30:20.331440 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 23:30:20.331606 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4181,"mutex_wait_us":76,"thread_start_us":6614,"threads_started":1}
I20260926 23:30:20.332962 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:20.318025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38160 (local address 127.0.97.1:33921)
0926 23:30:20.321587 (+ 3562us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:20.321597 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:20.321646 (+ 49us) server_negotiation.cc:407] Connection header received
0926 23:30:20.321956 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:20.321969 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:20.322314 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:20.322597 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:30:20.324669 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.326026 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.328884 (+ 2858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.330533 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.330715 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:20.330982 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:20.332427 (+ 1445us) server_negotiation.cc:299] Negotiation successful
0926 23:30:20.332645 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3123,"thread_start_us":2750,"threads_started":1}
I20260926 23:30:20.333752 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:20.317860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47524 (local address 127.0.97.3:36383)
0926 23:30:20.320230 (+ 2370us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:20.320242 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:20.323364 (+ 3122us) server_negotiation.cc:407] Connection header received
0926 23:30:20.323531 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:20.323541 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:20.324103 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:20.324410 (+ 307us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:30:20.325452 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.327178 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.330038 (+ 2860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.331619 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.331757 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:20.332130 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:20.333213 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0926 23:30:20.333408 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2125,"thread_start_us":1856,"threads_started":1}
I20260926 23:30:20.334352 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:20.317202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36383 (local address 127.0.0.1:47524)
0926 23:30:20.322481 (+ 5279us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:20.322523 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:20.322908 (+ 385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:20.324556 (+ 1648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:20.324561 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:20.324596 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:30:20.325176 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:20.325191 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.327452 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.327457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:20.329832 (+ 2375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:20.329850 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.331890 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.331898 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:20.332580 (+ 682us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:20.332626 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 23:30:20.332745 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 23:30:20.332918 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4985,"mutex_wait_us":33,"thread_start_us":3473,"threads_started":1}
I20260926 23:30:20.336889 2428 tablet_service.cc:1467] Processing CreateTablet for tablet fafb70ef86bc4c67ba8cc3c7be3e2493 (DEFAULT_TABLE table=test-table [id=548eb5203f9f4e5bb886d3c1c34a5707]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:30:20.337559 2288 tablet_service.cc:1467] Processing CreateTablet for tablet fafb70ef86bc4c67ba8cc3c7be3e2493 (DEFAULT_TABLE table=test-table [id=548eb5203f9f4e5bb886d3c1c34a5707]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:30:20.339072 2428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fafb70ef86bc4c67ba8cc3c7be3e2493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:20.339239 2288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fafb70ef86bc4c67ba8cc3c7be3e2493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:20.339237 2568 tablet_service.cc:1467] Processing CreateTablet for tablet fafb70ef86bc4c67ba8cc3c7be3e2493 (DEFAULT_TABLE table=test-table [id=548eb5203f9f4e5bb886d3c1c34a5707]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:30:20.341084 2568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fafb70ef86bc4c67ba8cc3c7be3e2493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:20.352641 2657 tablet_bootstrap.cc:492] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56: Bootstrap starting.
I20260926 23:30:20.356954 2657 tablet_bootstrap.cc:654] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:20.358827 2657 log.cc:824] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56: Log is configured to *not* fsync() on all Append() calls
I20260926 23:30:20.359825 2659 tablet_bootstrap.cc:492] T fafb70ef86bc4c67ba8cc3c7be3e2493 P b45c671f77264096a1fbfa9e16df34dc: Bootstrap starting.
I20260926 23:30:20.360971 2658 tablet_bootstrap.cc:492] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 8cd8ead90d5c4ae6bb92ab20cc986875: Bootstrap starting.
I20260926 23:30:20.364776 2658 tablet_bootstrap.cc:654] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 8cd8ead90d5c4ae6bb92ab20cc986875: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:20.365087 2659 tablet_bootstrap.cc:654] T fafb70ef86bc4c67ba8cc3c7be3e2493 P b45c671f77264096a1fbfa9e16df34dc: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:20.366410 2658 log.cc:824] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 8cd8ead90d5c4ae6bb92ab20cc986875: Log is configured to *not* fsync() on all Append() calls
I20260926 23:30:20.366632 2657 tablet_bootstrap.cc:492] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56: No bootstrap required, opened a new log
I20260926 23:30:20.367007 2657 ts_tablet_manager.cc:1397] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s
I20260926 23:30:20.367339 2659 log.cc:824] T fafb70ef86bc4c67ba8cc3c7be3e2493 P b45c671f77264096a1fbfa9e16df34dc: Log is configured to *not* fsync() on all Append() calls
I20260926 23:30:20.369910 2658 tablet_bootstrap.cc:492] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 8cd8ead90d5c4ae6bb92ab20cc986875: No bootstrap required, opened a new log
I20260926 23:30:20.370244 2658 ts_tablet_manager.cc:1397] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 8cd8ead90d5c4ae6bb92ab20cc986875: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260926 23:30:20.371001 2659 tablet_bootstrap.cc:492] T fafb70ef86bc4c67ba8cc3c7be3e2493 P b45c671f77264096a1fbfa9e16df34dc: No bootstrap required, opened a new log
I20260926 23:30:20.371356 2659 ts_tablet_manager.cc:1397] T fafb70ef86bc4c67ba8cc3c7be3e2493 P b45c671f77264096a1fbfa9e16df34dc: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260926 23:30:20.377502 2657 raft_consensus.cc:348] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:20.378048 2657 raft_consensus.cc:374] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:20.378161 2657 raft_consensus.cc:729] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58372f3855154f02b1652297c709ac56, State: Initialized, Role: FOLLOWER
I20260926 23:30:20.378965 2657 consensus_queue.cc:260] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:20.378997 2658 raft_consensus.cc:348] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:20.379559 2658 raft_consensus.cc:374] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:20.379664 2658 raft_consensus.cc:729] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cd8ead90d5c4ae6bb92ab20cc986875, State: Initialized, Role: FOLLOWER
I20260926 23:30:20.380507 2658 consensus_queue.cc:260] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 8cd8ead90d5c4ae6bb92ab20cc986875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:20.381470 2657 ts_tablet_manager.cc:1428] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260926 23:30:20.383019 2497 heartbeater.cc:503] Master 127.0.97.62:37755 was elected leader, sending a full tablet report...
I20260926 23:30:20.382790 2659 raft_consensus.cc:348] T fafb70ef86bc4c67ba8cc3c7be3e2493 P b45c671f77264096a1fbfa9e16df34dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:20.383270 2659 raft_consensus.cc:374] T fafb70ef86bc4c67ba8cc3c7be3e2493 P b45c671f77264096a1fbfa9e16df34dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:20.383360 2659 raft_consensus.cc:729] T fafb70ef86bc4c67ba8cc3c7be3e2493 P b45c671f77264096a1fbfa9e16df34dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45c671f77264096a1fbfa9e16df34dc, State: Initialized, Role: FOLLOWER
I20260926 23:30:20.383766 2658 ts_tablet_manager.cc:1428] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 8cd8ead90d5c4ae6bb92ab20cc986875: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260926 23:30:20.384135 2659 consensus_queue.cc:260] T fafb70ef86bc4c67ba8cc3c7be3e2493 P b45c671f77264096a1fbfa9e16df34dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:20.385567 2638 heartbeater.cc:503] Master 127.0.97.62:37755 was elected leader, sending a full tablet report...
I20260926 23:30:20.386365 2659 ts_tablet_manager.cc:1428] T fafb70ef86bc4c67ba8cc3c7be3e2493 P b45c671f77264096a1fbfa9e16df34dc: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
W20260926 23:30:20.397464 2359 tablet.cc:2369] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:30:20.427597 2663 raft_consensus.cc:482] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:30:20.427958 2663 raft_consensus.cc:504] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:20.429826 2663 leader_election.cc:288] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8cd8ead90d5c4ae6bb92ab20cc986875 (127.0.97.2:46071), b45c671f77264096a1fbfa9e16df34dc (127.0.97.3:36383)
I20260926 23:30:20.437690 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:20.430415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38977 (local address 127.0.97.2:46071)
0926 23:30:20.430669 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:20.430677 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:20.431706 (+ 1029us) server_negotiation.cc:407] Connection header received
0926 23:30:20.431887 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:20.431892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:20.431966 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:20.432075 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:30:20.432870 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.433759 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.435805 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.436588 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.436710 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:20.436827 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:20.437241 (+ 414us) server_negotiation.cc:299] Negotiation successful
0926 23:30:20.437380 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 23:30:20.437702 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:20.430414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44527 (local address 127.0.97.3:36383)
0926 23:30:20.430652 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:20.430659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:20.431591 (+ 932us) server_negotiation.cc:407] Connection header received
0926 23:30:20.431814 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:20.431821 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:20.431909 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:20.431986 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:30:20.432729 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.433539 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.435790 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.436617 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.436751 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:20.436884 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:20.437253 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 23:30:20.437397 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 23:30:20.438050 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:20.430237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36383 (local address 127.0.97.1:44527)
0926 23:30:20.431443 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:20.431461 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:20.431622 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:20.432076 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:20.432081 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:20.432108 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:30:20.432563 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:20.432574 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.433712 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.433718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:20.435587 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:20.435606 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.436885 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.436890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:20.437081 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:20.437114 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 23:30:20.437207 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 23:30:20.437380 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1045,"mutex_wait_us":66,"thread_start_us":799,"threads_started":1}
I20260926 23:30:20.438284 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:20.430308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46071 (local address 127.0.97.1:38977)
0926 23:30:20.431520 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:20.431549 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:20.431740 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:20.432112 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:20.432116 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:20.432141 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:30:20.432673 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:20.432684 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.433948 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.433952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:20.435628 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:20.435641 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.436845 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.436852 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:20.437015 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:20.437097 (+ 82us) client_negotiation.cc:769] Sending connection context
0926 23:30:20.437183 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 23:30:20.437370 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1085,"mutex_wait_us":82,"thread_start_us":676,"threads_started":1}
I20260926 23:30:20.439438 2588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fafb70ef86bc4c67ba8cc3c7be3e2493" candidate_uuid: "58372f3855154f02b1652297c709ac56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b45c671f77264096a1fbfa9e16df34dc" is_pre_election: true
I20260926 23:30:20.439640 2448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fafb70ef86bc4c67ba8cc3c7be3e2493" candidate_uuid: "58372f3855154f02b1652297c709ac56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" is_pre_election: true
I20260926 23:30:20.440048 2588 raft_consensus.cc:2454] T fafb70ef86bc4c67ba8cc3c7be3e2493 P b45c671f77264096a1fbfa9e16df34dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58372f3855154f02b1652297c709ac56 in term 0.
I20260926 23:30:20.440230 2448 raft_consensus.cc:2454] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58372f3855154f02b1652297c709ac56 in term 0.
I20260926 23:30:20.440882 2238 leader_election.cc:302] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58372f3855154f02b1652297c709ac56, b45c671f77264096a1fbfa9e16df34dc; no voters:
I20260926 23:30:20.441357 2663 raft_consensus.cc:2781] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:30:20.441504 2663 raft_consensus.cc:482] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:30:20.441664 2663 raft_consensus.cc:3037] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:20.444010 2663 raft_consensus.cc:504] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:20.444741 2663 leader_election.cc:288] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56 [CANDIDATE]: Term 1 election: Requested vote from peers 8cd8ead90d5c4ae6bb92ab20cc986875 (127.0.97.2:46071), b45c671f77264096a1fbfa9e16df34dc (127.0.97.3:36383)
I20260926 23:30:20.445389 2448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fafb70ef86bc4c67ba8cc3c7be3e2493" candidate_uuid: "58372f3855154f02b1652297c709ac56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875"
I20260926 23:30:20.445497 2588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fafb70ef86bc4c67ba8cc3c7be3e2493" candidate_uuid: "58372f3855154f02b1652297c709ac56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b45c671f77264096a1fbfa9e16df34dc"
I20260926 23:30:20.445624 2448 raft_consensus.cc:3037] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:20.445756 2588 raft_consensus.cc:3037] T fafb70ef86bc4c67ba8cc3c7be3e2493 P b45c671f77264096a1fbfa9e16df34dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:20.448359 2448 raft_consensus.cc:2454] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58372f3855154f02b1652297c709ac56 in term 1.
I20260926 23:30:20.448617 2588 raft_consensus.cc:2454] T fafb70ef86bc4c67ba8cc3c7be3e2493 P b45c671f77264096a1fbfa9e16df34dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58372f3855154f02b1652297c709ac56 in term 1.
I20260926 23:30:20.449003 2239 leader_election.cc:302] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58372f3855154f02b1652297c709ac56, 8cd8ead90d5c4ae6bb92ab20cc986875; no voters:
I20260926 23:30:20.449409 2663 raft_consensus.cc:2781] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:30:20.449962 2663 raft_consensus.cc:686] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56 [term 1 LEADER]: Becoming Leader. State: Replica: 58372f3855154f02b1652297c709ac56, State: Running, Role: LEADER
I20260926 23:30:20.450485 2663 consensus_queue.cc:237] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:20.455966 2151 catalog_manager.cc:5495] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58372f3855154f02b1652297c709ac56 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "58372f3855154f02b1652297c709ac56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } health_report { overall_health: UNKNOWN } } }
W20260926 23:30:20.469889 2639 tablet.cc:2369] T fafb70ef86bc4c67ba8cc3c7be3e2493 P b45c671f77264096a1fbfa9e16df34dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:30:20.505965 2498 tablet.cc:2369] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 8cd8ead90d5c4ae6bb92ab20cc986875: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 23:30:20 dist-test-slave-gwfx krb5kdc[2106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:20.526646 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:20.510065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37755 (local address 127.0.0.1:56614)
0926 23:30:20.510813 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:20.510835 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:20.510978 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:20.511456 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:20.511460 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:20.512200 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:20.512656 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:20.512671 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.514311 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.514316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:20.515402 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:20.515417 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.516269 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.516274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:20.516403 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:20.517320 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:20.517341 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:20.520090 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:20.523622 (+ 3532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:20.523641 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:20.523648 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:20.524211 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:20.524736 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:20.524741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:20.524744 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:20.524932 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:20.525626 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:20.525633 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:20.525822 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 23:30:20.525984 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 23:30:20.526246 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":382,"threads_started":1}
I20260926 23:30:20.526899 2526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:20.510065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56614 (local address 127.0.97.62:37755)
0926 23:30:20.510307 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:20.510314 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:20.510932 (+ 618us) server_negotiation.cc:407] Connection header received
0926 23:30:20.511134 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:20.511140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:20.511229 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:20.511390 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:20.512870 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.514097 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.515594 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.516074 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.516195 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:20.520401 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:20.520409 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:20.520414 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:20.520459 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:20.523363 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:20.524399 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:20.524404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:20.524407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:20.524558 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:20.525086 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:20.525090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:20.525093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:20.525420 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:20.525550 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:20.526121 (+ 571us) server_negotiation.cc:299] Negotiation successful
0926 23:30:20.526378 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":78}
I20260926 23:30:20.533598 2151 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56614:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 23:30:20.534875 2151 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:30:20.543370 2288 tablet_service.cc:1467] Processing CreateTablet for tablet f861c5ab8c07406ea3b06f0a90612deb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6b11703f2ba3419a8d82fbb775344b08]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 23:30:20.543506 2428 tablet_service.cc:1467] Processing CreateTablet for tablet f861c5ab8c07406ea3b06f0a90612deb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6b11703f2ba3419a8d82fbb775344b08]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 23:30:20.543753 2568 tablet_service.cc:1467] Processing CreateTablet for tablet f861c5ab8c07406ea3b06f0a90612deb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6b11703f2ba3419a8d82fbb775344b08]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 23:30:20.544708 2428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f861c5ab8c07406ea3b06f0a90612deb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:20.544709 2288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f861c5ab8c07406ea3b06f0a90612deb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:20.544880 2568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f861c5ab8c07406ea3b06f0a90612deb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:20.551348 2659 tablet_bootstrap.cc:492] T f861c5ab8c07406ea3b06f0a90612deb P b45c671f77264096a1fbfa9e16df34dc: Bootstrap starting.
I20260926 23:30:20.552882 2657 tablet_bootstrap.cc:492] T f861c5ab8c07406ea3b06f0a90612deb P 58372f3855154f02b1652297c709ac56: Bootstrap starting.
I20260926 23:30:20.552882 2658 tablet_bootstrap.cc:492] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875: Bootstrap starting.
I20260926 23:30:20.553089 2659 tablet_bootstrap.cc:654] T f861c5ab8c07406ea3b06f0a90612deb P b45c671f77264096a1fbfa9e16df34dc: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:20.554812 2658 tablet_bootstrap.cc:654] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:20.554812 2657 tablet_bootstrap.cc:654] T f861c5ab8c07406ea3b06f0a90612deb P 58372f3855154f02b1652297c709ac56: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:20.555467 2659 tablet_bootstrap.cc:492] T f861c5ab8c07406ea3b06f0a90612deb P b45c671f77264096a1fbfa9e16df34dc: No bootstrap required, opened a new log
I20260926 23:30:20.555640 2659 ts_tablet_manager.cc:1397] T f861c5ab8c07406ea3b06f0a90612deb P b45c671f77264096a1fbfa9e16df34dc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 23:30:20.556450 2659 raft_consensus.cc:348] T f861c5ab8c07406ea3b06f0a90612deb P b45c671f77264096a1fbfa9e16df34dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:20.556720 2659 raft_consensus.cc:374] T f861c5ab8c07406ea3b06f0a90612deb P b45c671f77264096a1fbfa9e16df34dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:20.556821 2659 raft_consensus.cc:729] T f861c5ab8c07406ea3b06f0a90612deb P b45c671f77264096a1fbfa9e16df34dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45c671f77264096a1fbfa9e16df34dc, State: Initialized, Role: FOLLOWER
I20260926 23:30:20.557057 2659 consensus_queue.cc:260] T f861c5ab8c07406ea3b06f0a90612deb P b45c671f77264096a1fbfa9e16df34dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:20.557464 2657 tablet_bootstrap.cc:492] T f861c5ab8c07406ea3b06f0a90612deb P 58372f3855154f02b1652297c709ac56: No bootstrap required, opened a new log
I20260926 23:30:20.557622 2659 ts_tablet_manager.cc:1428] T f861c5ab8c07406ea3b06f0a90612deb P b45c671f77264096a1fbfa9e16df34dc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 23:30:20.557631 2657 ts_tablet_manager.cc:1397] T f861c5ab8c07406ea3b06f0a90612deb P 58372f3855154f02b1652297c709ac56: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 23:30:20.557497 2665 tablet_replica.cc:439] T f861c5ab8c07406ea3b06f0a90612deb P b45c671f77264096a1fbfa9e16df34dc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } } }
I20260926 23:30:20.557894 2665 tablet_replica.cc:442] T f861c5ab8c07406ea3b06f0a90612deb P b45c671f77264096a1fbfa9e16df34dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:30:20.558529 2657 raft_consensus.cc:348] T f861c5ab8c07406ea3b06f0a90612deb P 58372f3855154f02b1652297c709ac56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:20.558841 2657 raft_consensus.cc:374] T f861c5ab8c07406ea3b06f0a90612deb P 58372f3855154f02b1652297c709ac56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:20.558936 2657 raft_consensus.cc:729] T f861c5ab8c07406ea3b06f0a90612deb P 58372f3855154f02b1652297c709ac56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58372f3855154f02b1652297c709ac56, State: Initialized, Role: FOLLOWER
I20260926 23:30:20.559190 2657 consensus_queue.cc:260] T f861c5ab8c07406ea3b06f0a90612deb P 58372f3855154f02b1652297c709ac56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:20.559689 2663 tablet_replica.cc:439] T f861c5ab8c07406ea3b06f0a90612deb P 58372f3855154f02b1652297c709ac56: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } } }
I20260926 23:30:20.560055 2663 tablet_replica.cc:442] T f861c5ab8c07406ea3b06f0a90612deb P 58372f3855154f02b1652297c709ac56: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:30:20.560797 2657 ts_tablet_manager.cc:1428] T f861c5ab8c07406ea3b06f0a90612deb P 58372f3855154f02b1652297c709ac56: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 23:30:20.562134 2658 tablet_bootstrap.cc:492] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875: No bootstrap required, opened a new log
I20260926 23:30:20.562506 2658 ts_tablet_manager.cc:1397] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260926 23:30:20.563313 2658 raft_consensus.cc:348] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:20.563558 2658 raft_consensus.cc:374] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:20.563673 2658 raft_consensus.cc:729] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cd8ead90d5c4ae6bb92ab20cc986875, State: Initialized, Role: FOLLOWER
I20260926 23:30:20.564005 2658 consensus_queue.cc:260] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:20.564496 2664 tablet_replica.cc:439] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } } }
I20260926 23:30:20.564862 2664 tablet_replica.cc:442] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:30:20.565299 2658 ts_tablet_manager.cc:1428] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 23:30:20.653754 2664 raft_consensus.cc:482] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:30:20.654058 2664 raft_consensus.cc:504] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:20.656080 2664 leader_election.cc:288] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58372f3855154f02b1652297c709ac56 (127.0.97.1:33921), b45c671f77264096a1fbfa9e16df34dc (127.0.97.3:36383)
I20260926 23:30:20.664366 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:20.656783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38777 (local address 127.0.97.3:36383)
0926 23:30:20.657102 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:20.657112 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:20.658258 (+ 1146us) server_negotiation.cc:407] Connection header received
0926 23:30:20.658461 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:20.658466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:20.658532 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:20.658608 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:30:20.659379 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.660133 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.662300 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.663300 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.663435 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:20.663551 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:20.663926 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 23:30:20.664065 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260926 23:30:20.664386 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:20.656806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56799 (local address 127.0.97.1:33921)
0926 23:30:20.657070 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:20.657079 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:20.658116 (+ 1037us) server_negotiation.cc:407] Connection header received
0926 23:30:20.658371 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:20.658376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:20.658458 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:20.658537 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:30:20.659270 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.660103 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.662359 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.663338 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.663473 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:20.663607 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:20.663953 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 23:30:20.664106 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 23:30:20.664783 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:20.656538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33921 (local address 127.0.97.2:56799)
0926 23:30:20.657978 (+ 1440us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:20.658002 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:20.658202 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:20.658575 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:20.658580 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:20.658608 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:30:20.659077 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:20.659092 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.660247 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.660253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:20.662107 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:20.662126 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.663637 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.663644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:20.663794 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:20.663819 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 23:30:20.663909 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 23:30:20.664072 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1306,"mutex_wait_us":5,"thread_start_us":1081,"threads_started":1}
I20260926 23:30:20.664799 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:20.656762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36383 (local address 127.0.97.2:38777)
0926 23:30:20.658107 (+ 1345us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:20.658127 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:20.658315 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:20.658645 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:20.658649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:20.658717 (+ 68us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:30:20.659213 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:20.659224 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.660271 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.660275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:20.662120 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:20.662138 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.663589 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.663596 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:20.663745 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:20.663775 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 23:30:20.663867 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 23:30:20.664030 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1189,"mutex_wait_us":88,"thread_start_us":628,"threads_started":1}
I20260926 23:30:20.665881 2588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f861c5ab8c07406ea3b06f0a90612deb" candidate_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b45c671f77264096a1fbfa9e16df34dc" is_pre_election: true
I20260926 23:30:20.666199 2588 raft_consensus.cc:2454] T f861c5ab8c07406ea3b06f0a90612deb P b45c671f77264096a1fbfa9e16df34dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cd8ead90d5c4ae6bb92ab20cc986875 in term 0.
I20260926 23:30:20.666218 2308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f861c5ab8c07406ea3b06f0a90612deb" candidate_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58372f3855154f02b1652297c709ac56" is_pre_election: true
I20260926 23:30:20.666733 2308 raft_consensus.cc:2454] T f861c5ab8c07406ea3b06f0a90612deb P 58372f3855154f02b1652297c709ac56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cd8ead90d5c4ae6bb92ab20cc986875 in term 0.
I20260926 23:30:20.666795 2378 leader_election.cc:302] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cd8ead90d5c4ae6bb92ab20cc986875, b45c671f77264096a1fbfa9e16df34dc; no voters:
I20260926 23:30:20.667239 2664 raft_consensus.cc:2781] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:30:20.667378 2664 raft_consensus.cc:482] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:30:20.667484 2664 raft_consensus.cc:3037] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:20.669727 2664 raft_consensus.cc:504] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:20.670373 2664 leader_election.cc:288] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875 [CANDIDATE]: Term 1 election: Requested vote from peers 58372f3855154f02b1652297c709ac56 (127.0.97.1:33921), b45c671f77264096a1fbfa9e16df34dc (127.0.97.3:36383)
I20260926 23:30:20.671036 2588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f861c5ab8c07406ea3b06f0a90612deb" candidate_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b45c671f77264096a1fbfa9e16df34dc"
I20260926 23:30:20.671141 2308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f861c5ab8c07406ea3b06f0a90612deb" candidate_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58372f3855154f02b1652297c709ac56"
I20260926 23:30:20.671332 2308 raft_consensus.cc:3037] T f861c5ab8c07406ea3b06f0a90612deb P 58372f3855154f02b1652297c709ac56 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:20.671343 2588 raft_consensus.cc:3037] T f861c5ab8c07406ea3b06f0a90612deb P b45c671f77264096a1fbfa9e16df34dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:20.673795 2308 raft_consensus.cc:2454] T f861c5ab8c07406ea3b06f0a90612deb P 58372f3855154f02b1652297c709ac56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cd8ead90d5c4ae6bb92ab20cc986875 in term 1.
I20260926 23:30:20.673914 2588 raft_consensus.cc:2454] T f861c5ab8c07406ea3b06f0a90612deb P b45c671f77264096a1fbfa9e16df34dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cd8ead90d5c4ae6bb92ab20cc986875 in term 1.
I20260926 23:30:20.674296 2381 leader_election.cc:302] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58372f3855154f02b1652297c709ac56, 8cd8ead90d5c4ae6bb92ab20cc986875; no voters:
I20260926 23:30:20.674760 2664 raft_consensus.cc:2781] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:30:20.675235 2664 raft_consensus.cc:686] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 1 LEADER]: Becoming Leader. State: Replica: 8cd8ead90d5c4ae6bb92ab20cc986875, State: Running, Role: LEADER
I20260926 23:30:20.675793 2664 consensus_queue.cc:237] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:20.679234 2681 tablet_replica.cc:439] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875: TxnStatusTablet state changed. Reason: New leader 8cd8ead90d5c4ae6bb92ab20cc986875. Latest consensus state: current_term: 1 leader_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } } }
I20260926 23:30:20.679739 2681 tablet_replica.cc:442] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875: This TxnStatusTablet replica's current role is: LEADER
I20260926 23:30:20.680889 2683 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 23:30:20.681300 2683 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 23:30:20.681200 2151 catalog_manager.cc:5495] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8cd8ead90d5c4ae6bb92ab20cc986875 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } health_report { overall_health: UNKNOWN } } }
I20260926 23:30:20.719973 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:20.714082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46071 (local address 127.0.0.1:33060)
0926 23:30:20.714353 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:20.714375 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:20.714520 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:20.715110 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:20.715115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:20.715127 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:30:20.715309 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:20.715319 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.716389 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.716394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:20.717392 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:20.717405 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.718172 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.718178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:20.718326 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:20.718393 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:30:20.719395 (+ 1002us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:30:20.719417 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 23:30:20.719508 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 23:30:20.719669 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 23:30:20.719995 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:20.714389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33060 (local address 127.0.97.2:46071)
0926 23:30:20.714646 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:20.714652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:20.714673 (+ 21us) server_negotiation.cc:407] Connection header received
0926 23:30:20.714801 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:20.714804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:20.714945 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:20.715075 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:30:20.715442 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.716261 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.717562 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.718011 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.718118 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:20.718579 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:30:20.719287 (+ 708us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:30:20.719423 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:20.719546 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 23:30:20.719633 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 23:30:20.729696 2588 raft_consensus.cc:1261] T f861c5ab8c07406ea3b06f0a90612deb P b45c671f77264096a1fbfa9e16df34dc [term 1 FOLLOWER]: Refusing update from remote peer 8cd8ead90d5c4ae6bb92ab20cc986875: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:30:20.730077 2308 raft_consensus.cc:1261] T f861c5ab8c07406ea3b06f0a90612deb P 58372f3855154f02b1652297c709ac56 [term 1 FOLLOWER]: Refusing update from remote peer 8cd8ead90d5c4ae6bb92ab20cc986875: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:30:20.730854 2664 consensus_queue.cc:1035] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:30:20.731324 2681 consensus_queue.cc:1035] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:30:20.743062 2665 tablet_replica.cc:439] T f861c5ab8c07406ea3b06f0a90612deb P b45c671f77264096a1fbfa9e16df34dc: TxnStatusTablet state changed. Reason: New leader 8cd8ead90d5c4ae6bb92ab20cc986875. Latest consensus state: current_term: 1 leader_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } } }
I20260926 23:30:20.743424 2665 tablet_replica.cc:442] T f861c5ab8c07406ea3b06f0a90612deb P b45c671f77264096a1fbfa9e16df34dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:30:20.744728 2663 tablet_replica.cc:439] T f861c5ab8c07406ea3b06f0a90612deb P 58372f3855154f02b1652297c709ac56: TxnStatusTablet state changed. Reason: New leader 8cd8ead90d5c4ae6bb92ab20cc986875. Latest consensus state: current_term: 1 leader_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } } }
I20260926 23:30:20.745119 2663 tablet_replica.cc:442] T f861c5ab8c07406ea3b06f0a90612deb P 58372f3855154f02b1652297c709ac56: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:30:20.747709 2686 mvcc.cc:205] Tried to move back new op lower bound from 7333746363297644544 to 7333746363091603456. Current Snapshot: MvccSnapshot[applied={T|T < 7333746363297644544}]
I20260926 23:30:20.748534 2681 tablet_replica.cc:439] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } } }
I20260926 23:30:20.748852 2681 tablet_replica.cc:442] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875: This TxnStatusTablet replica's current role is: LEADER
I20260926 23:30:20.752357 2664 tablet_replica.cc:439] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } } }
I20260926 23:30:20.752697 2664 tablet_replica.cc:442] T f861c5ab8c07406ea3b06f0a90612deb P 8cd8ead90d5c4ae6bb92ab20cc986875: This TxnStatusTablet replica's current role is: LEADER
I20260926 23:30:20.760727 2663 tablet_replica.cc:439] T f861c5ab8c07406ea3b06f0a90612deb P 58372f3855154f02b1652297c709ac56: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } } }
I20260926 23:30:20.761070 2663 tablet_replica.cc:442] T f861c5ab8c07406ea3b06f0a90612deb P 58372f3855154f02b1652297c709ac56: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:30:20.762003 2665 tablet_replica.cc:439] T f861c5ab8c07406ea3b06f0a90612deb P b45c671f77264096a1fbfa9e16df34dc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } } }
I20260926 23:30:20.762328 2665 tablet_replica.cc:442] T f861c5ab8c07406ea3b06f0a90612deb P b45c671f77264096a1fbfa9e16df34dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:30:20.772258 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:20.766849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38164 (local address 127.0.97.1:33921)
0926 23:30:20.767009 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:20.767015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:20.767312 (+ 297us) server_negotiation.cc:407] Connection header received
0926 23:30:20.767535 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:20.767541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:20.767617 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:20.767748 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:30:20.768101 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.768824 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.770114 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.770493 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.770595 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:20.771087 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:30:20.771642 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:30:20.771772 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:20.771861 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 23:30:20.771945 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 23:30:20.790722 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:20.785037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35241 (local address 127.0.97.2:46071)
0926 23:30:20.785217 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:20.785223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:20.785766 (+ 543us) server_negotiation.cc:407] Connection header received
0926 23:30:20.785970 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:20.785974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:20.786070 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:20.786149 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:30:20.786631 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.787441 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.788888 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.789387 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.789498 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:20.789962 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:30:20.790099 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:30:20.790189 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:20.790305 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 23:30:20.790382 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 23:30:20.790810 2697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:20.784941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46071 (local address 127.0.97.1:35241)
0926 23:30:20.785634 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:20.785653 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:20.785832 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:20.786173 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:20.786178 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:20.786189 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:30:20.786486 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:20.786495 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.787587 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.787591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:20.788742 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:20.788754 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.789572 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.789577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:20.789697 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:20.789759 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:30:20.790196 (+ 437us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:30:20.790206 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 23:30:20.790277 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 23:30:20.790422 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":323,"threads_started":1}
I20260926 23:30:20.803027 2588 raft_consensus.cc:1261] T fafb70ef86bc4c67ba8cc3c7be3e2493 P b45c671f77264096a1fbfa9e16df34dc [term 1 FOLLOWER]: Refusing update from remote peer 58372f3855154f02b1652297c709ac56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:30:20.803747 2448 raft_consensus.cc:1261] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 1 FOLLOWER]: Refusing update from remote peer 58372f3855154f02b1652297c709ac56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:30:20.803881 2668 consensus_queue.cc:1035] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:30:20.804708 2668 consensus_queue.cc:1035] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:30:20.815912 2689 mvcc.cc:205] Tried to move back new op lower bound from 7333746363599253504 to 7333746362169442304. Current Snapshot: MvccSnapshot[applied={T|T < 7333746363599253504}]
I20260926 23:30:20.853338 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:20.846751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35887 (local address 127.0.97.1:33921)
0926 23:30:20.846965 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:20.846971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:20.847275 (+ 304us) server_negotiation.cc:407] Connection header received
0926 23:30:20.847564 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:20.847568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:20.847659 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:20.847763 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:30:20.848555 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.849459 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.851511 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.851969 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.852081 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:20.852540 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:30:20.852655 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:30:20.852760 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:20.852956 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 23:30:20.853042 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 23:30:20.854162 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:20.845907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33921 (local address 127.0.97.2:35887)
0926 23:30:20.847134 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:20.847162 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:20.847404 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:20.847812 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:20.847817 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:20.847831 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:30:20.848227 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:20.848239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.849612 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.849619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:20.851369 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:20.851383 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:20.852157 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:20.852171 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:20.852301 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:20.852372 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:30:20.852819 (+ 447us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:30:20.852830 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 23:30:20.853565 (+ 735us) client_negotiation.cc:240] Negotiation successful
0926 23:30:20.853733 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"spinlock_wait_cycles":2432,"thread_start_us":900,"threads_started":1}
I20260926 23:30:20.896622 2150 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:30:20.897034 2150 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:30:20.903218 2150 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd9f94aebcc1490c842e1a85f4df9918: Sending DeleteTablet for 3 replicas of tablet fafb70ef86bc4c67ba8cc3c7be3e2493
I20260926 23:30:20.904870 2288 tablet_service.cc:1514] Processing DeleteTablet for tablet fafb70ef86bc4c67ba8cc3c7be3e2493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:30:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38160
I20260926 23:30:20.906381 2428 tablet_service.cc:1514] Processing DeleteTablet for tablet fafb70ef86bc4c67ba8cc3c7be3e2493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:30:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33052
I20260926 23:30:20.911283 2706 tablet_replica.cc:330] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 8cd8ead90d5c4ae6bb92ab20cc986875: stopping tablet replica
I20260926 23:30:20.911870 2706 raft_consensus.cc:2229] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:30:20.912290 2706 raft_consensus.cc:2258] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:30:20.912302 2568 tablet_service.cc:1514] Processing DeleteTablet for tablet fafb70ef86bc4c67ba8cc3c7be3e2493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:30:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47524
I20260926 23:30:20.920202 2150 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 23:30:20.920807 2706 ts_tablet_manager.cc:1905] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 8cd8ead90d5c4ae6bb92ab20cc986875: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 23:30:20.921068 2150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:30:20.930778 2706 ts_tablet_manager.cc:1918] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 8cd8ead90d5c4ae6bb92ab20cc986875: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:30:20.931023 2706 log.cc:1196] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 8cd8ead90d5c4ae6bb92ab20cc986875: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-1/wal/wals/fafb70ef86bc4c67ba8cc3c7be3e2493
I20260926 23:30:20.931613 2706 ts_tablet_manager.cc:1939] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 8cd8ead90d5c4ae6bb92ab20cc986875: Deleting consensus metadata
I20260926 23:30:20.934108 2136 catalog_manager.cc:4841] TS 8cd8ead90d5c4ae6bb92ab20cc986875 (127.0.97.2:46071): tablet fafb70ef86bc4c67ba8cc3c7be3e2493 (table test-table [id=548eb5203f9f4e5bb886d3c1c34a5707]) successfully deleted
I20260926 23:30:20.936271 2287 tablet_service.cc:1467] Processing CreateTablet for tablet decc068927a7483b926d6dbc12106363 (DEFAULT_TABLE table=test-table [id=ea00821b765149d581cc758f70fcaa7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:30:20.936496 2428 tablet_service.cc:1467] Processing CreateTablet for tablet decc068927a7483b926d6dbc12106363 (DEFAULT_TABLE table=test-table [id=ea00821b765149d581cc758f70fcaa7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:30:20.936681 2567 tablet_service.cc:1467] Processing CreateTablet for tablet decc068927a7483b926d6dbc12106363 (DEFAULT_TABLE table=test-table [id=ea00821b765149d581cc758f70fcaa7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:30:20.936908 2428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet decc068927a7483b926d6dbc12106363. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:20.936908 2287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet decc068927a7483b926d6dbc12106363. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:20.937160 2567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet decc068927a7483b926d6dbc12106363. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:20.944355 2657 tablet_bootstrap.cc:492] T decc068927a7483b926d6dbc12106363 P 58372f3855154f02b1652297c709ac56: Bootstrap starting.
I20260926 23:30:20.944355 2659 tablet_bootstrap.cc:492] T decc068927a7483b926d6dbc12106363 P b45c671f77264096a1fbfa9e16df34dc: Bootstrap starting.
I20260926 23:30:20.946276 2657 tablet_bootstrap.cc:654] T decc068927a7483b926d6dbc12106363 P 58372f3855154f02b1652297c709ac56: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:20.946300 2659 tablet_bootstrap.cc:654] T decc068927a7483b926d6dbc12106363 P b45c671f77264096a1fbfa9e16df34dc: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:20.950630 2658 tablet_bootstrap.cc:492] T decc068927a7483b926d6dbc12106363 P 8cd8ead90d5c4ae6bb92ab20cc986875: Bootstrap starting.
I20260926 23:30:20.952665 2658 tablet_bootstrap.cc:654] T decc068927a7483b926d6dbc12106363 P 8cd8ead90d5c4ae6bb92ab20cc986875: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:20.955456 2705 tablet_replica.cc:330] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56: stopping tablet replica
I20260926 23:30:20.956097 2705 raft_consensus.cc:2229] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:30:20.956930 2705 raft_consensus.cc:2258] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:30:20.957844 2707 tablet_replica.cc:330] T fafb70ef86bc4c67ba8cc3c7be3e2493 P b45c671f77264096a1fbfa9e16df34dc: stopping tablet replica
I20260926 23:30:20.958278 2707 raft_consensus.cc:2229] T fafb70ef86bc4c67ba8cc3c7be3e2493 P b45c671f77264096a1fbfa9e16df34dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:30:20.958658 2707 raft_consensus.cc:2258] T fafb70ef86bc4c67ba8cc3c7be3e2493 P b45c671f77264096a1fbfa9e16df34dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:30:20.961195 2707 ts_tablet_manager.cc:1905] T fafb70ef86bc4c67ba8cc3c7be3e2493 P b45c671f77264096a1fbfa9e16df34dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:30:20.962224 2658 tablet_bootstrap.cc:492] T decc068927a7483b926d6dbc12106363 P 8cd8ead90d5c4ae6bb92ab20cc986875: No bootstrap required, opened a new log
I20260926 23:30:20.962464 2658 ts_tablet_manager.cc:1397] T decc068927a7483b926d6dbc12106363 P 8cd8ead90d5c4ae6bb92ab20cc986875: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.004s
I20260926 23:30:20.963369 2705 ts_tablet_manager.cc:1905] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:30:20.963428 2658 raft_consensus.cc:348] T decc068927a7483b926d6dbc12106363 P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:20.963721 2658 raft_consensus.cc:374] T decc068927a7483b926d6dbc12106363 P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:20.963807 2658 raft_consensus.cc:729] T decc068927a7483b926d6dbc12106363 P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cd8ead90d5c4ae6bb92ab20cc986875, State: Initialized, Role: FOLLOWER
I20260926 23:30:20.965457 2658 consensus_queue.cc:260] T decc068927a7483b926d6dbc12106363 P 8cd8ead90d5c4ae6bb92ab20cc986875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:20.966259 2658 ts_tablet_manager.cc:1428] T decc068927a7483b926d6dbc12106363 P 8cd8ead90d5c4ae6bb92ab20cc986875: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260926 23:30:20.973338 2707 ts_tablet_manager.cc:1918] T fafb70ef86bc4c67ba8cc3c7be3e2493 P b45c671f77264096a1fbfa9e16df34dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:30:20.973584 2707 log.cc:1196] T fafb70ef86bc4c67ba8cc3c7be3e2493 P b45c671f77264096a1fbfa9e16df34dc: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-2/wal/wals/fafb70ef86bc4c67ba8cc3c7be3e2493
I20260926 23:30:20.974253 2707 ts_tablet_manager.cc:1939] T fafb70ef86bc4c67ba8cc3c7be3e2493 P b45c671f77264096a1fbfa9e16df34dc: Deleting consensus metadata
I20260926 23:30:20.974249 2659 tablet_bootstrap.cc:492] T decc068927a7483b926d6dbc12106363 P b45c671f77264096a1fbfa9e16df34dc: No bootstrap required, opened a new log
I20260926 23:30:20.974498 2659 ts_tablet_manager.cc:1397] T decc068927a7483b926d6dbc12106363 P b45c671f77264096a1fbfa9e16df34dc: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.011s
I20260926 23:30:20.975551 2659 raft_consensus.cc:348] T decc068927a7483b926d6dbc12106363 P b45c671f77264096a1fbfa9e16df34dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:20.975857 2659 raft_consensus.cc:374] T decc068927a7483b926d6dbc12106363 P b45c671f77264096a1fbfa9e16df34dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:20.975942 2659 raft_consensus.cc:729] T decc068927a7483b926d6dbc12106363 P b45c671f77264096a1fbfa9e16df34dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45c671f77264096a1fbfa9e16df34dc, State: Initialized, Role: FOLLOWER
I20260926 23:30:20.976269 2659 consensus_queue.cc:260] T decc068927a7483b926d6dbc12106363 P b45c671f77264096a1fbfa9e16df34dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:20.980602 2659 ts_tablet_manager.cc:1428] T decc068927a7483b926d6dbc12106363 P b45c671f77264096a1fbfa9e16df34dc: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260926 23:30:20.983913 2705 ts_tablet_manager.cc:1918] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:30:20.984161 2705 log.cc:1196] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-0/wal/wals/fafb70ef86bc4c67ba8cc3c7be3e2493
I20260926 23:30:20.984539 2657 tablet_bootstrap.cc:492] T decc068927a7483b926d6dbc12106363 P 58372f3855154f02b1652297c709ac56: No bootstrap required, opened a new log
I20260926 23:30:20.984735 2657 ts_tablet_manager.cc:1397] T decc068927a7483b926d6dbc12106363 P 58372f3855154f02b1652297c709ac56: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.005s
I20260926 23:30:20.984813 2705 ts_tablet_manager.cc:1939] T fafb70ef86bc4c67ba8cc3c7be3e2493 P 58372f3855154f02b1652297c709ac56: Deleting consensus metadata
I20260926 23:30:20.985553 2657 raft_consensus.cc:348] T decc068927a7483b926d6dbc12106363 P 58372f3855154f02b1652297c709ac56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:20.985834 2657 raft_consensus.cc:374] T decc068927a7483b926d6dbc12106363 P 58372f3855154f02b1652297c709ac56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:20.985918 2657 raft_consensus.cc:729] T decc068927a7483b926d6dbc12106363 P 58372f3855154f02b1652297c709ac56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58372f3855154f02b1652297c709ac56, State: Initialized, Role: FOLLOWER
I20260926 23:30:20.986913 2138 catalog_manager.cc:4841] TS 58372f3855154f02b1652297c709ac56 (127.0.97.1:33921): tablet fafb70ef86bc4c67ba8cc3c7be3e2493 (table test-table [id=548eb5203f9f4e5bb886d3c1c34a5707]) successfully deleted
I20260926 23:30:20.986212 2657 consensus_queue.cc:260] T decc068927a7483b926d6dbc12106363 P 58372f3855154f02b1652297c709ac56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:20.988936 2135 catalog_manager.cc:4841] TS b45c671f77264096a1fbfa9e16df34dc (127.0.97.3:36383): tablet fafb70ef86bc4c67ba8cc3c7be3e2493 (table test-table [id=548eb5203f9f4e5bb886d3c1c34a5707]) successfully deleted
I20260926 23:30:20.989527 2657 ts_tablet_manager.cc:1428] T decc068927a7483b926d6dbc12106363 P 58372f3855154f02b1652297c709ac56: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260926 23:30:21.009508 2665 raft_consensus.cc:482] T decc068927a7483b926d6dbc12106363 P b45c671f77264096a1fbfa9e16df34dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:30:21.009824 2665 raft_consensus.cc:504] T decc068927a7483b926d6dbc12106363 P b45c671f77264096a1fbfa9e16df34dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:21.012135 2665 leader_election.cc:288] T decc068927a7483b926d6dbc12106363 P b45c671f77264096a1fbfa9e16df34dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58372f3855154f02b1652297c709ac56 (127.0.97.1:33921), 8cd8ead90d5c4ae6bb92ab20cc986875 (127.0.97.2:46071)
I20260926 23:30:21.031368 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:21.013488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37027 (local address 127.0.97.1:33921)
0926 23:30:21.013703 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:21.013709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:21.022444 (+ 8735us) server_negotiation.cc:407] Connection header received
0926 23:30:21.022723 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:21.022729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:21.022827 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:21.022931 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:30:21.023994 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:21.025241 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:21.028322 (+ 3081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:21.029793 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:21.029920 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:21.030071 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:21.030811 (+ 740us) server_negotiation.cc:299] Negotiation successful
0926 23:30:21.030994 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 23:30:21.032400 2712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:21.012526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33921 (local address 127.0.97.3:37027)
0926 23:30:21.022267 (+ 9741us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:21.022301 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:21.022534 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:21.023297 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:21.023303 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:21.023334 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:30:21.023788 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:21.023803 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:21.025450 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:21.025455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:21.028090 (+ 2635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:21.028109 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:21.030295 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:21.030302 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:21.030524 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:21.030565 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 23:30:21.031730 (+ 1165us) client_negotiation.cc:240] Negotiation successful
0926 23:30:21.031925 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9532,"mutex_wait_us":41,"thread_start_us":10175,"threads_started":1}
I20260926 23:30:21.032763 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:21.013050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50691 (local address 127.0.97.2:46071)
0926 23:30:21.013316 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:21.013324 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:21.023671 (+ 10347us) server_negotiation.cc:407] Connection header received
0926 23:30:21.024018 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:21.024025 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:21.024118 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:21.024209 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:30:21.025063 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:21.026261 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:21.029358 (+ 3097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:21.030918 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:21.031028 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:21.031164 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:21.032254 (+ 1090us) server_negotiation.cc:299] Negotiation successful
0926 23:30:21.032433 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 23:30:21.033300 2308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "decc068927a7483b926d6dbc12106363" candidate_uuid: "b45c671f77264096a1fbfa9e16df34dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58372f3855154f02b1652297c709ac56" is_pre_election: true
I20260926 23:30:21.033639 2308 raft_consensus.cc:2454] T decc068927a7483b926d6dbc12106363 P 58372f3855154f02b1652297c709ac56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b45c671f77264096a1fbfa9e16df34dc in term 0.
I20260926 23:30:21.034214 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:21.012621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46071 (local address 127.0.97.3:50691)
0926 23:30:21.023537 (+ 10916us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:21.023561 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:21.023822 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:21.024419 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:21.024425 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:21.024451 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:30:21.024841 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:21.024855 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:21.026449 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:21.026454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:21.029163 (+ 2709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:21.029179 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:21.031364 (+ 2185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:21.031370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:21.032028 (+ 658us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:21.032064 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 23:30:21.033404 (+ 1340us) client_negotiation.cc:240] Negotiation successful
0926 23:30:21.033655 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10765,"mutex_wait_us":1,"thread_start_us":10442,"threads_started":1}
I20260926 23:30:21.034345 2521 leader_election.cc:302] T decc068927a7483b926d6dbc12106363 P b45c671f77264096a1fbfa9e16df34dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58372f3855154f02b1652297c709ac56, b45c671f77264096a1fbfa9e16df34dc; no voters:
I20260926 23:30:21.034909 2665 raft_consensus.cc:2781] T decc068927a7483b926d6dbc12106363 P b45c671f77264096a1fbfa9e16df34dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:30:21.035022 2665 raft_consensus.cc:482] T decc068927a7483b926d6dbc12106363 P b45c671f77264096a1fbfa9e16df34dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:30:21.035110 2665 raft_consensus.cc:3037] T decc068927a7483b926d6dbc12106363 P b45c671f77264096a1fbfa9e16df34dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:21.035147 2448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "decc068927a7483b926d6dbc12106363" candidate_uuid: "b45c671f77264096a1fbfa9e16df34dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" is_pre_election: true
I20260926 23:30:21.035416 2448 raft_consensus.cc:2454] T decc068927a7483b926d6dbc12106363 P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b45c671f77264096a1fbfa9e16df34dc in term 0.
I20260926 23:30:21.038677 2665 raft_consensus.cc:504] T decc068927a7483b926d6dbc12106363 P b45c671f77264096a1fbfa9e16df34dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:21.040170 2448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "decc068927a7483b926d6dbc12106363" candidate_uuid: "b45c671f77264096a1fbfa9e16df34dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875"
I20260926 23:30:21.040441 2448 raft_consensus.cc:3037] T decc068927a7483b926d6dbc12106363 P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:21.043053 2308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "decc068927a7483b926d6dbc12106363" candidate_uuid: "b45c671f77264096a1fbfa9e16df34dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58372f3855154f02b1652297c709ac56"
I20260926 23:30:21.043330 2308 raft_consensus.cc:3037] T decc068927a7483b926d6dbc12106363 P 58372f3855154f02b1652297c709ac56 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:21.043946 2448 raft_consensus.cc:2454] T decc068927a7483b926d6dbc12106363 P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b45c671f77264096a1fbfa9e16df34dc in term 1.
I20260926 23:30:21.044539 2519 leader_election.cc:302] T decc068927a7483b926d6dbc12106363 P b45c671f77264096a1fbfa9e16df34dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cd8ead90d5c4ae6bb92ab20cc986875, b45c671f77264096a1fbfa9e16df34dc; no voters:
I20260926 23:30:21.046959 2665 leader_election.cc:288] T decc068927a7483b926d6dbc12106363 P b45c671f77264096a1fbfa9e16df34dc [CANDIDATE]: Term 1 election: Requested vote from peers 58372f3855154f02b1652297c709ac56 (127.0.97.1:33921), 8cd8ead90d5c4ae6bb92ab20cc986875 (127.0.97.2:46071)
I20260926 23:30:21.046669 2308 raft_consensus.cc:2454] T decc068927a7483b926d6dbc12106363 P 58372f3855154f02b1652297c709ac56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b45c671f77264096a1fbfa9e16df34dc in term 1.
I20260926 23:30:21.047446 2665 raft_consensus.cc:2781] T decc068927a7483b926d6dbc12106363 P b45c671f77264096a1fbfa9e16df34dc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:30:21.048246 2665 raft_consensus.cc:686] T decc068927a7483b926d6dbc12106363 P b45c671f77264096a1fbfa9e16df34dc [term 1 LEADER]: Becoming Leader. State: Replica: b45c671f77264096a1fbfa9e16df34dc, State: Running, Role: LEADER
I20260926 23:30:21.048856 2665 consensus_queue.cc:237] T decc068927a7483b926d6dbc12106363 P b45c671f77264096a1fbfa9e16df34dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:21.066177 2151 catalog_manager.cc:5495] T decc068927a7483b926d6dbc12106363 P b45c671f77264096a1fbfa9e16df34dc reported cstate change: term changed from 0 to 1, leader changed from <none> to b45c671f77264096a1fbfa9e16df34dc (127.0.97.3). New cstate: current_term: 1 leader_uuid: "b45c671f77264096a1fbfa9e16df34dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } health_report { overall_health: HEALTHY } } }
I20260926 23:30:21.114127 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:21.105821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47528 (local address 127.0.97.3:36383)
0926 23:30:21.106007 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:21.106015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:21.106039 (+ 24us) server_negotiation.cc:407] Connection header received
0926 23:30:21.106183 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:21.106189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:21.106347 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:21.106543 (+ 196us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:30:21.107207 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:21.108391 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:21.110414 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:21.111158 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:21.111297 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:21.111979 (+ 682us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:30:21.112972 (+ 993us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:30:21.113137 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:21.113692 (+ 555us) server_negotiation.cc:299] Negotiation successful
0926 23:30:21.113787 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 23:30:21.143337 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:21.130229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55147 (local address 127.0.97.2:46071)
0926 23:30:21.130476 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:21.130483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:21.135694 (+ 5211us) server_negotiation.cc:407] Connection header received
0926 23:30:21.135958 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:21.135965 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:21.136074 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:21.136203 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:30:21.136908 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:21.138109 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:21.140329 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:21.141221 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:21.141412 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:21.142243 (+ 831us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:30:21.142473 (+ 230us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:30:21.142549 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:21.142824 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 23:30:21.142941 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 23:30:21.144634 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:21.130107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46071 (local address 127.0.97.3:55147)
0926 23:30:21.135481 (+ 5374us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:21.135512 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:21.135765 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:21.136235 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:21.136240 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:21.136255 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:30:21.136685 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:21.136699 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:21.138269 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:21.138275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:21.140117 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:21.140140 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:21.141667 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:21.141675 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:21.141890 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:21.141991 (+ 101us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:30:21.142639 (+ 648us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:30:21.142650 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 23:30:21.143981 (+ 1331us) client_negotiation.cc:240] Negotiation successful
0926 23:30:21.144199 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5100,"thread_start_us":4840,"threads_started":1}
I20260926 23:30:21.171476 2448 raft_consensus.cc:1261] T decc068927a7483b926d6dbc12106363 P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 1 FOLLOWER]: Refusing update from remote peer b45c671f77264096a1fbfa9e16df34dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:30:21.172772 2713 consensus_queue.cc:1035] T decc068927a7483b926d6dbc12106363 P b45c671f77264096a1fbfa9e16df34dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:30:21.176631 2308 raft_consensus.cc:1261] T decc068927a7483b926d6dbc12106363 P 58372f3855154f02b1652297c709ac56 [term 1 FOLLOWER]: Refusing update from remote peer b45c671f77264096a1fbfa9e16df34dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:30:21.177649 2713 consensus_queue.cc:1035] T decc068927a7483b926d6dbc12106363 P b45c671f77264096a1fbfa9e16df34dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:30:21.266924 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:21.249263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40205 (local address 127.0.97.3:36383)
0926 23:30:21.249480 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:21.249487 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:21.249508 (+ 21us) server_negotiation.cc:407] Connection header received
0926 23:30:21.249580 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:21.249586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:21.249676 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:21.249763 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:30:21.250720 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:21.251891 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:21.262903 (+ 11012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:21.263731 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:21.263916 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:21.265700 (+ 1784us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:30:21.265875 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:30:21.265963 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:21.266453 (+ 490us) server_negotiation.cc:299] Negotiation successful
0926 23:30:21.266568 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 23:30:21.267539 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:21.248343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36383 (local address 127.0.97.2:40205)
0926 23:30:21.248745 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:21.248769 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:21.248909 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:21.250278 (+ 1369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:21.250283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:21.250298 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:30:21.250526 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:21.250537 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:21.252500 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:21.252506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:21.254197 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:21.254216 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:21.265026 (+ 10810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:21.265034 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:21.265402 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:21.265474 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:30:21.266015 (+ 541us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:30:21.266029 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 23:30:21.266106 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 23:30:21.267181 (+ 1075us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240}
W20260926 23:30:21.268891 2689 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 23:30:22.891001 2151 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:30:22.891359 2151 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:30:22.897693 2151 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd9f94aebcc1490c842e1a85f4df9918: Sending DeleteTablet for 3 replicas of tablet decc068927a7483b926d6dbc12106363
I20260926 23:30:22.898874 2428 tablet_service.cc:1514] Processing DeleteTablet for tablet decc068927a7483b926d6dbc12106363 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:30:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33052
I20260926 23:30:22.899384 2287 tablet_service.cc:1514] Processing DeleteTablet for tablet decc068927a7483b926d6dbc12106363 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:30:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38160
I20260926 23:30:22.899981 2568 tablet_service.cc:1514] Processing DeleteTablet for tablet decc068927a7483b926d6dbc12106363 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:30:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47524
I20260926 23:30:22.902540 2746 tablet_replica.cc:330] T decc068927a7483b926d6dbc12106363 P b45c671f77264096a1fbfa9e16df34dc: stopping tablet replica
I20260926 23:30:22.902953 2746 raft_consensus.cc:2229] T decc068927a7483b926d6dbc12106363 P b45c671f77264096a1fbfa9e16df34dc [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:30:22.903720 2746 raft_consensus.cc:2258] T decc068927a7483b926d6dbc12106363 P b45c671f77264096a1fbfa9e16df34dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:30:22.905218 2746 ts_tablet_manager.cc:1905] T decc068927a7483b926d6dbc12106363 P b45c671f77264096a1fbfa9e16df34dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:30:22.910386 2745 tablet_replica.cc:330] T decc068927a7483b926d6dbc12106363 P 58372f3855154f02b1652297c709ac56: stopping tablet replica
I20260926 23:30:22.910822 2745 raft_consensus.cc:2229] T decc068927a7483b926d6dbc12106363 P 58372f3855154f02b1652297c709ac56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:30:22.911185 2745 raft_consensus.cc:2258] T decc068927a7483b926d6dbc12106363 P 58372f3855154f02b1652297c709ac56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:30:22.912223 2744 tablet_replica.cc:330] T decc068927a7483b926d6dbc12106363 P 8cd8ead90d5c4ae6bb92ab20cc986875: stopping tablet replica
I20260926 23:30:22.912498 2744 raft_consensus.cc:2229] T decc068927a7483b926d6dbc12106363 P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:30:22.912823 2744 raft_consensus.cc:2258] T decc068927a7483b926d6dbc12106363 P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:30:22.914062 2746 ts_tablet_manager.cc:1918] T decc068927a7483b926d6dbc12106363 P b45c671f77264096a1fbfa9e16df34dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 23:30:22.914144 2745 ts_tablet_manager.cc:1905] T decc068927a7483b926d6dbc12106363 P 58372f3855154f02b1652297c709ac56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:30:22.914340 2746 log.cc:1196] T decc068927a7483b926d6dbc12106363 P b45c671f77264096a1fbfa9e16df34dc: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-2/wal/wals/decc068927a7483b926d6dbc12106363
I20260926 23:30:22.914924 2746 ts_tablet_manager.cc:1939] T decc068927a7483b926d6dbc12106363 P b45c671f77264096a1fbfa9e16df34dc: Deleting consensus metadata
I20260926 23:30:22.916383 2135 catalog_manager.cc:4841] TS b45c671f77264096a1fbfa9e16df34dc (127.0.97.3:36383): tablet decc068927a7483b926d6dbc12106363 (table test-table [id=ea00821b765149d581cc758f70fcaa7b]) successfully deleted
I20260926 23:30:22.917357 2744 ts_tablet_manager.cc:1905] T decc068927a7483b926d6dbc12106363 P 8cd8ead90d5c4ae6bb92ab20cc986875: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:30:22.920686 2745 ts_tablet_manager.cc:1918] T decc068927a7483b926d6dbc12106363 P 58372f3855154f02b1652297c709ac56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 23:30:22.920864 2745 log.cc:1196] T decc068927a7483b926d6dbc12106363 P 58372f3855154f02b1652297c709ac56: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-0/wal/wals/decc068927a7483b926d6dbc12106363
I20260926 23:30:22.921245 2745 ts_tablet_manager.cc:1939] T decc068927a7483b926d6dbc12106363 P 58372f3855154f02b1652297c709ac56: Deleting consensus metadata
I20260926 23:30:22.922808 2138 catalog_manager.cc:4841] TS 58372f3855154f02b1652297c709ac56 (127.0.97.1:33921): tablet decc068927a7483b926d6dbc12106363 (table test-table [id=ea00821b765149d581cc758f70fcaa7b]) successfully deleted
I20260926 23:30:22.924043 2744 ts_tablet_manager.cc:1918] T decc068927a7483b926d6dbc12106363 P 8cd8ead90d5c4ae6bb92ab20cc986875: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 23:30:22.924158 2744 log.cc:1196] T decc068927a7483b926d6dbc12106363 P 8cd8ead90d5c4ae6bb92ab20cc986875: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-1/wal/wals/decc068927a7483b926d6dbc12106363
I20260926 23:30:22.924459 2744 ts_tablet_manager.cc:1939] T decc068927a7483b926d6dbc12106363 P 8cd8ead90d5c4ae6bb92ab20cc986875: Deleting consensus metadata
I20260926 23:30:22.925668 2136 catalog_manager.cc:4841] TS 8cd8ead90d5c4ae6bb92ab20cc986875 (127.0.97.2:46071): tablet decc068927a7483b926d6dbc12106363 (table test-table [id=ea00821b765149d581cc758f70fcaa7b]) successfully deleted
Sep 26 23:30:22 dist-test-slave-gwfx krb5kdc[2106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465422, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 23:30:22 dist-test-slave-gwfx krb5kdc[2106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465422, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:22.966806 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:22.948302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56628 (local address 127.0.97.62:37755)
0926 23:30:22.949003 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:22.949008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:22.949036 (+ 28us) server_negotiation.cc:407] Connection header received
0926 23:30:22.949243 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:22.949248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:22.949338 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:22.949534 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:22.951475 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:22.952509 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:22.954321 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:22.954921 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:22.955062 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:22.959822 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:22.959836 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:22.959840 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:22.959888 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:22.962609 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:22.963626 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:22.963632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:22.963635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:22.963818 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:22.964413 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:22.964419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:22.964423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:22.964983 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:22.965216 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:22.965810 (+ 594us) server_negotiation.cc:299] Negotiation successful
0926 23:30:22.966154 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":358,"threads_started":1}
I20260926 23:30:22.972149 2151 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:30:22.972882 2151 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:30:22.981213 2568 tablet_service.cc:1467] Processing CreateTablet for tablet 3e0e07853126489aa73f525513de72ff (DEFAULT_TABLE table=test-table [id=31c8be00b94d4e5496b3743aab0cfc70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:30:22.981305 2428 tablet_service.cc:1467] Processing CreateTablet for tablet 3e0e07853126489aa73f525513de72ff (DEFAULT_TABLE table=test-table [id=31c8be00b94d4e5496b3743aab0cfc70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:30:22.981251 2287 tablet_service.cc:1467] Processing CreateTablet for tablet 3e0e07853126489aa73f525513de72ff (DEFAULT_TABLE table=test-table [id=31c8be00b94d4e5496b3743aab0cfc70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:30:22.981729 2568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e0e07853126489aa73f525513de72ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:22.981918 2428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e0e07853126489aa73f525513de72ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:22.982785 2287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e0e07853126489aa73f525513de72ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:22.988395 2755 tablet_bootstrap.cc:492] T 3e0e07853126489aa73f525513de72ff P b45c671f77264096a1fbfa9e16df34dc: Bootstrap starting.
I20260926 23:30:22.990598 2755 tablet_bootstrap.cc:654] T 3e0e07853126489aa73f525513de72ff P b45c671f77264096a1fbfa9e16df34dc: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:22.991025 2757 tablet_bootstrap.cc:492] T 3e0e07853126489aa73f525513de72ff P 58372f3855154f02b1652297c709ac56: Bootstrap starting.
I20260926 23:30:22.992112 2756 tablet_bootstrap.cc:492] T 3e0e07853126489aa73f525513de72ff P 8cd8ead90d5c4ae6bb92ab20cc986875: Bootstrap starting.
I20260926 23:30:22.993623 2757 tablet_bootstrap.cc:654] T 3e0e07853126489aa73f525513de72ff P 58372f3855154f02b1652297c709ac56: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:22.994289 2755 tablet_bootstrap.cc:492] T 3e0e07853126489aa73f525513de72ff P b45c671f77264096a1fbfa9e16df34dc: No bootstrap required, opened a new log
I20260926 23:30:22.994541 2755 ts_tablet_manager.cc:1397] T 3e0e07853126489aa73f525513de72ff P b45c671f77264096a1fbfa9e16df34dc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 23:30:22.994855 2756 tablet_bootstrap.cc:654] T 3e0e07853126489aa73f525513de72ff P 8cd8ead90d5c4ae6bb92ab20cc986875: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:22.995637 2755 raft_consensus.cc:348] T 3e0e07853126489aa73f525513de72ff P b45c671f77264096a1fbfa9e16df34dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:22.995952 2755 raft_consensus.cc:374] T 3e0e07853126489aa73f525513de72ff P b45c671f77264096a1fbfa9e16df34dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:22.996246 2755 raft_consensus.cc:729] T 3e0e07853126489aa73f525513de72ff P b45c671f77264096a1fbfa9e16df34dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45c671f77264096a1fbfa9e16df34dc, State: Initialized, Role: FOLLOWER
I20260926 23:30:22.996363 2757 tablet_bootstrap.cc:492] T 3e0e07853126489aa73f525513de72ff P 58372f3855154f02b1652297c709ac56: No bootstrap required, opened a new log
I20260926 23:30:22.996647 2757 ts_tablet_manager.cc:1397] T 3e0e07853126489aa73f525513de72ff P 58372f3855154f02b1652297c709ac56: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260926 23:30:22.996685 2755 consensus_queue.cc:260] T 3e0e07853126489aa73f525513de72ff P b45c671f77264096a1fbfa9e16df34dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:22.997418 2755 ts_tablet_manager.cc:1428] T 3e0e07853126489aa73f525513de72ff P b45c671f77264096a1fbfa9e16df34dc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 23:30:22.997787 2757 raft_consensus.cc:348] T 3e0e07853126489aa73f525513de72ff P 58372f3855154f02b1652297c709ac56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:22.998766 2756 tablet_bootstrap.cc:492] T 3e0e07853126489aa73f525513de72ff P 8cd8ead90d5c4ae6bb92ab20cc986875: No bootstrap required, opened a new log
I20260926 23:30:22.999081 2756 ts_tablet_manager.cc:1397] T 3e0e07853126489aa73f525513de72ff P 8cd8ead90d5c4ae6bb92ab20cc986875: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260926 23:30:22.999665 2757 raft_consensus.cc:374] T 3e0e07853126489aa73f525513de72ff P 58372f3855154f02b1652297c709ac56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:22.999841 2757 raft_consensus.cc:729] T 3e0e07853126489aa73f525513de72ff P 58372f3855154f02b1652297c709ac56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58372f3855154f02b1652297c709ac56, State: Initialized, Role: FOLLOWER
I20260926 23:30:22.999961 2756 raft_consensus.cc:348] T 3e0e07853126489aa73f525513de72ff P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:23.000226 2756 raft_consensus.cc:374] T 3e0e07853126489aa73f525513de72ff P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:23.000124 2757 consensus_queue.cc:260] T 3e0e07853126489aa73f525513de72ff P 58372f3855154f02b1652297c709ac56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:23.000319 2756 raft_consensus.cc:729] T 3e0e07853126489aa73f525513de72ff P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cd8ead90d5c4ae6bb92ab20cc986875, State: Initialized, Role: FOLLOWER
I20260926 23:30:23.000568 2756 consensus_queue.cc:260] T 3e0e07853126489aa73f525513de72ff P 8cd8ead90d5c4ae6bb92ab20cc986875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:23.001287 2757 ts_tablet_manager.cc:1428] T 3e0e07853126489aa73f525513de72ff P 58372f3855154f02b1652297c709ac56: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20260926 23:30:23.001660 2756 ts_tablet_manager.cc:1428] T 3e0e07853126489aa73f525513de72ff P 8cd8ead90d5c4ae6bb92ab20cc986875: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 23:30:23.349614 2721 raft_consensus.cc:482] T 3e0e07853126489aa73f525513de72ff P b45c671f77264096a1fbfa9e16df34dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:30:23.349861 2721 raft_consensus.cc:504] T 3e0e07853126489aa73f525513de72ff P b45c671f77264096a1fbfa9e16df34dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:23.350605 2721 leader_election.cc:288] T 3e0e07853126489aa73f525513de72ff P b45c671f77264096a1fbfa9e16df34dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58372f3855154f02b1652297c709ac56 (127.0.97.1:33921), 8cd8ead90d5c4ae6bb92ab20cc986875 (127.0.97.2:46071)
I20260926 23:30:23.351349 2307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e0e07853126489aa73f525513de72ff" candidate_uuid: "b45c671f77264096a1fbfa9e16df34dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58372f3855154f02b1652297c709ac56" is_pre_election: true
I20260926 23:30:23.351596 2307 raft_consensus.cc:2454] T 3e0e07853126489aa73f525513de72ff P 58372f3855154f02b1652297c709ac56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b45c671f77264096a1fbfa9e16df34dc in term 0.
I20260926 23:30:23.351574 2448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e0e07853126489aa73f525513de72ff" candidate_uuid: "b45c671f77264096a1fbfa9e16df34dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" is_pre_election: true
I20260926 23:30:23.351985 2448 raft_consensus.cc:2454] T 3e0e07853126489aa73f525513de72ff P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b45c671f77264096a1fbfa9e16df34dc in term 0.
I20260926 23:30:23.352169 2521 leader_election.cc:302] T 3e0e07853126489aa73f525513de72ff P b45c671f77264096a1fbfa9e16df34dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58372f3855154f02b1652297c709ac56, b45c671f77264096a1fbfa9e16df34dc; no voters:
I20260926 23:30:23.352496 2721 raft_consensus.cc:2781] T 3e0e07853126489aa73f525513de72ff P b45c671f77264096a1fbfa9e16df34dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:30:23.352623 2721 raft_consensus.cc:482] T 3e0e07853126489aa73f525513de72ff P b45c671f77264096a1fbfa9e16df34dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:30:23.352708 2721 raft_consensus.cc:3037] T 3e0e07853126489aa73f525513de72ff P b45c671f77264096a1fbfa9e16df34dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:23.354673 2721 raft_consensus.cc:504] T 3e0e07853126489aa73f525513de72ff P b45c671f77264096a1fbfa9e16df34dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:23.355383 2721 leader_election.cc:288] T 3e0e07853126489aa73f525513de72ff P b45c671f77264096a1fbfa9e16df34dc [CANDIDATE]: Term 1 election: Requested vote from peers 58372f3855154f02b1652297c709ac56 (127.0.97.1:33921), 8cd8ead90d5c4ae6bb92ab20cc986875 (127.0.97.2:46071)
I20260926 23:30:23.355899 2307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e0e07853126489aa73f525513de72ff" candidate_uuid: "b45c671f77264096a1fbfa9e16df34dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58372f3855154f02b1652297c709ac56"
I20260926 23:30:23.356140 2307 raft_consensus.cc:3037] T 3e0e07853126489aa73f525513de72ff P 58372f3855154f02b1652297c709ac56 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:23.356439 2448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e0e07853126489aa73f525513de72ff" candidate_uuid: "b45c671f77264096a1fbfa9e16df34dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875"
I20260926 23:30:23.356788 2448 raft_consensus.cc:3037] T 3e0e07853126489aa73f525513de72ff P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:23.359752 2307 raft_consensus.cc:2454] T 3e0e07853126489aa73f525513de72ff P 58372f3855154f02b1652297c709ac56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b45c671f77264096a1fbfa9e16df34dc in term 1.
I20260926 23:30:23.360378 2521 leader_election.cc:302] T 3e0e07853126489aa73f525513de72ff P b45c671f77264096a1fbfa9e16df34dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58372f3855154f02b1652297c709ac56, b45c671f77264096a1fbfa9e16df34dc; no voters:
I20260926 23:30:23.360515 2448 raft_consensus.cc:2454] T 3e0e07853126489aa73f525513de72ff P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b45c671f77264096a1fbfa9e16df34dc in term 1.
I20260926 23:30:23.360747 2721 raft_consensus.cc:2781] T 3e0e07853126489aa73f525513de72ff P b45c671f77264096a1fbfa9e16df34dc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:30:23.361269 2721 raft_consensus.cc:686] T 3e0e07853126489aa73f525513de72ff P b45c671f77264096a1fbfa9e16df34dc [term 1 LEADER]: Becoming Leader. State: Replica: b45c671f77264096a1fbfa9e16df34dc, State: Running, Role: LEADER
I20260926 23:30:23.361567 2721 consensus_queue.cc:237] T 3e0e07853126489aa73f525513de72ff P b45c671f77264096a1fbfa9e16df34dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:23.365563 2150 catalog_manager.cc:5495] T 3e0e07853126489aa73f525513de72ff P b45c671f77264096a1fbfa9e16df34dc reported cstate change: term changed from 0 to 1, leader changed from <none> to b45c671f77264096a1fbfa9e16df34dc (127.0.97.3). New cstate: current_term: 1 leader_uuid: "b45c671f77264096a1fbfa9e16df34dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } health_report { overall_health: HEALTHY } } }
I20260926 23:30:23.477696 2772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:23.471435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47538 (local address 127.0.97.3:36383)
0926 23:30:23.472376 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:23.472381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:23.472398 (+ 17us) server_negotiation.cc:407] Connection header received
0926 23:30:23.472570 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:23.472574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:23.472654 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:23.472758 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:30:23.473350 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:23.474368 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:23.475883 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:23.476313 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:23.476435 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:23.476870 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:30:23.477018 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:30:23.477098 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:23.477239 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 23:30:23.477327 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":799,"thread_start_us":661,"threads_started":1}
I20260926 23:30:23.489270 2307 raft_consensus.cc:1261] T 3e0e07853126489aa73f525513de72ff P 58372f3855154f02b1652297c709ac56 [term 1 FOLLOWER]: Refusing update from remote peer b45c671f77264096a1fbfa9e16df34dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:30:23.489290 2448 raft_consensus.cc:1261] T 3e0e07853126489aa73f525513de72ff P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 1 FOLLOWER]: Refusing update from remote peer b45c671f77264096a1fbfa9e16df34dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:30:23.489876 2721 consensus_queue.cc:1035] T 3e0e07853126489aa73f525513de72ff P b45c671f77264096a1fbfa9e16df34dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:30:23.490185 2736 consensus_queue.cc:1035] T 3e0e07853126489aa73f525513de72ff P b45c671f77264096a1fbfa9e16df34dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:30:23.497816 2766 mvcc.cc:205] Tried to move back new op lower bound from 7333746374606442496 to 7333746374092738560. Current Snapshot: MvccSnapshot[applied={T|T < 7333746374606442496}]
I20260926 23:30:23.564519 2149 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:30:23.564785 2149 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:30:23.568456 2149 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd9f94aebcc1490c842e1a85f4df9918: Sending DeleteTablet for 3 replicas of tablet 3e0e07853126489aa73f525513de72ff
I20260926 23:30:23.569553 2428 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e0e07853126489aa73f525513de72ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:30:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33052
I20260926 23:30:23.569594 2287 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e0e07853126489aa73f525513de72ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:30:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38160
I20260926 23:30:23.569878 2568 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e0e07853126489aa73f525513de72ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:30:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47524
I20260926 23:30:23.571089 2149 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:30:23.572141 2149 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:30:23.575857 2781 tablet_replica.cc:330] T 3e0e07853126489aa73f525513de72ff P 58372f3855154f02b1652297c709ac56: stopping tablet replica
I20260926 23:30:23.576128 2781 raft_consensus.cc:2229] T 3e0e07853126489aa73f525513de72ff P 58372f3855154f02b1652297c709ac56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:30:23.576397 2781 raft_consensus.cc:2258] T 3e0e07853126489aa73f525513de72ff P 58372f3855154f02b1652297c709ac56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:30:23.577145 2780 tablet_replica.cc:330] T 3e0e07853126489aa73f525513de72ff P 8cd8ead90d5c4ae6bb92ab20cc986875: stopping tablet replica
I20260926 23:30:23.577385 2780 raft_consensus.cc:2229] T 3e0e07853126489aa73f525513de72ff P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:30:23.577502 2782 tablet_replica.cc:330] T 3e0e07853126489aa73f525513de72ff P b45c671f77264096a1fbfa9e16df34dc: stopping tablet replica
I20260926 23:30:23.577701 2780 raft_consensus.cc:2258] T 3e0e07853126489aa73f525513de72ff P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:30:23.577728 2782 raft_consensus.cc:2229] T 3e0e07853126489aa73f525513de72ff P b45c671f77264096a1fbfa9e16df34dc [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:30:23.578315 2782 raft_consensus.cc:2258] T 3e0e07853126489aa73f525513de72ff P b45c671f77264096a1fbfa9e16df34dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:30:23.578958 2781 ts_tablet_manager.cc:1905] T 3e0e07853126489aa73f525513de72ff P 58372f3855154f02b1652297c709ac56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:30:23.579490 2782 ts_tablet_manager.cc:1905] T 3e0e07853126489aa73f525513de72ff P b45c671f77264096a1fbfa9e16df34dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:30:23.582104 2287 tablet_service.cc:1467] Processing CreateTablet for tablet 7d31fb5b68f44ba89e29409391a08943 (DEFAULT_TABLE table=test-table [id=f592e39667ab40e88e9090f906b9bae8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:30:23.582185 2428 tablet_service.cc:1467] Processing CreateTablet for tablet 7d31fb5b68f44ba89e29409391a08943 (DEFAULT_TABLE table=test-table [id=f592e39667ab40e88e9090f906b9bae8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:30:23.582719 2287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d31fb5b68f44ba89e29409391a08943. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:23.583413 2428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d31fb5b68f44ba89e29409391a08943. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:23.583673 2780 ts_tablet_manager.cc:1905] T 3e0e07853126489aa73f525513de72ff P 8cd8ead90d5c4ae6bb92ab20cc986875: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:30:23.584932 2568 tablet_service.cc:1467] Processing CreateTablet for tablet 7d31fb5b68f44ba89e29409391a08943 (DEFAULT_TABLE table=test-table [id=f592e39667ab40e88e9090f906b9bae8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:30:23.585582 2568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d31fb5b68f44ba89e29409391a08943. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:23.588577 2781 ts_tablet_manager.cc:1918] T 3e0e07853126489aa73f525513de72ff P 58372f3855154f02b1652297c709ac56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:30:23.588888 2781 log.cc:1196] T 3e0e07853126489aa73f525513de72ff P 58372f3855154f02b1652297c709ac56: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-0/wal/wals/3e0e07853126489aa73f525513de72ff
I20260926 23:30:23.589963 2781 ts_tablet_manager.cc:1939] T 3e0e07853126489aa73f525513de72ff P 58372f3855154f02b1652297c709ac56: Deleting consensus metadata
I20260926 23:30:23.591511 2138 catalog_manager.cc:4841] TS 58372f3855154f02b1652297c709ac56 (127.0.97.1:33921): tablet 3e0e07853126489aa73f525513de72ff (table test-table [id=31c8be00b94d4e5496b3743aab0cfc70]) successfully deleted
I20260926 23:30:23.597199 2783 tablet_bootstrap.cc:492] T 7d31fb5b68f44ba89e29409391a08943 P 58372f3855154f02b1652297c709ac56: Bootstrap starting.
I20260926 23:30:23.598347 2784 tablet_bootstrap.cc:492] T 7d31fb5b68f44ba89e29409391a08943 P 8cd8ead90d5c4ae6bb92ab20cc986875: Bootstrap starting.
I20260926 23:30:23.600179 2783 tablet_bootstrap.cc:654] T 7d31fb5b68f44ba89e29409391a08943 P 58372f3855154f02b1652297c709ac56: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:23.601150 2784 tablet_bootstrap.cc:654] T 7d31fb5b68f44ba89e29409391a08943 P 8cd8ead90d5c4ae6bb92ab20cc986875: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:23.603719 2782 ts_tablet_manager.cc:1918] T 3e0e07853126489aa73f525513de72ff P b45c671f77264096a1fbfa9e16df34dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:30:23.603994 2782 log.cc:1196] T 3e0e07853126489aa73f525513de72ff P b45c671f77264096a1fbfa9e16df34dc: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-2/wal/wals/3e0e07853126489aa73f525513de72ff
I20260926 23:30:23.604595 2782 ts_tablet_manager.cc:1939] T 3e0e07853126489aa73f525513de72ff P b45c671f77264096a1fbfa9e16df34dc: Deleting consensus metadata
I20260926 23:30:23.605573 2780 ts_tablet_manager.cc:1918] T 3e0e07853126489aa73f525513de72ff P 8cd8ead90d5c4ae6bb92ab20cc986875: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:30:23.605762 2780 log.cc:1196] T 3e0e07853126489aa73f525513de72ff P 8cd8ead90d5c4ae6bb92ab20cc986875: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-1/wal/wals/3e0e07853126489aa73f525513de72ff
I20260926 23:30:23.605968 2135 catalog_manager.cc:4841] TS b45c671f77264096a1fbfa9e16df34dc (127.0.97.3:36383): tablet 3e0e07853126489aa73f525513de72ff (table test-table [id=31c8be00b94d4e5496b3743aab0cfc70]) successfully deleted
I20260926 23:30:23.606309 2780 ts_tablet_manager.cc:1939] T 3e0e07853126489aa73f525513de72ff P 8cd8ead90d5c4ae6bb92ab20cc986875: Deleting consensus metadata
I20260926 23:30:23.607153 2784 tablet_bootstrap.cc:492] T 7d31fb5b68f44ba89e29409391a08943 P 8cd8ead90d5c4ae6bb92ab20cc986875: No bootstrap required, opened a new log
I20260926 23:30:23.607700 2136 catalog_manager.cc:4841] TS 8cd8ead90d5c4ae6bb92ab20cc986875 (127.0.97.2:46071): tablet 3e0e07853126489aa73f525513de72ff (table test-table [id=31c8be00b94d4e5496b3743aab0cfc70]) successfully deleted
I20260926 23:30:23.607358 2784 ts_tablet_manager.cc:1397] T 7d31fb5b68f44ba89e29409391a08943 P 8cd8ead90d5c4ae6bb92ab20cc986875: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260926 23:30:23.608467 2785 tablet_bootstrap.cc:492] T 7d31fb5b68f44ba89e29409391a08943 P b45c671f77264096a1fbfa9e16df34dc: Bootstrap starting.
I20260926 23:30:23.609001 2783 tablet_bootstrap.cc:492] T 7d31fb5b68f44ba89e29409391a08943 P 58372f3855154f02b1652297c709ac56: No bootstrap required, opened a new log
I20260926 23:30:23.609180 2783 ts_tablet_manager.cc:1397] T 7d31fb5b68f44ba89e29409391a08943 P 58372f3855154f02b1652297c709ac56: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20260926 23:30:23.608893 2784 raft_consensus.cc:348] T 7d31fb5b68f44ba89e29409391a08943 P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:23.609889 2784 raft_consensus.cc:374] T 7d31fb5b68f44ba89e29409391a08943 P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:23.610013 2784 raft_consensus.cc:729] T 7d31fb5b68f44ba89e29409391a08943 P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cd8ead90d5c4ae6bb92ab20cc986875, State: Initialized, Role: FOLLOWER
I20260926 23:30:23.610131 2783 raft_consensus.cc:348] T 7d31fb5b68f44ba89e29409391a08943 P 58372f3855154f02b1652297c709ac56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:23.610392 2783 raft_consensus.cc:374] T 7d31fb5b68f44ba89e29409391a08943 P 58372f3855154f02b1652297c709ac56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:23.610450 2785 tablet_bootstrap.cc:654] T 7d31fb5b68f44ba89e29409391a08943 P b45c671f77264096a1fbfa9e16df34dc: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:23.610291 2784 consensus_queue.cc:260] T 7d31fb5b68f44ba89e29409391a08943 P 8cd8ead90d5c4ae6bb92ab20cc986875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:23.610483 2783 raft_consensus.cc:729] T 7d31fb5b68f44ba89e29409391a08943 P 58372f3855154f02b1652297c709ac56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58372f3855154f02b1652297c709ac56, State: Initialized, Role: FOLLOWER
I20260926 23:30:23.610810 2783 consensus_queue.cc:260] T 7d31fb5b68f44ba89e29409391a08943 P 58372f3855154f02b1652297c709ac56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:23.612026 2784 ts_tablet_manager.cc:1428] T 7d31fb5b68f44ba89e29409391a08943 P 8cd8ead90d5c4ae6bb92ab20cc986875: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 23:30:23.612098 2783 ts_tablet_manager.cc:1428] T 7d31fb5b68f44ba89e29409391a08943 P 58372f3855154f02b1652297c709ac56: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 23:30:23.613718 2785 tablet_bootstrap.cc:492] T 7d31fb5b68f44ba89e29409391a08943 P b45c671f77264096a1fbfa9e16df34dc: No bootstrap required, opened a new log
I20260926 23:30:23.613897 2785 ts_tablet_manager.cc:1397] T 7d31fb5b68f44ba89e29409391a08943 P b45c671f77264096a1fbfa9e16df34dc: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260926 23:30:23.614786 2785 raft_consensus.cc:348] T 7d31fb5b68f44ba89e29409391a08943 P b45c671f77264096a1fbfa9e16df34dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:23.615057 2785 raft_consensus.cc:374] T 7d31fb5b68f44ba89e29409391a08943 P b45c671f77264096a1fbfa9e16df34dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:23.615149 2785 raft_consensus.cc:729] T 7d31fb5b68f44ba89e29409391a08943 P b45c671f77264096a1fbfa9e16df34dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45c671f77264096a1fbfa9e16df34dc, State: Initialized, Role: FOLLOWER
I20260926 23:30:23.615391 2785 consensus_queue.cc:260] T 7d31fb5b68f44ba89e29409391a08943 P b45c671f77264096a1fbfa9e16df34dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:23.615990 2785 ts_tablet_manager.cc:1428] T 7d31fb5b68f44ba89e29409391a08943 P b45c671f77264096a1fbfa9e16df34dc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 23:30:23.817910 2761 raft_consensus.cc:482] T 7d31fb5b68f44ba89e29409391a08943 P 58372f3855154f02b1652297c709ac56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:30:23.818172 2761 raft_consensus.cc:504] T 7d31fb5b68f44ba89e29409391a08943 P 58372f3855154f02b1652297c709ac56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:23.818984 2761 leader_election.cc:288] T 7d31fb5b68f44ba89e29409391a08943 P 58372f3855154f02b1652297c709ac56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8cd8ead90d5c4ae6bb92ab20cc986875 (127.0.97.2:46071), b45c671f77264096a1fbfa9e16df34dc (127.0.97.3:36383)
I20260926 23:30:23.819700 2448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d31fb5b68f44ba89e29409391a08943" candidate_uuid: "58372f3855154f02b1652297c709ac56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" is_pre_election: true
I20260926 23:30:23.819926 2448 raft_consensus.cc:2454] T 7d31fb5b68f44ba89e29409391a08943 P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58372f3855154f02b1652297c709ac56 in term 0.
I20260926 23:30:23.819854 2588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d31fb5b68f44ba89e29409391a08943" candidate_uuid: "58372f3855154f02b1652297c709ac56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b45c671f77264096a1fbfa9e16df34dc" is_pre_election: true
I20260926 23:30:23.820214 2588 raft_consensus.cc:2454] T 7d31fb5b68f44ba89e29409391a08943 P b45c671f77264096a1fbfa9e16df34dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58372f3855154f02b1652297c709ac56 in term 0.
I20260926 23:30:23.820423 2239 leader_election.cc:302] T 7d31fb5b68f44ba89e29409391a08943 P 58372f3855154f02b1652297c709ac56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58372f3855154f02b1652297c709ac56, 8cd8ead90d5c4ae6bb92ab20cc986875; no voters:
I20260926 23:30:23.820829 2761 raft_consensus.cc:2781] T 7d31fb5b68f44ba89e29409391a08943 P 58372f3855154f02b1652297c709ac56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:30:23.820982 2761 raft_consensus.cc:482] T 7d31fb5b68f44ba89e29409391a08943 P 58372f3855154f02b1652297c709ac56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:30:23.821079 2761 raft_consensus.cc:3037] T 7d31fb5b68f44ba89e29409391a08943 P 58372f3855154f02b1652297c709ac56 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:23.823475 2761 raft_consensus.cc:504] T 7d31fb5b68f44ba89e29409391a08943 P 58372f3855154f02b1652297c709ac56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:23.824082 2761 leader_election.cc:288] T 7d31fb5b68f44ba89e29409391a08943 P 58372f3855154f02b1652297c709ac56 [CANDIDATE]: Term 1 election: Requested vote from peers 8cd8ead90d5c4ae6bb92ab20cc986875 (127.0.97.2:46071), b45c671f77264096a1fbfa9e16df34dc (127.0.97.3:36383)
I20260926 23:30:23.824837 2588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d31fb5b68f44ba89e29409391a08943" candidate_uuid: "58372f3855154f02b1652297c709ac56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b45c671f77264096a1fbfa9e16df34dc"
I20260926 23:30:23.825045 2588 raft_consensus.cc:3037] T 7d31fb5b68f44ba89e29409391a08943 P b45c671f77264096a1fbfa9e16df34dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:23.824939 2448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d31fb5b68f44ba89e29409391a08943" candidate_uuid: "58372f3855154f02b1652297c709ac56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875"
I20260926 23:30:23.825194 2448 raft_consensus.cc:3037] T 7d31fb5b68f44ba89e29409391a08943 P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:23.828502 2588 raft_consensus.cc:2454] T 7d31fb5b68f44ba89e29409391a08943 P b45c671f77264096a1fbfa9e16df34dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58372f3855154f02b1652297c709ac56 in term 1.
I20260926 23:30:23.828644 2448 raft_consensus.cc:2454] T 7d31fb5b68f44ba89e29409391a08943 P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58372f3855154f02b1652297c709ac56 in term 1.
I20260926 23:30:23.829020 2238 leader_election.cc:302] T 7d31fb5b68f44ba89e29409391a08943 P 58372f3855154f02b1652297c709ac56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58372f3855154f02b1652297c709ac56, b45c671f77264096a1fbfa9e16df34dc; no voters:
I20260926 23:30:23.829391 2761 raft_consensus.cc:2781] T 7d31fb5b68f44ba89e29409391a08943 P 58372f3855154f02b1652297c709ac56 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:30:23.829865 2761 raft_consensus.cc:686] T 7d31fb5b68f44ba89e29409391a08943 P 58372f3855154f02b1652297c709ac56 [term 1 LEADER]: Becoming Leader. State: Replica: 58372f3855154f02b1652297c709ac56, State: Running, Role: LEADER
I20260926 23:30:23.830190 2761 consensus_queue.cc:237] T 7d31fb5b68f44ba89e29409391a08943 P 58372f3855154f02b1652297c709ac56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } }
I20260926 23:30:23.833240 2151 catalog_manager.cc:5495] T 7d31fb5b68f44ba89e29409391a08943 P 58372f3855154f02b1652297c709ac56 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58372f3855154f02b1652297c709ac56 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "58372f3855154f02b1652297c709ac56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58372f3855154f02b1652297c709ac56" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 } health_report { overall_health: UNKNOWN } } }
I20260926 23:30:23.908737 2791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:23.901750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38172 (local address 127.0.97.1:33921)
0926 23:30:23.902500 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:23.902505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:23.902529 (+ 24us) server_negotiation.cc:407] Connection header received
0926 23:30:23.902611 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:23.902617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:23.902728 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:23.902862 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:30:23.903368 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:23.904482 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:23.906500 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:23.907086 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:23.907220 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:23.907750 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:30:23.907884 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:30:23.907955 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:23.908189 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 23:30:23.908274 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":438,"threads_started":1}
I20260926 23:30:23.920804 2448 raft_consensus.cc:1261] T 7d31fb5b68f44ba89e29409391a08943 P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 1 FOLLOWER]: Refusing update from remote peer 58372f3855154f02b1652297c709ac56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:30:23.920849 2588 raft_consensus.cc:1261] T 7d31fb5b68f44ba89e29409391a08943 P b45c671f77264096a1fbfa9e16df34dc [term 1 FOLLOWER]: Refusing update from remote peer 58372f3855154f02b1652297c709ac56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:30:23.921437 2789 consensus_queue.cc:1035] T 7d31fb5b68f44ba89e29409391a08943 P 58372f3855154f02b1652297c709ac56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b45c671f77264096a1fbfa9e16df34dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:30:23.921810 2761 consensus_queue.cc:1035] T 7d31fb5b68f44ba89e29409391a08943 P 58372f3855154f02b1652297c709ac56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cd8ead90d5c4ae6bb92ab20cc986875" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 23:30:23.947576 2766 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 23:30:25.561513 2149 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:30:25.561868 2149 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:30:25.567397 2149 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd9f94aebcc1490c842e1a85f4df9918: Sending DeleteTablet for 3 replicas of tablet 7d31fb5b68f44ba89e29409391a08943
I20260926 23:30:25.568374 2287 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d31fb5b68f44ba89e29409391a08943 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:30:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38160
I20260926 23:30:25.568548 2428 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d31fb5b68f44ba89e29409391a08943 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:30:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33052
I20260926 23:30:25.568564 2568 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d31fb5b68f44ba89e29409391a08943 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:30:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47524
I20260926 23:30:25.569289 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 2222
I20260926 23:30:25.582513 2822 tablet_replica.cc:330] T 7d31fb5b68f44ba89e29409391a08943 P 58372f3855154f02b1652297c709ac56: stopping tablet replica
I20260926 23:30:25.582713 2823 tablet_replica.cc:330] T 7d31fb5b68f44ba89e29409391a08943 P 8cd8ead90d5c4ae6bb92ab20cc986875: stopping tablet replica
I20260926 23:30:25.583005 2823 raft_consensus.cc:2229] T 7d31fb5b68f44ba89e29409391a08943 P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:30:25.583015 2822 raft_consensus.cc:2229] T 7d31fb5b68f44ba89e29409391a08943 P 58372f3855154f02b1652297c709ac56 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:30:25.583202 2824 tablet_replica.cc:330] T 7d31fb5b68f44ba89e29409391a08943 P b45c671f77264096a1fbfa9e16df34dc: stopping tablet replica
I20260926 23:30:25.583436 2822 raft_consensus.cc:2258] T 7d31fb5b68f44ba89e29409391a08943 P 58372f3855154f02b1652297c709ac56 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 26 23:30:25 dist-test-slave-gwfx krb5kdc[2106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465422, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:30:25.584358 2824 raft_consensus.cc:2229] T 7d31fb5b68f44ba89e29409391a08943 P b45c671f77264096a1fbfa9e16df34dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:30:25.584403 2823 raft_consensus.cc:2258] T 7d31fb5b68f44ba89e29409391a08943 P 8cd8ead90d5c4ae6bb92ab20cc986875 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:30:25.584673 2824 raft_consensus.cc:2258] T 7d31fb5b68f44ba89e29409391a08943 P b45c671f77264096a1fbfa9e16df34dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:30:25.584964 2822 ts_tablet_manager.cc:1905] T 7d31fb5b68f44ba89e29409391a08943 P 58372f3855154f02b1652297c709ac56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:30:25.585819 2824 ts_tablet_manager.cc:1905] T 7d31fb5b68f44ba89e29409391a08943 P b45c671f77264096a1fbfa9e16df34dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:30:25.585901 2823 ts_tablet_manager.cc:1905] T 7d31fb5b68f44ba89e29409391a08943 P 8cd8ead90d5c4ae6bb92ab20cc986875: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:30:25.592926 2824 ts_tablet_manager.cc:1918] T 7d31fb5b68f44ba89e29409391a08943 P b45c671f77264096a1fbfa9e16df34dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 23:30:25.592940 2823 ts_tablet_manager.cc:1918] T 7d31fb5b68f44ba89e29409391a08943 P 8cd8ead90d5c4ae6bb92ab20cc986875: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 23:30:25.593097 2823 log.cc:1196] T 7d31fb5b68f44ba89e29409391a08943 P 8cd8ead90d5c4ae6bb92ab20cc986875: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-1/wal/wals/7d31fb5b68f44ba89e29409391a08943
I20260926 23:30:25.593137 2822 ts_tablet_manager.cc:1918] T 7d31fb5b68f44ba89e29409391a08943 P 58372f3855154f02b1652297c709ac56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 23:30:25.593226 2824 log.cc:1196] T 7d31fb5b68f44ba89e29409391a08943 P b45c671f77264096a1fbfa9e16df34dc: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-2/wal/wals/7d31fb5b68f44ba89e29409391a08943
I20260926 23:30:25.593273 2822 log.cc:1196] T 7d31fb5b68f44ba89e29409391a08943 P 58372f3855154f02b1652297c709ac56: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790465400694755-388-0/minicluster-data/ts-0/wal/wals/7d31fb5b68f44ba89e29409391a08943
I20260926 23:30:25.593794 2823 ts_tablet_manager.cc:1939] T 7d31fb5b68f44ba89e29409391a08943 P 8cd8ead90d5c4ae6bb92ab20cc986875: Deleting consensus metadata
I20260926 23:30:25.593801 2822 ts_tablet_manager.cc:1939] T 7d31fb5b68f44ba89e29409391a08943 P 58372f3855154f02b1652297c709ac56: Deleting consensus metadata
I20260926 23:30:25.593899 2824 ts_tablet_manager.cc:1939] T 7d31fb5b68f44ba89e29409391a08943 P b45c671f77264096a1fbfa9e16df34dc: Deleting consensus metadata
I20260926 23:30:25.594331 2825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:25.570363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38182 (local address 127.0.97.1:33921)
0926 23:30:25.572863 (+ 2500us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:25.572870 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:25.572896 (+ 26us) server_negotiation.cc:407] Connection header received
0926 23:30:25.572969 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:25.572975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:25.573084 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:25.575092 (+ 2008us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:25.575275 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:25.576594 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:25.578677 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:25.579585 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:25.582413 (+ 2828us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:25.585400 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:25.585426 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:25.585455 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:25.585503 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:25.589564 (+ 4061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:25.590554 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:25.590564 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:25.590568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:25.590816 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:25.591280 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:25.591285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:25.591288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:25.591771 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:25.593219 (+ 1448us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:25.593413 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 23:30:25.593677 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2356,"thread_start_us":2216,"threads_started":1}
I20260926 23:30:25.595184 2136 catalog_manager.cc:4841] TS 8cd8ead90d5c4ae6bb92ab20cc986875 (127.0.97.2:46071): tablet 7d31fb5b68f44ba89e29409391a08943 (table test-table [id=f592e39667ab40e88e9090f906b9bae8]) successfully deleted
I20260926 23:30:25.595184 2135 catalog_manager.cc:4841] TS b45c671f77264096a1fbfa9e16df34dc (127.0.97.3:36383): tablet 7d31fb5b68f44ba89e29409391a08943 (table test-table [id=f592e39667ab40e88e9090f906b9bae8]) successfully deleted
I20260926 23:30:25.595250 2138 catalog_manager.cc:4841] TS 58372f3855154f02b1652297c709ac56 (127.0.97.1:33921): tablet 7d31fb5b68f44ba89e29409391a08943 (table test-table [id=f592e39667ab40e88e9090f906b9bae8]) successfully deleted
W20260926 23:30:25.595492 2353 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38182
W20260926 23:30:25.596282 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 2222: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:30:25.596562 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 2222
I20260926 23:30:25.616209 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 2362
Sep 26 23:30:25 dist-test-slave-gwfx krb5kdc[2106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465422, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:30:25.634101 2827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:25.617215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33074 (local address 127.0.97.2:46071)
0926 23:30:25.617838 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:25.617844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:25.617863 (+ 19us) server_negotiation.cc:407] Connection header received
0926 23:30:25.617938 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:25.617942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:25.618015 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:25.618133 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:25.619627 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:25.620513 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:25.621900 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:25.622394 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:25.622512 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:25.627034 (+ 4522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:25.627056 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:25.627079 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:25.627141 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:25.630427 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:25.631386 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:25.631395 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:25.631398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:25.631552 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:25.632062 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:25.632067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:25.632107 (+ 40us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:25.632678 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:25.633096 (+ 418us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:25.633433 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 23:30:25.633683 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":344,"threads_started":1}
W20260926 23:30:25.635041 2488 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33074
W20260926 23:30:25.635931 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 2362: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:30:25.636094 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 2362
I20260926 23:30:25.655234 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 2502
Sep 26 23:30:25 dist-test-slave-gwfx krb5kdc[2106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465422, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:30:25.673568 2828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:25.656098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47548 (local address 127.0.97.3:36383)
0926 23:30:25.656707 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:25.656712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:25.656733 (+ 21us) server_negotiation.cc:407] Connection header received
0926 23:30:25.656802 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:25.656807 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:25.656884 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:25.657006 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:25.658664 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:25.659582 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:25.660996 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:25.661481 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:25.661616 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:25.666878 (+ 5262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:25.666901 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:25.666924 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:25.666978 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:25.670235 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:25.671262 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:25.671272 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:25.671275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:25.671452 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:25.671900 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:25.671903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:25.671905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:25.672353 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:25.672602 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:25.672948 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 23:30:25.673175 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":340,"threads_started":1}
W20260926 23:30:25.674593 2633 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47548
W20260926 23:30:25.675495 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 2502: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:30:25.675770 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 2502
I20260926 23:30:25.694756 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 2119
I20260926 23:30:25.696233 2191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:30:25.828783 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 2119
2026-09-26T23:30:25Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9426 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:30:25 dist-test-slave-gwfx krb5kdc[2832](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:30:25 dist-test-slave-gwfx krb5kdc[2832](info): set up 2 sockets
Sep 26 23:30:25 dist-test-slave-gwfx krb5kdc[2832](info): commencing operation
krb5kdc: starting...
W20260926 23:30:26.863222 388 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:30:26 dist-test-slave-gwfx krb5kdc[2832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:30:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:30:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:30:27.080171 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35473
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:30:27.472537 2845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:27.472935 2845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:27.473034 2845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:27.480962 2845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:30:27.481119 2845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:27.481168 2845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:27.481220 2845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:30:27.481258 2845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:30:27.493223 2845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:27.495029 2845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:27.496552 2845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:27.510278 2851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:27.510299 2853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:27.511076 2850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:27.511360 2845 server_base.cc:1034] running on GCE node
I20260926 23:30:27.512331 2845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:27.514182 2845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:27.515496 2845 hybrid_clock.cc:648] HybridClock initialized: now 1790465427515416 us; error 76 us; skew 500 ppm
I20260926 23:30:27.515980 2845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:27 dist-test-slave-gwfx krb5kdc[2832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:27.523847 2845 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:30:27.525812 2845 webserver.cc:460] Webserver started at http://127.0.97.62:42203/ using document root <none> and password file <none>
I20260926 23:30:27.526722 2845 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:27.526849 2845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:27.527179 2845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:27.530535 2845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "14cbe408e4c7411e91100eb8968ed624"
format_stamp: "Formatted at 2026-09-26 23:30:27 on dist-test-slave-gwfx"
server_key: "4e220fb044d404305f5406e86d044987"
server_key_iv: "f49f2ff8d16f54c2ecf8304c0ee2922f"
server_key_version: "encryptionkey@0"
I20260926 23:30:27.531438 2845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "14cbe408e4c7411e91100eb8968ed624"
format_stamp: "Formatted at 2026-09-26 23:30:27 on dist-test-slave-gwfx"
server_key: "4e220fb044d404305f5406e86d044987"
server_key_iv: "f49f2ff8d16f54c2ecf8304c0ee2922f"
server_key_version: "encryptionkey@0"
I20260926 23:30:27.537528 2845 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260926 23:30:27.541891 2860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:27.543826 2845 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 23:30:27.544059 2845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "14cbe408e4c7411e91100eb8968ed624"
format_stamp: "Formatted at 2026-09-26 23:30:27 on dist-test-slave-gwfx"
server_key: "4e220fb044d404305f5406e86d044987"
server_key_iv: "f49f2ff8d16f54c2ecf8304c0ee2922f"
server_key_version: "encryptionkey@0"
I20260926 23:30:27.544296 2845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:27.561898 2845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:27.567113 2845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:27.567415 2845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:27.588256 2845 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35473
I20260926 23:30:27.588284 2911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35473 every 8 connection(s)
I20260926 23:30:27.589963 2845 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:30:27.592990 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 2845
I20260926 23:30:27.593233 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:30:27.593907 388 external_mini_cluster.cc:1428] Setting key 6408259a6efe2e1a757e2cc2472e63ad
I20260926 23:30:27.594826 2912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:27.610877 2912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14cbe408e4c7411e91100eb8968ed624: Bootstrap starting.
Sep 26 23:30:27 dist-test-slave-gwfx krb5kdc[2832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:27.614863 2912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14cbe408e4c7411e91100eb8968ed624: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:27.616573 2912 log.cc:824] T 00000000000000000000000000000000 P 14cbe408e4c7411e91100eb8968ed624: Log is configured to *not* fsync() on all Append() calls
I20260926 23:30:27.619947 2912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14cbe408e4c7411e91100eb8968ed624: No bootstrap required, opened a new log
I20260926 23:30:27.623091 2915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:27.596422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45286 (local address 127.0.97.62:35473)
0926 23:30:27.597386 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:27.597402 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:27.597454 (+ 52us) server_negotiation.cc:407] Connection header received
0926 23:30:27.600044 (+ 2590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:27.600079 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:27.600679 (+ 600us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:27.601169 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:27.603291 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:27.604840 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:27.606795 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:27.607693 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:27.607924 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:27.613903 (+ 5979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:27.613916 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:27.613939 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:27.613998 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:27.617778 (+ 3780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:27.618767 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:27.618776 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:27.618785 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:27.618986 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:27.619390 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:27.619394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:27.619396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:27.620104 (+ 708us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:27.620390 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:27.621128 (+ 738us) server_negotiation.cc:299] Negotiation successful
0926 23:30:27.621641 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":437,"threads_started":1}
I20260926 23:30:27.629084 2912 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14cbe408e4c7411e91100eb8968ed624 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cbe408e4c7411e91100eb8968ed624" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35473 } }
I20260926 23:30:27.629626 2912 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14cbe408e4c7411e91100eb8968ed624 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:27.629721 2912 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14cbe408e4c7411e91100eb8968ed624 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14cbe408e4c7411e91100eb8968ed624, State: Initialized, Role: FOLLOWER
I20260926 23:30:27.630434 2912 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14cbe408e4c7411e91100eb8968ed624 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cbe408e4c7411e91100eb8968ed624" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35473 } }
I20260926 23:30:27.630673 2912 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14cbe408e4c7411e91100eb8968ed624 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:30:27.630861 2912 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14cbe408e4c7411e91100eb8968ed624 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:30:27.631000 2912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14cbe408e4c7411e91100eb8968ed624 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:27.633431 2912 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14cbe408e4c7411e91100eb8968ed624 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cbe408e4c7411e91100eb8968ed624" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35473 } }
I20260926 23:30:27.634065 2912 leader_election.cc:302] T 00000000000000000000000000000000 P 14cbe408e4c7411e91100eb8968ed624 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14cbe408e4c7411e91100eb8968ed624; no voters:
I20260926 23:30:27.635067 2912 leader_election.cc:288] T 00000000000000000000000000000000 P 14cbe408e4c7411e91100eb8968ed624 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:30:27.635183 2917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14cbe408e4c7411e91100eb8968ed624 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:30:27.636531 2917 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14cbe408e4c7411e91100eb8968ed624 [term 1 LEADER]: Becoming Leader. State: Replica: 14cbe408e4c7411e91100eb8968ed624, State: Running, Role: LEADER
I20260926 23:30:27.637282 2917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14cbe408e4c7411e91100eb8968ed624 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cbe408e4c7411e91100eb8968ed624" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35473 } }
I20260926 23:30:27.637619 2912 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14cbe408e4c7411e91100eb8968ed624 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:30:27.640908 2918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14cbe408e4c7411e91100eb8968ed624 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14cbe408e4c7411e91100eb8968ed624" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cbe408e4c7411e91100eb8968ed624" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35473 } } }
I20260926 23:30:27.641476 2918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14cbe408e4c7411e91100eb8968ed624 [sys.catalog]: This master's current role is: LEADER
I20260926 23:30:27.640926 2919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14cbe408e4c7411e91100eb8968ed624 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14cbe408e4c7411e91100eb8968ed624. Latest consensus state: current_term: 1 leader_uuid: "14cbe408e4c7411e91100eb8968ed624" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cbe408e4c7411e91100eb8968ed624" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35473 } } }
I20260926 23:30:27.643533 2919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14cbe408e4c7411e91100eb8968ed624 [sys.catalog]: This master's current role is: LEADER
I20260926 23:30:27.651366 2925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:30:27.657959 2925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:30:27.668406 2925 catalog_manager.cc:1347] Generated new cluster ID: 21ef3c291c384f9b83994ed5cbf784a4
I20260926 23:30:27.668563 2925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:30:27.699218 2925 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:30:27.701655 2925 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:30:27.716830 2925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14cbe408e4c7411e91100eb8968ed624: Generated new TSK 0
I20260926 23:30:27.718205 2925 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:30:27.779772 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35473
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:30:28.166059 2938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:28.166463 2938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:28.166601 2938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:28.174819 2938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:28.174975 2938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:28.175136 2938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:30:28.186239 2938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:28.187963 2938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:28.189643 2938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:28.206568 2944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:28.206597 2946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:28.206650 2943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:28.213032 2938 server_base.cc:1034] running on GCE node
I20260926 23:30:28.213944 2938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:28.215299 2938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:28.216544 2938 hybrid_clock.cc:648] HybridClock initialized: now 1790465428216498 us; error 47 us; skew 500 ppm
I20260926 23:30:28.216976 2938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:28 dist-test-slave-gwfx krb5kdc[2832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:28.220933 2938 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:30:28.222965 2938 webserver.cc:460] Webserver started at http://127.0.97.1:36981/ using document root <none> and password file <none>
I20260926 23:30:28.223860 2938 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:28.223976 2938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:28.224339 2938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:28.227877 2938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/data/instance:
uuid: "1e131fbdd9b34f6895ac9e464ccf0b07"
format_stamp: "Formatted at 2026-09-26 23:30:28 on dist-test-slave-gwfx"
server_key: "77ffce9c22c6e087866d14561284f9d9"
server_key_iv: "b90ee9d09be136aef18d7b811b5238d2"
server_key_version: "encryptionkey@0"
I20260926 23:30:28.228742 2938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1e131fbdd9b34f6895ac9e464ccf0b07"
format_stamp: "Formatted at 2026-09-26 23:30:28 on dist-test-slave-gwfx"
server_key: "77ffce9c22c6e087866d14561284f9d9"
server_key_iv: "b90ee9d09be136aef18d7b811b5238d2"
server_key_version: "encryptionkey@0"
I20260926 23:30:28.235036 2938 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 23:30:28.239567 2953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:28.241444 2938 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 23:30:28.241650 2938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "1e131fbdd9b34f6895ac9e464ccf0b07"
format_stamp: "Formatted at 2026-09-26 23:30:28 on dist-test-slave-gwfx"
server_key: "77ffce9c22c6e087866d14561284f9d9"
server_key_iv: "b90ee9d09be136aef18d7b811b5238d2"
server_key_version: "encryptionkey@0"
I20260926 23:30:28.241907 2938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:28.279193 2938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:28.284236 2938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:28.284571 2938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:28.285683 2938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:30:28.287292 2938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:30:28.287369 2938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:28.287490 2938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:30:28.287545 2938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:28.322575 2938 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35709
I20260926 23:30:28.322631 3065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35709 every 8 connection(s)
I20260926 23:30:28.324151 2938 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:30:28.325788 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 2938
I20260926 23:30:28.326036 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:30:28.326870 388 external_mini_cluster.cc:1428] Setting key 5dd5e4b608eccaadac473e7c38aed3f3
Sep 26 23:30:28 dist-test-slave-gwfx krb5kdc[2832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 23:30:28.350380 3068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:28.327535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33609 (local address 127.0.97.62:35473)
0926 23:30:28.328056 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:28.328063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:28.329375 (+ 1312us) server_negotiation.cc:407] Connection header received
0926 23:30:28.331606 (+ 2231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:28.331613 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:28.331710 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:28.331815 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:28.333812 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:28.334643 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:28.337542 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:28.338028 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:28.338223 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:28.343502 (+ 5279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:28.343509 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:28.343512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:28.343555 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:28.346419 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:28.347527 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:28.347533 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:28.347537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:28.347683 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:28.348172 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:28.348176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:28.348179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:28.348499 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:28.348642 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:28.349611 (+ 969us) server_negotiation.cc:299] Negotiation successful
0926 23:30:28.349963 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":257,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 23:30:28.351851 3069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:28.328303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35473 (local address 127.0.97.1:33609)
0926 23:30:28.329218 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:28.329266 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:28.331263 (+ 1997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:28.332159 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:28.332171 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:28.332835 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:28.333595 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:28.333615 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:28.334877 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:28.334883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:28.337307 (+ 2424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:28.337363 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:28.338365 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:28.338373 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:28.338791 (+ 418us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:28.339797 (+ 1006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:28.339838 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:28.343151 (+ 3313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:28.346622 (+ 3471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:28.346636 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:28.346662 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:28.347385 (+ 723us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:28.347797 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:28.347802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:28.347805 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:28.348052 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:28.348655 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:28.348666 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:28.349182 (+ 516us) client_negotiation.cc:769] Sending connection context
0926 23:30:28.349469 (+ 287us) client_negotiation.cc:240] Negotiation successful
0926 23:30:28.350155 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":345,"threads_started":1}
I20260926 23:30:28.353914 3066 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35473
I20260926 23:30:28.354346 3066 heartbeater.cc:464] Registering TS with master...
I20260926 23:30:28.355494 3066 heartbeater.cc:511] Master 127.0.97.62:35473 requested a full tablet report, sending...
I20260926 23:30:28.358865 2877 ts_manager.cc:194] Registered new tserver with Master: 1e131fbdd9b34f6895ac9e464ccf0b07 (127.0.97.1:35709)
I20260926 23:30:28.360596 2877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:30:28.379648 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35473
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:30:28.777309 3072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:28.777698 3072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:28.777829 3072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:28.785820 3072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:28.785989 3072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:28.786155 3072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:30:28.797359 3072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:28.799106 3072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:28.800722 3072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:28.815941 3078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:28.816627 3077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:28.817417 3080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:28.817970 3072 server_base.cc:1034] running on GCE node
I20260926 23:30:28.818859 3072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:28.820101 3072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:28.821357 3072 hybrid_clock.cc:648] HybridClock initialized: now 1790465428821280 us; error 67 us; skew 500 ppm
I20260926 23:30:28.821790 3072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:28 dist-test-slave-gwfx krb5kdc[2832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:28.825911 3072 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:30:28.828040 3072 webserver.cc:460] Webserver started at http://127.0.97.2:36649/ using document root <none> and password file <none>
I20260926 23:30:28.829094 3072 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:28.829267 3072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:28.829696 3072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:28.833685 3072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/data/instance:
uuid: "0a880f29bb7f429fa1f3380e9d21a525"
format_stamp: "Formatted at 2026-09-26 23:30:28 on dist-test-slave-gwfx"
server_key: "1dbb727c553daf0df3427035c5f1261f"
server_key_iv: "29c8247d28f9450077a3af0391333fde"
server_key_version: "encryptionkey@0"
I20260926 23:30:28.834780 3072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0a880f29bb7f429fa1f3380e9d21a525"
format_stamp: "Formatted at 2026-09-26 23:30:28 on dist-test-slave-gwfx"
server_key: "1dbb727c553daf0df3427035c5f1261f"
server_key_iv: "29c8247d28f9450077a3af0391333fde"
server_key_version: "encryptionkey@0"
I20260926 23:30:28.842155 3072 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260926 23:30:28.847009 3087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:28.848865 3072 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260926 23:30:28.849069 3072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "0a880f29bb7f429fa1f3380e9d21a525"
format_stamp: "Formatted at 2026-09-26 23:30:28 on dist-test-slave-gwfx"
server_key: "1dbb727c553daf0df3427035c5f1261f"
server_key_iv: "29c8247d28f9450077a3af0391333fde"
server_key_version: "encryptionkey@0"
I20260926 23:30:28.849324 3072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:28.865053 3072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:28.870317 3072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:28.870666 3072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:28.871850 3072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:30:28.873538 3072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:30:28.873623 3072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:28.873754 3072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:30:28.873817 3072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:28.907588 3072 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39499
I20260926 23:30:28.907644 3199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39499 every 8 connection(s)
I20260926 23:30:28.909090 3072 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:30:28.913815 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 3072
I20260926 23:30:28.914044 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:30:28.914733 388 external_mini_cluster.cc:1428] Setting key 379158567f178527d9685a1fefdb0c35
Sep 26 23:30:28 dist-test-slave-gwfx krb5kdc[2832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:28.934341 3202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:28.912123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47949 (local address 127.0.97.62:35473)
0926 23:30:28.912670 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:28.912677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:28.914067 (+ 1390us) server_negotiation.cc:407] Connection header received
0926 23:30:28.916543 (+ 2476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:28.916548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:28.916633 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:28.916762 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:28.919241 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:28.920067 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:28.922361 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:28.922878 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:28.922985 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:28.927351 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:28.927359 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:28.927364 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:28.927412 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:28.930571 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:28.931532 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:28.931537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:28.931540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:28.931704 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:28.932238 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:28.932243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:28.932246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:28.932658 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:28.932822 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:28.933642 (+ 820us) server_negotiation.cc:299] Negotiation successful
0926 23:30:28.933917 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":257,"threads_started":1}
I20260926 23:30:28.935477 3203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:28.912828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35473 (local address 127.0.97.2:47949)
0926 23:30:28.913787 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:28.913852 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:28.916235 (+ 2383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:28.917057 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:28.917091 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:28.917937 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:28.918987 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:28.919023 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:28.920231 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:28.920237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:28.922174 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:28.922191 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:28.923114 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:28.923121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:28.923412 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:28.924243 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:28.924272 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:28.927020 (+ 2748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:28.930799 (+ 3779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:28.930808 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:28.930828 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:28.931388 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:28.931856 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:28.931861 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:28.931864 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:28.932103 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:28.932780 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:28.932787 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:28.933215 (+ 428us) client_negotiation.cc:769] Sending connection context
0926 23:30:28.933485 (+ 270us) client_negotiation.cc:240] Negotiation successful
0926 23:30:28.934026 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":351,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 23:30:28.937227 3200 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35473
I20260926 23:30:28.937682 3200 heartbeater.cc:464] Registering TS with master...
I20260926 23:30:28.938884 3200 heartbeater.cc:511] Master 127.0.97.62:35473 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 23:30:28.940919 2877 ts_manager.cc:194] Registered new tserver with Master: 0a880f29bb7f429fa1f3380e9d21a525 (127.0.97.2:39499)
I20260926 23:30:28.941947 2877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47949
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:30:28.967453 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35473
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:30:29.364513 3066 heartbeater.cc:503] Master 127.0.97.62:35473 was elected leader, sending a full tablet report...
W20260926 23:30:29.375274 3206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:29.375664 3206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:29.375785 3206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:29.383913 3206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:29.384071 3206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:29.384202 3206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:30:29.395298 3206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:29.397040 3206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:29.398746 3206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:29.415017 3214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:29.415107 3211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:29.415126 3212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:29.416209 3206 server_base.cc:1034] running on GCE node
I20260926 23:30:29.416987 3206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:29.418193 3206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:29.419447 3206 hybrid_clock.cc:648] HybridClock initialized: now 1790465429419387 us; error 49 us; skew 500 ppm
I20260926 23:30:29.419862 3206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:29 dist-test-slave-gwfx krb5kdc[2832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:29.424072 3206 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:30:29.426321 3206 webserver.cc:460] Webserver started at http://127.0.97.3:46025/ using document root <none> and password file <none>
I20260926 23:30:29.427385 3206 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:29.427529 3206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:29.427894 3206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:29.431671 3206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/data/instance:
uuid: "189fe573992644d69740d00bedc33adc"
format_stamp: "Formatted at 2026-09-26 23:30:29 on dist-test-slave-gwfx"
server_key: "74ff865b78266cc734179a16920480c8"
server_key_iv: "07f1ef7c07b04bebffd8342e60c18b01"
server_key_version: "encryptionkey@0"
I20260926 23:30:29.432657 3206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/wal/instance:
uuid: "189fe573992644d69740d00bedc33adc"
format_stamp: "Formatted at 2026-09-26 23:30:29 on dist-test-slave-gwfx"
server_key: "74ff865b78266cc734179a16920480c8"
server_key_iv: "07f1ef7c07b04bebffd8342e60c18b01"
server_key_version: "encryptionkey@0"
I20260926 23:30:29.439759 3206 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260926 23:30:29.444861 3221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:29.446821 3206 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260926 23:30:29.447018 3206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "189fe573992644d69740d00bedc33adc"
format_stamp: "Formatted at 2026-09-26 23:30:29 on dist-test-slave-gwfx"
server_key: "74ff865b78266cc734179a16920480c8"
server_key_iv: "07f1ef7c07b04bebffd8342e60c18b01"
server_key_version: "encryptionkey@0"
I20260926 23:30:29.447242 3206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:29.479978 3206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:29.485304 3206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:29.485575 3206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:29.486652 3206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:30:29.488251 3206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:30:29.488334 3206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:29.488559 3206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:30:29.488715 3206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:29.525930 3206 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44105
I20260926 23:30:29.526134 3333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44105 every 8 connection(s)
I20260926 23:30:29.527878 3206 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:30:29.532702 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 3206
I20260926 23:30:29.532931 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:30:29.533473 388 external_mini_cluster.cc:1428] Setting key 5ed5ac71520c46ed1e3db03cb82eaae2
Sep 26 23:30:29 dist-test-slave-gwfx krb5kdc[2832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:29.554631 3336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:29.531441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40923 (local address 127.0.97.62:35473)
0926 23:30:29.532107 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:29.532112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:29.533318 (+ 1206us) server_negotiation.cc:407] Connection header received
0926 23:30:29.535899 (+ 2581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:29.535905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:29.536055 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:29.536166 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:29.538572 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:29.539486 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:29.541477 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:29.541939 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:29.542066 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:29.546719 (+ 4653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:29.546728 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:29.546733 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:29.546789 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:29.550323 (+ 3534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:29.551446 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:29.551453 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:29.551455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:29.551727 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:29.552337 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:29.552343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:29.552346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:29.552779 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:29.552940 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:29.553900 (+ 960us) server_negotiation.cc:299] Negotiation successful
0926 23:30:29.554191 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":382,"threads_started":1}
I20260926 23:30:29.555744 3337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:29.531977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35473 (local address 127.0.97.3:40923)
0926 23:30:29.533086 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:29.533155 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:29.535501 (+ 2346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:29.536539 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:29.536554 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:29.537510 (+ 956us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:29.538345 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:29.538370 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:29.539664 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:29.539669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:29.541205 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:29.541220 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:29.542230 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:29.542236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:29.542521 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:29.543434 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:29.543462 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:29.546317 (+ 2855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:29.550561 (+ 4244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:29.550572 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:29.550594 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:29.551224 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:29.551868 (+ 644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:29.551873 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:29.551876 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:29.552164 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:29.552974 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:29.552985 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:29.553429 (+ 444us) client_negotiation.cc:769] Sending connection context
0926 23:30:29.553769 (+ 340us) client_negotiation.cc:240] Negotiation successful
0926 23:30:29.554412 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":442,"threads_started":1}
I20260926 23:30:29.557505 3334 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35473
I20260926 23:30:29.557928 3334 heartbeater.cc:464] Registering TS with master...
I20260926 23:30:29.558941 3334 heartbeater.cc:511] Master 127.0.97.62:35473 requested a full tablet report, sending...
I20260926 23:30:29.560979 2877 ts_manager.cc:194] Registered new tserver with Master: 189fe573992644d69740d00bedc33adc (127.0.97.3:44105)
I20260926 23:30:29.562013 2877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40923
I20260926 23:30:29.562857 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 23:30:29.601281 3336 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:30:29.598308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45298 (local address 127.0.97.62:35473)
0926 23:30:29.598530 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:29.598535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:29.598921 (+ 386us) server_negotiation.cc:407] Connection header received
0926 23:30:29.599111 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:29.599115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:29.599221 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:29.599351 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:29.601037 (+ 1686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45298: BlockingRecv error: recv got EOF from 127.0.0.1:45298 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 23:30:29.604550 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 2938
W20260926 23:30:29.608122 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 2938: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:35709: server requires authentication, but client does not have Kerberos credentials available
I20260926 23:30:29.608347 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 2938
I20260926 23:30:29.623900 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 3072
W20260926 23:30:29.627528 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 3072: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:39499: server requires authentication, but client does not have Kerberos credentials available
I20260926 23:30:29.627727 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 3072
W20260926 23:30:29.627816 3347 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:30:29.625201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37618 (local address 127.0.97.2:39499)
0926 23:30:29.625952 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:29.625961 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:29.626016 (+ 55us) server_negotiation.cc:407] Connection header received
0926 23:30:29.626089 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:29.626096 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:29.626345 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:29.626544 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:29.627425 (+ 881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37618: BlockingRecv error: recv got EOF from 127.0.0.1:37618 (error 108)
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":385,"threads_started":1}
I20260926 23:30:29.642668 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 3206
W20260926 23:30:29.646535 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 3206: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:44105: server requires authentication, but client does not have Kerberos credentials available
I20260926 23:30:29.646735 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 3206
I20260926 23:30:29.662004 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 2845
I20260926 23:30:29.663844 2907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:30:29.781939 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 2845
2026-09-26T23:30:29Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3952 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:30:29 dist-test-slave-gwfx krb5kdc[3352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:30:29 dist-test-slave-gwfx krb5kdc[3352](info): set up 2 sockets
Sep 26 23:30:29 dist-test-slave-gwfx krb5kdc[3352](info): commencing operation
krb5kdc: starting...
W20260926 23:30:30.815301 388 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.005s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:30:30 dist-test-slave-gwfx krb5kdc[3352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:30:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:30:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:30:31.043689 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40379
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:30:31.436169 3365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:31.436609 3365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:31.436754 3365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:31.444970 3365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:30:31.445163 3365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:31.445251 3365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:31.445338 3365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:30:31.445407 3365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:30:31.457011 3365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:31.458894 3365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:31.460566 3365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:31.475759 3370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:31.475695 3371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:31.475778 3373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:31.476552 3365 server_base.cc:1034] running on GCE node
I20260926 23:30:31.477581 3365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:31.479671 3365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:31.480931 3365 hybrid_clock.cc:648] HybridClock initialized: now 1790465431480877 us; error 60 us; skew 500 ppm
I20260926 23:30:31.481386 3365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:31 dist-test-slave-gwfx krb5kdc[3352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:31.485173 3365 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:30:31.487368 3365 webserver.cc:460] Webserver started at http://127.0.97.62:44011/ using document root <none> and password file <none>
I20260926 23:30:31.488251 3365 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:31.488374 3365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:31.488693 3365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:31.492626 3365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "3dc445e41eb64683af52961d2c60d275"
format_stamp: "Formatted at 2026-09-26 23:30:31 on dist-test-slave-gwfx"
server_key: "6a887c376a4989b51edaf195cce749e2"
server_key_iv: "9947373abf7843d07dea2e6c78369fb9"
server_key_version: "encryptionkey@0"
I20260926 23:30:31.493585 3365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "3dc445e41eb64683af52961d2c60d275"
format_stamp: "Formatted at 2026-09-26 23:30:31 on dist-test-slave-gwfx"
server_key: "6a887c376a4989b51edaf195cce749e2"
server_key_iv: "9947373abf7843d07dea2e6c78369fb9"
server_key_version: "encryptionkey@0"
I20260926 23:30:31.500635 3365 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260926 23:30:31.505502 3380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:31.507578 3365 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 23:30:31.507831 3365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "3dc445e41eb64683af52961d2c60d275"
format_stamp: "Formatted at 2026-09-26 23:30:31 on dist-test-slave-gwfx"
server_key: "6a887c376a4989b51edaf195cce749e2"
server_key_iv: "9947373abf7843d07dea2e6c78369fb9"
server_key_version: "encryptionkey@0"
I20260926 23:30:31.508101 3365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:31.522307 3365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:31.527495 3365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:31.527801 3365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:31.548101 3365 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40379
I20260926 23:30:31.548166 3431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40379 every 8 connection(s)
I20260926 23:30:31.549846 3365 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:30:31.555138 3432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:31.558218 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 3365
I20260926 23:30:31.558416 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:30:31.559120 388 external_mini_cluster.cc:1428] Setting key 40a2561d4063a39f34f0dbbfe6cd63c8
I20260926 23:30:31.570392 3432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dc445e41eb64683af52961d2c60d275: Bootstrap starting.
I20260926 23:30:31.574856 3432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3dc445e41eb64683af52961d2c60d275: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:31.576098 3432 log.cc:824] T 00000000000000000000000000000000 P 3dc445e41eb64683af52961d2c60d275: Log is configured to *not* fsync() on all Append() calls
Sep 26 23:30:31 dist-test-slave-gwfx krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:31.580677 3432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dc445e41eb64683af52961d2c60d275: No bootstrap required, opened a new log
I20260926 23:30:31.588001 3435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:31.561626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45878 (local address 127.0.97.62:40379)
0926 23:30:31.562599 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:31.562618 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:31.562678 (+ 60us) server_negotiation.cc:407] Connection header received
0926 23:30:31.565194 (+ 2516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:31.565232 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:31.565913 (+ 681us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:31.566474 (+ 561us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:31.568114 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:31.570299 (+ 2185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:31.571917 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:31.573129 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:31.573340 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:31.577897 (+ 4557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:31.577911 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:31.577941 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:31.577991 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:31.582143 (+ 4152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:31.583121 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:31.583132 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:31.583148 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:31.583391 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:31.583842 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:31.583848 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:31.583851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:31.584780 (+ 929us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:31.585127 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:31.585646 (+ 519us) server_negotiation.cc:299] Negotiation successful
0926 23:30:31.586200 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":408,"threads_started":1}
I20260926 23:30:31.591867 3432 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3dc445e41eb64683af52961d2c60d275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc445e41eb64683af52961d2c60d275" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40379 } }
I20260926 23:30:31.592348 3432 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3dc445e41eb64683af52961d2c60d275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:31.592469 3432 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3dc445e41eb64683af52961d2c60d275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc445e41eb64683af52961d2c60d275, State: Initialized, Role: FOLLOWER
I20260926 23:30:31.593146 3432 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3dc445e41eb64683af52961d2c60d275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc445e41eb64683af52961d2c60d275" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40379 } }
I20260926 23:30:31.593415 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:30:31.593442 3432 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3dc445e41eb64683af52961d2c60d275 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:30:31.593566 3432 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3dc445e41eb64683af52961d2c60d275 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:30:31.593782 3432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3dc445e41eb64683af52961d2c60d275 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:31.596374 3432 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3dc445e41eb64683af52961d2c60d275 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc445e41eb64683af52961d2c60d275" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40379 } }
I20260926 23:30:31.597023 3432 leader_election.cc:302] T 00000000000000000000000000000000 P 3dc445e41eb64683af52961d2c60d275 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3dc445e41eb64683af52961d2c60d275; no voters:
I20260926 23:30:31.597716 3432 leader_election.cc:288] T 00000000000000000000000000000000 P 3dc445e41eb64683af52961d2c60d275 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:30:31.597910 3437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3dc445e41eb64683af52961d2c60d275 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:30:31.599063 3437 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3dc445e41eb64683af52961d2c60d275 [term 1 LEADER]: Becoming Leader. State: Replica: 3dc445e41eb64683af52961d2c60d275, State: Running, Role: LEADER
I20260926 23:30:31.599819 3437 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3dc445e41eb64683af52961d2c60d275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc445e41eb64683af52961d2c60d275" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40379 } }
I20260926 23:30:31.600378 3432 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3dc445e41eb64683af52961d2c60d275 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:30:31.604496 3439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dc445e41eb64683af52961d2c60d275 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3dc445e41eb64683af52961d2c60d275. Latest consensus state: current_term: 1 leader_uuid: "3dc445e41eb64683af52961d2c60d275" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc445e41eb64683af52961d2c60d275" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40379 } } }
I20260926 23:30:31.604961 3439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dc445e41eb64683af52961d2c60d275 [sys.catalog]: This master's current role is: LEADER
I20260926 23:30:31.605633 3438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dc445e41eb64683af52961d2c60d275 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3dc445e41eb64683af52961d2c60d275" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc445e41eb64683af52961d2c60d275" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40379 } } }
I20260926 23:30:31.605962 3438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dc445e41eb64683af52961d2c60d275 [sys.catalog]: This master's current role is: LEADER
I20260926 23:30:31.615365 3445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:30:31.622555 3445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:30:31.642200 3445 catalog_manager.cc:1347] Generated new cluster ID: 75f75b14d5844783b49a70de8a64a28b
I20260926 23:30:31.642567 3445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:30:31.673694 3445 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:30:31.676182 3445 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:30:31.685545 3435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:31.666325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45894 (local address 127.0.97.62:40379)
0926 23:30:31.666613 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:31.666621 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:31.673507 (+ 6886us) server_negotiation.cc:407] Connection header received
0926 23:30:31.674136 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:31.674146 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:31.674259 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:31.674346 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:31.676245 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:31.677641 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:31.679966 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:31.680831 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:31.680957 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:31.682968 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:31.682978 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:31.682984 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:30:31.684250 (+ 1266us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:31.684362 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:30:31.684415 (+ 53us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:31.684535 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:31.684807 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 23:30:31.685026 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":109}
I20260926 23:30:31.688586 3445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3dc445e41eb64683af52961d2c60d275: Generated new TSK 0
I20260926 23:30:31.689770 3445 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:30:31.701072 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 3365
I20260926 23:30:31.702425 3427 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:30:31.801276 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 3365
2026-09-26T23:30:31Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2012 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:30:31 dist-test-slave-gwfx krb5kdc[3466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:30:31 dist-test-slave-gwfx krb5kdc[3466](info): set up 2 sockets
Sep 26 23:30:31 dist-test-slave-gwfx krb5kdc[3466](info): commencing operation
krb5kdc: starting...
W20260926 23:30:32.823397 388 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:30:32 dist-test-slave-gwfx krb5kdc[3466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:30:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:30:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:30:33.047860 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38339
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38339
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:30:33.448279 3479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:33.448764 3479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:33.449100 3479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:33.459261 3479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:30:33.459451 3479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:33.459506 3479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:33.459570 3479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:30:33.459618 3479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:30:33.471657 3479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:33.473485 3479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:33.475240 3479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:33.489950 3485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:33.490192 3484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:33.490182 3487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:33.490828 3479 server_base.cc:1034] running on GCE node
I20260926 23:30:33.491904 3479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:33.493831 3479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:33.495110 3479 hybrid_clock.cc:648] HybridClock initialized: now 1790465433495041 us; error 55 us; skew 500 ppm
I20260926 23:30:33.495635 3479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:33 dist-test-slave-gwfx krb5kdc[3466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:33.505851 3479 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:30:33.507869 3479 webserver.cc:460] Webserver started at http://127.0.97.62:35911/ using document root <none> and password file <none>
I20260926 23:30:33.508737 3479 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:33.508864 3479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:33.509229 3479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:33.513019 3479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "ecdc2001d5a14cff88690004910ac717"
format_stamp: "Formatted at 2026-09-26 23:30:33 on dist-test-slave-gwfx"
server_key: "d6e9e53e8e5ed1f1207a3932776a3f5f"
server_key_iv: "69c5476e4c3d775be1ca74c5723a806e"
server_key_version: "encryptionkey@0"
I20260926 23:30:33.513935 3479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "ecdc2001d5a14cff88690004910ac717"
format_stamp: "Formatted at 2026-09-26 23:30:33 on dist-test-slave-gwfx"
server_key: "d6e9e53e8e5ed1f1207a3932776a3f5f"
server_key_iv: "69c5476e4c3d775be1ca74c5723a806e"
server_key_version: "encryptionkey@0"
I20260926 23:30:33.520570 3479 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 23:30:33.525076 3494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:33.527279 3479 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:30:33.527531 3479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "ecdc2001d5a14cff88690004910ac717"
format_stamp: "Formatted at 2026-09-26 23:30:33 on dist-test-slave-gwfx"
server_key: "d6e9e53e8e5ed1f1207a3932776a3f5f"
server_key_iv: "69c5476e4c3d775be1ca74c5723a806e"
server_key_version: "encryptionkey@0"
I20260926 23:30:33.527774 3479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:33.552299 3479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:33.557750 3479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:33.558102 3479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:33.579008 3479 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38339
I20260926 23:30:33.579020 3545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38339 every 8 connection(s)
I20260926 23:30:33.580834 3479 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:30:33.582361 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 3479
I20260926 23:30:33.582585 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:30:33.583480 388 external_mini_cluster.cc:1428] Setting key fcc3cf14a474fbdb0a5013185d401575
I20260926 23:30:33.586565 3546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 23:30:33 dist-test-slave-gwfx krb5kdc[3466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:33.601526 3546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecdc2001d5a14cff88690004910ac717: Bootstrap starting.
I20260926 23:30:33.605700 3546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecdc2001d5a14cff88690004910ac717: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:33.607046 3546 log.cc:824] T 00000000000000000000000000000000 P ecdc2001d5a14cff88690004910ac717: Log is configured to *not* fsync() on all Append() calls
I20260926 23:30:33.609931 3549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:33.585922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35468 (local address 127.0.97.62:38339)
0926 23:30:33.586830 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:33.586845 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:33.586899 (+ 54us) server_negotiation.cc:407] Connection header received
0926 23:30:33.588707 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:33.588735 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:33.589230 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:33.589657 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:33.591661 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:33.593250 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:33.595149 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:33.596091 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:33.596258 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:33.601271 (+ 5013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:33.601281 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:33.601302 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:33.601362 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:33.604845 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:33.605724 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:33.605733 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:33.605741 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:33.605974 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:33.606425 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:33.606429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:33.606431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:33.607134 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:33.607434 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:33.608061 (+ 627us) server_negotiation.cc:299] Negotiation successful
0926 23:30:33.608659 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":395,"threads_started":1}
I20260926 23:30:33.611079 3546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecdc2001d5a14cff88690004910ac717: No bootstrap required, opened a new log
I20260926 23:30:33.619787 3546 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecdc2001d5a14cff88690004910ac717 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecdc2001d5a14cff88690004910ac717" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38339 } }
I20260926 23:30:33.620410 3546 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecdc2001d5a14cff88690004910ac717 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:33.620541 3546 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecdc2001d5a14cff88690004910ac717 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecdc2001d5a14cff88690004910ac717, State: Initialized, Role: FOLLOWER
I20260926 23:30:33.621243 3546 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecdc2001d5a14cff88690004910ac717 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecdc2001d5a14cff88690004910ac717" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38339 } }
I20260926 23:30:33.621481 3546 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecdc2001d5a14cff88690004910ac717 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:30:33.621624 3546 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecdc2001d5a14cff88690004910ac717 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:30:33.621788 3546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecdc2001d5a14cff88690004910ac717 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:33.624337 3546 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecdc2001d5a14cff88690004910ac717 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecdc2001d5a14cff88690004910ac717" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38339 } }
I20260926 23:30:33.624998 3546 leader_election.cc:302] T 00000000000000000000000000000000 P ecdc2001d5a14cff88690004910ac717 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecdc2001d5a14cff88690004910ac717; no voters:
I20260926 23:30:33.625751 3546 leader_election.cc:288] T 00000000000000000000000000000000 P ecdc2001d5a14cff88690004910ac717 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:30:33.625902 3551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecdc2001d5a14cff88690004910ac717 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:30:33.626982 3551 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecdc2001d5a14cff88690004910ac717 [term 1 LEADER]: Becoming Leader. State: Replica: ecdc2001d5a14cff88690004910ac717, State: Running, Role: LEADER
I20260926 23:30:33.627650 3551 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecdc2001d5a14cff88690004910ac717 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecdc2001d5a14cff88690004910ac717" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38339 } }
I20260926 23:30:33.628479 3546 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecdc2001d5a14cff88690004910ac717 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:30:33.631393 3552 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecdc2001d5a14cff88690004910ac717 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecdc2001d5a14cff88690004910ac717" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecdc2001d5a14cff88690004910ac717" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38339 } } }
I20260926 23:30:33.632091 3552 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecdc2001d5a14cff88690004910ac717 [sys.catalog]: This master's current role is: LEADER
I20260926 23:30:33.632404 3553 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecdc2001d5a14cff88690004910ac717 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecdc2001d5a14cff88690004910ac717. Latest consensus state: current_term: 1 leader_uuid: "ecdc2001d5a14cff88690004910ac717" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecdc2001d5a14cff88690004910ac717" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38339 } } }
I20260926 23:30:33.632746 3553 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecdc2001d5a14cff88690004910ac717 [sys.catalog]: This master's current role is: LEADER
I20260926 23:30:33.634636 3558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:30:33.642706 3558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:30:33.654881 3558 catalog_manager.cc:1347] Generated new cluster ID: 1ce57b484c874c7d867dae24f9634abe
I20260926 23:30:33.655049 3558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:30:33.683887 3558 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:30:33.685560 3558 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:30:33.696053 3558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecdc2001d5a14cff88690004910ac717: Generated new TSK 0
I20260926 23:30:33.697459 3558 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:30:33.767871 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38339
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:30:34.173913 3572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:34.174355 3572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:34.174520 3572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:34.182840 3572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:34.183015 3572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:34.183188 3572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:30:34.194584 3572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:34.196496 3572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:34.198148 3572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:34.214017 3577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:34.214428 3578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:34.214651 3580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:34.215339 3572 server_base.cc:1034] running on GCE node
I20260926 23:30:34.216228 3572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:34.217500 3572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:34.218809 3572 hybrid_clock.cc:648] HybridClock initialized: now 1790465434218741 us; error 64 us; skew 500 ppm
I20260926 23:30:34.219220 3572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:34 dist-test-slave-gwfx krb5kdc[3466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:34.223999 3572 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:30:34.226189 3572 webserver.cc:460] Webserver started at http://127.0.97.1:41769/ using document root <none> and password file <none>
I20260926 23:30:34.227451 3572 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:34.227620 3572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:34.228041 3572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:34.232342 3572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/data/instance:
uuid: "c9d8ce56296d4d5ea78f01c2c998e791"
format_stamp: "Formatted at 2026-09-26 23:30:34 on dist-test-slave-gwfx"
server_key: "3ec1ec9be7dfdd1cd6d1ff4d03d9511b"
server_key_iv: "5783cac5b4244cf3b26f539ff04b1b7a"
server_key_version: "encryptionkey@0"
I20260926 23:30:34.233489 3572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c9d8ce56296d4d5ea78f01c2c998e791"
format_stamp: "Formatted at 2026-09-26 23:30:34 on dist-test-slave-gwfx"
server_key: "3ec1ec9be7dfdd1cd6d1ff4d03d9511b"
server_key_iv: "5783cac5b4244cf3b26f539ff04b1b7a"
server_key_version: "encryptionkey@0"
I20260926 23:30:34.240267 3572 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 23:30:34.244575 3587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:34.246359 3572 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260926 23:30:34.246563 3572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "c9d8ce56296d4d5ea78f01c2c998e791"
format_stamp: "Formatted at 2026-09-26 23:30:34 on dist-test-slave-gwfx"
server_key: "3ec1ec9be7dfdd1cd6d1ff4d03d9511b"
server_key_iv: "5783cac5b4244cf3b26f539ff04b1b7a"
server_key_version: "encryptionkey@0"
I20260926 23:30:34.246878 3572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:34.259950 3572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:34.264955 3572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:34.265290 3572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:34.266417 3572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:30:34.268060 3572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:30:34.268147 3572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:34.268286 3572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:30:34.268340 3572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:34.303510 3572 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42341
I20260926 23:30:34.303579 3699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42341 every 8 connection(s)
I20260926 23:30:34.305049 3572 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:30:34.314888 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 3572
I20260926 23:30:34.315107 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:30:34.315827 388 external_mini_cluster.cc:1428] Setting key 14ebc6b1cdf5f736fcfbd56729f37b31
Sep 26 23:30:34 dist-test-slave-gwfx krb5kdc[3466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:34.331162 3702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:34.308494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55755 (local address 127.0.97.62:38339)
0926 23:30:34.309050 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:34.309056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:34.310188 (+ 1132us) server_negotiation.cc:407] Connection header received
0926 23:30:34.312422 (+ 2234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:34.312430 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:34.312537 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:34.312648 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:34.314760 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:34.315693 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:34.317885 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:34.318354 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:34.318489 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:34.323188 (+ 4699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:34.323195 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:34.323205 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:34.323248 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:34.326783 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:34.327851 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:34.327857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:34.327861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:34.328062 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:34.328659 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:34.328664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:34.328667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:34.329128 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:34.329301 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:34.330361 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0926 23:30:34.330655 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":253,"threads_started":1}
I20260926 23:30:34.332345 3703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:34.309105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38339 (local address 127.0.97.1:55755)
0926 23:30:34.309978 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:34.310032 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:34.312068 (+ 2036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:34.312955 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:34.312966 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:34.313618 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:34.314383 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:34.314411 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:34.315898 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:34.315905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:34.317675 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:34.317696 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:34.318653 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:34.318663 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:34.318973 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:34.319823 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:34.319852 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:34.322822 (+ 2970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:34.327020 (+ 4198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:34.327031 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:34.327052 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:34.327694 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:34.328221 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:34.328227 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:34.328230 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:34.328485 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:34.329287 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:34.329295 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:34.329871 (+ 576us) client_negotiation.cc:769] Sending connection context
0926 23:30:34.330197 (+ 326us) client_negotiation.cc:240] Negotiation successful
0926 23:30:34.330821 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":350,"threads_started":1}
I20260926 23:30:34.334568 3700 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38339
I20260926 23:30:34.335000 3700 heartbeater.cc:464] Registering TS with master...
I20260926 23:30:34.336023 3700 heartbeater.cc:511] Master 127.0.97.62:38339 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 23:30:34.339069 3511 ts_manager.cc:194] Registered new tserver with Master: c9d8ce56296d4d5ea78f01c2c998e791 (127.0.97.1:42341)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 23:30:34.341055 3511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:30:34.369930 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38339
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:30:34.762423 3706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:34.762856 3706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:34.763020 3706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:34.771135 3706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:34.771277 3706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:34.771437 3706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:30:34.782253 3706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:34.784010 3706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:34.785679 3706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:34.800644 3711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:34.800830 3714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:34.800837 3712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:34.801832 3706 server_base.cc:1034] running on GCE node
I20260926 23:30:34.802613 3706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:34.803807 3706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:34.805107 3706 hybrid_clock.cc:648] HybridClock initialized: now 1790465434805027 us; error 50 us; skew 500 ppm
I20260926 23:30:34.805492 3706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:34 dist-test-slave-gwfx krb5kdc[3466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:34.809350 3706 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:30:34.811362 3706 webserver.cc:460] Webserver started at http://127.0.97.2:35177/ using document root <none> and password file <none>
I20260926 23:30:34.812276 3706 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:34.812399 3706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:34.812711 3706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:34.816411 3706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/data/instance:
uuid: "62a11451e80f48239ff8b4162c7afeb8"
format_stamp: "Formatted at 2026-09-26 23:30:34 on dist-test-slave-gwfx"
server_key: "7d2f3b6b5e0831648f49b842650762dd"
server_key_iv: "a766976235af06182636460e9f584621"
server_key_version: "encryptionkey@0"
I20260926 23:30:34.817345 3706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/wal/instance:
uuid: "62a11451e80f48239ff8b4162c7afeb8"
format_stamp: "Formatted at 2026-09-26 23:30:34 on dist-test-slave-gwfx"
server_key: "7d2f3b6b5e0831648f49b842650762dd"
server_key_iv: "a766976235af06182636460e9f584621"
server_key_version: "encryptionkey@0"
I20260926 23:30:34.824640 3706 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260926 23:30:34.829592 3721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:34.831664 3706 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260926 23:30:34.831898 3706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "62a11451e80f48239ff8b4162c7afeb8"
format_stamp: "Formatted at 2026-09-26 23:30:34 on dist-test-slave-gwfx"
server_key: "7d2f3b6b5e0831648f49b842650762dd"
server_key_iv: "a766976235af06182636460e9f584621"
server_key_version: "encryptionkey@0"
I20260926 23:30:34.832177 3706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:34.866886 3706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:34.872140 3706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:34.872442 3706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:34.873651 3706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:30:34.875362 3706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:30:34.875447 3706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:34.875567 3706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:30:34.875615 3706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:34.909636 3706 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36701
I20260926 23:30:34.909657 3833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36701 every 8 connection(s)
I20260926 23:30:34.911197 3706 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:30:34.914060 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 3706
I20260926 23:30:34.914291 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:30:34.914939 388 external_mini_cluster.cc:1428] Setting key 5705114174221b4ea56392684f2d48f7
Sep 26 23:30:34 dist-test-slave-gwfx krb5kdc[3466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:34.937885 3836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:34.914335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39783 (local address 127.0.97.62:38339)
0926 23:30:34.914909 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:34.914918 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:34.916194 (+ 1276us) server_negotiation.cc:407] Connection header received
0926 23:30:34.918769 (+ 2575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:34.918774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:34.918854 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:34.918965 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:34.921547 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:34.922390 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:34.924912 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:34.925397 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:34.925530 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:34.930289 (+ 4759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:34.930296 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:34.930299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:34.930345 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:34.933350 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:34.934594 (+ 1244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:34.934600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:34.934602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:34.934793 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:34.935435 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:34.935439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:34.935442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:34.935826 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:34.935956 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:34.937162 (+ 1206us) server_negotiation.cc:299] Negotiation successful
0926 23:30:34.937426 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":244,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 23:30:34.939298 3837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:34.915108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38339 (local address 127.0.97.2:39783)
0926 23:30:34.916015 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:34.916079 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:34.918341 (+ 2262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:34.919344 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:34.919359 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:34.920268 (+ 909us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:34.921279 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:34.921309 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:34.922591 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:34.922599 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:34.924701 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:34.924722 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:34.925697 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:34.925705 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:34.926061 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:34.926919 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:34.926950 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:34.929891 (+ 2941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:34.933611 (+ 3720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:34.933625 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:34.933653 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:34.934444 (+ 791us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:34.934917 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:34.934924 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:34.934928 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:34.935267 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:34.935980 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:34.935991 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:34.936620 (+ 629us) client_negotiation.cc:769] Sending connection context
0926 23:30:34.937008 (+ 388us) client_negotiation.cc:240] Negotiation successful
0926 23:30:34.937721 (+ 713us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":359,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 23:30:34.941152 3834 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38339
I20260926 23:30:34.941628 3834 heartbeater.cc:464] Registering TS with master...
I20260926 23:30:34.942811 3834 heartbeater.cc:511] Master 127.0.97.62:38339 requested a full tablet report, sending...
I20260926 23:30:34.944938 3511 ts_manager.cc:194] Registered new tserver with Master: 62a11451e80f48239ff8b4162c7afeb8 (127.0.97.2:36701)
I20260926 23:30:34.945976 3511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39783
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:30:34.971846 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38339
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:30:35.346064 3700 heartbeater.cc:503] Master 127.0.97.62:38339 was elected leader, sending a full tablet report...
W20260926 23:30:35.372164 3840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:35.372557 3840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:35.372677 3840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:35.380860 3840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:35.381033 3840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:35.381191 3840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:30:35.393106 3840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:35.394972 3840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:35.396679 3840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:35.413480 3845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:35.413403 3848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:35.413460 3846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:35.414573 3840 server_base.cc:1034] running on GCE node
I20260926 23:30:35.415447 3840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:35.416728 3840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:35.417989 3840 hybrid_clock.cc:648] HybridClock initialized: now 1790465435417923 us; error 60 us; skew 500 ppm
I20260926 23:30:35.418437 3840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:35 dist-test-slave-gwfx krb5kdc[3466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:35.422495 3840 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:30:35.424587 3840 webserver.cc:460] Webserver started at http://127.0.97.3:42349/ using document root <none> and password file <none>
I20260926 23:30:35.425588 3840 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:35.425719 3840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:35.426060 3840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:35.429885 3840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/data/instance:
uuid: "b30b61130718497da309079329ae6ddb"
format_stamp: "Formatted at 2026-09-26 23:30:35 on dist-test-slave-gwfx"
server_key: "dd1d065c43edb746223a01b309270b45"
server_key_iv: "6c4238d993471303ea458e9b78667d9e"
server_key_version: "encryptionkey@0"
I20260926 23:30:35.431010 3840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b30b61130718497da309079329ae6ddb"
format_stamp: "Formatted at 2026-09-26 23:30:35 on dist-test-slave-gwfx"
server_key: "dd1d065c43edb746223a01b309270b45"
server_key_iv: "6c4238d993471303ea458e9b78667d9e"
server_key_version: "encryptionkey@0"
I20260926 23:30:35.438537 3840 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260926 23:30:35.443432 3855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:35.445621 3840 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 23:30:35.445891 3840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "b30b61130718497da309079329ae6ddb"
format_stamp: "Formatted at 2026-09-26 23:30:35 on dist-test-slave-gwfx"
server_key: "dd1d065c43edb746223a01b309270b45"
server_key_iv: "6c4238d993471303ea458e9b78667d9e"
server_key_version: "encryptionkey@0"
I20260926 23:30:35.446163 3840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:35.481391 3840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:35.487279 3840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:35.487632 3840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:35.488862 3840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:30:35.490651 3840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:30:35.490774 3840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 23:30:35.490912 3840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:30:35.490965 3840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:35.532604 3840 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37981
I20260926 23:30:35.532675 3967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37981 every 8 connection(s)
I20260926 23:30:35.534614 3840 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:30:35.537711 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 3840
I20260926 23:30:35.537923 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790465400694755-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:30:35.538486 388 external_mini_cluster.cc:1428] Setting key f7372c7669c79d6c08102b99230d216f
Sep 26 23:30:35 dist-test-slave-gwfx krb5kdc[3466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:35.558329 3970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:35.537948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45177 (local address 127.0.97.62:38339)
0926 23:30:35.539135 (+ 1187us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:35.539142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:35.540147 (+ 1005us) server_negotiation.cc:407] Connection header received
0926 23:30:35.542288 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:35.542293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:35.542391 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:35.542500 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:35.544437 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:35.545274 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:35.547045 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:35.547493 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:35.547618 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:35.551738 (+ 4120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:35.551744 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:35.551748 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:35.551789 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:35.554902 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:35.555706 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:35.555711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:35.555714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:35.555841 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:35.556305 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:35.556309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:35.556311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:35.556635 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:35.556774 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:35.557614 (+ 840us) server_negotiation.cc:299] Negotiation successful
0926 23:30:35.557884 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1048,"thread_start_us":904,"threads_started":1}
I20260926 23:30:35.559629 3971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:35.538661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38339 (local address 127.0.97.3:45177)
0926 23:30:35.539951 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:35.540010 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:35.541937 (+ 1927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:35.542849 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:35.542861 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:35.543512 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:35.544239 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:35.544261 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:35.545448 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:35.545453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:35.546903 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:35.546918 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:35.547739 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:35.547746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:35.547995 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:35.548739 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:35.548774 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:35.551433 (+ 2659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:35.555055 (+ 3622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:35.555066 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:35.555093 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:35.555571 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:35.555956 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:35.555962 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:35.555966 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:35.556187 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:35.556758 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:35.556767 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:35.557207 (+ 440us) client_negotiation.cc:769] Sending connection context
0926 23:30:35.557485 (+ 278us) client_negotiation.cc:240] Negotiation successful
0926 23:30:35.558142 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":916,"thread_start_us":671,"threads_started":1}
I20260926 23:30:35.561689 3968 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38339
I20260926 23:30:35.562153 3968 heartbeater.cc:464] Registering TS with master...
I20260926 23:30:35.563144 3968 heartbeater.cc:511] Master 127.0.97.62:38339 requested a full tablet report, sending...
I20260926 23:30:35.565212 3511 ts_manager.cc:194] Registered new tserver with Master: b30b61130718497da309079329ae6ddb (127.0.97.3:37981)
I20260926 23:30:35.566447 3511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45177
I20260926 23:30:35.568404 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 23:30:35 dist-test-slave-gwfx krb5kdc[3466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465435, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 23:30:35 dist-test-slave-gwfx krb5kdc[3466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465435, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:35.627537 3970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:35.609759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35472 (local address 127.0.97.62:38339)
0926 23:30:35.610005 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:35.610010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:35.610390 (+ 380us) server_negotiation.cc:407] Connection header received
0926 23:30:35.610595 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:35.610602 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:35.610737 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:35.610870 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:35.612481 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:35.613257 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:35.614998 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:35.615451 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:35.615589 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:35.620595 (+ 5006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:35.620602 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:35.620606 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:35.620662 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:35.624231 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:35.625275 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:35.625280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:35.625283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:35.625536 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:35.625951 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:35.625954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:35.625956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:35.626338 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:35.626592 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:35.627041 (+ 449us) server_negotiation.cc:299] Negotiation successful
0926 23:30:35.627276 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260926 23:30:35.628224 3511 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35472
I20260926 23:30:35.630916 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 3572
Sep 26 23:30:35 dist-test-slave-gwfx krb5kdc[3466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465435, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:30:35.650120 3980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:35.632176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44626 (local address 127.0.97.1:42341)
0926 23:30:35.633044 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:35.633055 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:35.633093 (+ 38us) server_negotiation.cc:407] Connection header received
0926 23:30:35.633161 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:35.633168 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:35.633457 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:35.633611 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:35.635051 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:35.636265 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:35.637940 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:35.638563 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:35.638739 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:35.643450 (+ 4711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:35.643479 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:35.643499 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:35.643553 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:35.646822 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:35.647803 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:35.647812 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:35.647814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:35.648007 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:35.648464 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:35.648468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:35.648470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:35.648867 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:35.649113 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:35.649474 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 23:30:35.649716 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":459,"threads_started":1}
W20260926 23:30:35.652025 3695 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44626
W20260926 23:30:35.653609 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 3572: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:30:35.653889 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 3572
I20260926 23:30:35.669763 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 3706
Sep 26 23:30:35 dist-test-slave-gwfx krb5kdc[3466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465435, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:30:35.689042 3981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:35.670958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56434 (local address 127.0.97.2:36701)
0926 23:30:35.671631 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:35.671640 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:35.671670 (+ 30us) server_negotiation.cc:407] Connection header received
0926 23:30:35.671741 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:35.671748 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:35.671992 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:35.672149 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:35.673847 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:35.675059 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:35.676615 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:35.677213 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:35.677339 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:35.682429 (+ 5090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:35.682447 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:35.682465 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:35.682512 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:35.685757 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:35.686606 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:35.686615 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:35.686617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:35.686791 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:35.687145 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:35.687151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:35.687154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:35.687536 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:35.687820 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:35.688324 (+ 504us) server_negotiation.cc:299] Negotiation successful
0926 23:30:35.688648 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":308,"threads_started":1}
W20260926 23:30:35.690869 3829 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56434
W20260926 23:30:35.692257 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 3706: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:30:35.692478 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 3706
I20260926 23:30:35.706966 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 3840
Sep 26 23:30:35 dist-test-slave-gwfx krb5kdc[3466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465435, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:30:35.726652 3982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:35.708143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49948 (local address 127.0.97.3:37981)
0926 23:30:35.708952 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:35.708959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:35.708993 (+ 34us) server_negotiation.cc:407] Connection header received
0926 23:30:35.709064 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:35.709070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:35.709299 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:35.709467 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:35.710944 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:35.712163 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:35.713711 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:35.714464 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:35.714625 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:35.719828 (+ 5203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:35.719848 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:35.719869 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:35.719923 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:35.723261 (+ 3338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:35.724230 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:35.724239 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:35.724242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:35.724401 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:35.724865 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:35.724868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:35.724871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:35.725308 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:35.725574 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:35.725942 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 23:30:35.726242 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":399,"threads_started":1}
W20260926 23:30:35.728446 3963 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49948
W20260926 23:30:35.729938 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 3840: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:30:35.730162 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 3840
I20260926 23:30:35.743454 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 3479
I20260926 23:30:35.744946 3541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:30:35.850875 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 3479
2026-09-26T23:30:35Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4055 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:30:35 dist-test-slave-gwfx krb5kdc[3986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:30:35 dist-test-slave-gwfx krb5kdc[3986](info): set up 2 sockets
Sep 26 23:30:35 dist-test-slave-gwfx krb5kdc[3986](info): commencing operation
krb5kdc: starting...
W20260926 23:30:36.879477 388 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.003s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:30:37 dist-test-slave-gwfx krb5kdc[3986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:30:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:30:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:30:37.115772 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40159
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40159
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:30:37.518157 3999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:37.518656 3999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:37.518833 3999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:37.527412 3999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:30:37.527588 3999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:37.527639 3999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:37.527704 3999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:30:37.527750 3999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:30:37.539152 3999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:37.540858 3999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:37.542443 3999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:37.556398 4007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:37.556603 4004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:37.556517 4005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:37.557227 3999 server_base.cc:1034] running on GCE node
I20260926 23:30:37.558156 3999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:37.559996 3999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:37.561223 3999 hybrid_clock.cc:648] HybridClock initialized: now 1790465437561170 us; error 48 us; skew 500 ppm
I20260926 23:30:37.561650 3999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:37 dist-test-slave-gwfx krb5kdc[3986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:37.565451 3999 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:30:37.567610 3999 webserver.cc:460] Webserver started at http://127.0.97.62:36421/ using document root <none> and password file <none>
I20260926 23:30:37.568537 3999 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:37.568681 3999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:37.569046 3999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:37.572917 3999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "91b77990864644f2a86326d4aab1aa42"
format_stamp: "Formatted at 2026-09-26 23:30:37 on dist-test-slave-gwfx"
server_key: "0046ccf5c303c511c0057bdd80498b95"
server_key_iv: "3750f9851c87e52241e41849e10e46a5"
server_key_version: "encryptionkey@0"
I20260926 23:30:37.574164 3999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "91b77990864644f2a86326d4aab1aa42"
format_stamp: "Formatted at 2026-09-26 23:30:37 on dist-test-slave-gwfx"
server_key: "0046ccf5c303c511c0057bdd80498b95"
server_key_iv: "3750f9851c87e52241e41849e10e46a5"
server_key_version: "encryptionkey@0"
I20260926 23:30:37.581434 3999 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260926 23:30:37.586333 4014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:37.589160 3999 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260926 23:30:37.589433 3999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "91b77990864644f2a86326d4aab1aa42"
format_stamp: "Formatted at 2026-09-26 23:30:37 on dist-test-slave-gwfx"
server_key: "0046ccf5c303c511c0057bdd80498b95"
server_key_iv: "3750f9851c87e52241e41849e10e46a5"
server_key_version: "encryptionkey@0"
I20260926 23:30:37.589709 3999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:37.621694 3999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:37.626981 3999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:37.627310 3999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:37.648643 3999 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40159
I20260926 23:30:37.648721 4065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40159 every 8 connection(s)
I20260926 23:30:37.650359 3999 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:30:37.651623 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 3999
I20260926 23:30:37.651862 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:30:37.652774 388 external_mini_cluster.cc:1428] Setting key 2a6ce6dfe929ef3bea2f51f7aa63a1bf
I20260926 23:30:37.655908 4066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 23:30:37 dist-test-slave-gwfx krb5kdc[3986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:37.671579 4066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91b77990864644f2a86326d4aab1aa42: Bootstrap starting.
I20260926 23:30:37.676502 4066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91b77990864644f2a86326d4aab1aa42: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:37.677855 4066 log.cc:824] T 00000000000000000000000000000000 P 91b77990864644f2a86326d4aab1aa42: Log is configured to *not* fsync() on all Append() calls
I20260926 23:30:37.681607 4069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:37.655481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40670 (local address 127.0.97.62:40159)
0926 23:30:37.656454 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:37.656474 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:37.656538 (+ 64us) server_negotiation.cc:407] Connection header received
0926 23:30:37.659101 (+ 2563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:37.659138 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:37.659839 (+ 701us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:37.660472 (+ 633us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:37.662113 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:37.664373 (+ 2260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:37.665835 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:37.667007 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:37.667241 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:37.672155 (+ 4914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:37.672170 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:37.672199 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:37.672272 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:37.676415 (+ 4143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:37.677408 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:37.677417 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:37.677426 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:37.677646 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:37.678056 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:37.678060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:37.678062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:37.678798 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:37.679144 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:37.679618 (+ 474us) server_negotiation.cc:299] Negotiation successful
0926 23:30:37.680156 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":383,"threads_started":1}
I20260926 23:30:37.681813 4066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91b77990864644f2a86326d4aab1aa42: No bootstrap required, opened a new log
I20260926 23:30:37.691346 4066 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91b77990864644f2a86326d4aab1aa42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91b77990864644f2a86326d4aab1aa42" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40159 } }
I20260926 23:30:37.691918 4066 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91b77990864644f2a86326d4aab1aa42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:37.692032 4066 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91b77990864644f2a86326d4aab1aa42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91b77990864644f2a86326d4aab1aa42, State: Initialized, Role: FOLLOWER
I20260926 23:30:37.692792 4066 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91b77990864644f2a86326d4aab1aa42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91b77990864644f2a86326d4aab1aa42" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40159 } }
I20260926 23:30:37.693117 4066 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91b77990864644f2a86326d4aab1aa42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:30:37.693274 4066 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91b77990864644f2a86326d4aab1aa42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:30:37.693542 4066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91b77990864644f2a86326d4aab1aa42 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:37.696436 4066 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91b77990864644f2a86326d4aab1aa42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91b77990864644f2a86326d4aab1aa42" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40159 } }
I20260926 23:30:37.697147 4066 leader_election.cc:302] T 00000000000000000000000000000000 P 91b77990864644f2a86326d4aab1aa42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91b77990864644f2a86326d4aab1aa42; no voters:
I20260926 23:30:37.698020 4066 leader_election.cc:288] T 00000000000000000000000000000000 P 91b77990864644f2a86326d4aab1aa42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:30:37.698163 4071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91b77990864644f2a86326d4aab1aa42 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:30:37.698873 4071 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91b77990864644f2a86326d4aab1aa42 [term 1 LEADER]: Becoming Leader. State: Replica: 91b77990864644f2a86326d4aab1aa42, State: Running, Role: LEADER
I20260926 23:30:37.699457 4071 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91b77990864644f2a86326d4aab1aa42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91b77990864644f2a86326d4aab1aa42" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40159 } }
I20260926 23:30:37.700441 4066 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91b77990864644f2a86326d4aab1aa42 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:30:37.702557 4073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b77990864644f2a86326d4aab1aa42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91b77990864644f2a86326d4aab1aa42. Latest consensus state: current_term: 1 leader_uuid: "91b77990864644f2a86326d4aab1aa42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91b77990864644f2a86326d4aab1aa42" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40159 } } }
I20260926 23:30:37.703090 4073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b77990864644f2a86326d4aab1aa42 [sys.catalog]: This master's current role is: LEADER
I20260926 23:30:37.703995 4072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b77990864644f2a86326d4aab1aa42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91b77990864644f2a86326d4aab1aa42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91b77990864644f2a86326d4aab1aa42" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40159 } } }
I20260926 23:30:37.704329 4072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b77990864644f2a86326d4aab1aa42 [sys.catalog]: This master's current role is: LEADER
I20260926 23:30:37.705559 4076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:30:37.711860 4076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:30:37.721927 4076 catalog_manager.cc:1347] Generated new cluster ID: bdeab47c550c47099dde7ae511564b09
I20260926 23:30:37.722088 4076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:30:37.738795 4076 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:30:37.741000 4076 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:30:37.757965 4076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91b77990864644f2a86326d4aab1aa42: Generated new TSK 0
I20260926 23:30:37.759513 4076 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:30:37.824276 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40159
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:30:38.234156 4092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:38.234570 4092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:38.234724 4092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:38.243878 4092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:38.244036 4092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:38.244170 4092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:30:38.255493 4092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:38.257268 4092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:38.259063 4092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:38.275755 4097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:38.275792 4098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:38.276031 4100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:38.276875 4092 server_base.cc:1034] running on GCE node
I20260926 23:30:38.277702 4092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:38.279034 4092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:38.280326 4092 hybrid_clock.cc:648] HybridClock initialized: now 1790465438280253 us; error 56 us; skew 500 ppm
I20260926 23:30:38.280821 4092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:38 dist-test-slave-gwfx krb5kdc[3986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:38.285020 4092 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:30:38.287065 4092 webserver.cc:460] Webserver started at http://127.0.97.1:40075/ using document root <none> and password file <none>
I20260926 23:30:38.288025 4092 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:38.288172 4092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:38.288542 4092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:38.292281 4092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-0/data/instance:
uuid: "0d75af8d4bd4496ca676c823b2f262b5"
format_stamp: "Formatted at 2026-09-26 23:30:38 on dist-test-slave-gwfx"
server_key: "d407413c0a69cfe00c6805fa88f7d58f"
server_key_iv: "f8e6f75c37d005c9cca2d449afc18389"
server_key_version: "encryptionkey@0"
I20260926 23:30:38.293318 4092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0d75af8d4bd4496ca676c823b2f262b5"
format_stamp: "Formatted at 2026-09-26 23:30:38 on dist-test-slave-gwfx"
server_key: "d407413c0a69cfe00c6805fa88f7d58f"
server_key_iv: "f8e6f75c37d005c9cca2d449afc18389"
server_key_version: "encryptionkey@0"
I20260926 23:30:38.300597 4092 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260926 23:30:38.305702 4107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:38.307962 4092 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260926 23:30:38.308207 4092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "0d75af8d4bd4496ca676c823b2f262b5"
format_stamp: "Formatted at 2026-09-26 23:30:38 on dist-test-slave-gwfx"
server_key: "d407413c0a69cfe00c6805fa88f7d58f"
server_key_iv: "f8e6f75c37d005c9cca2d449afc18389"
server_key_version: "encryptionkey@0"
I20260926 23:30:38.308533 4092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:38.365130 4092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:38.370718 4092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:38.371007 4092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:38.372157 4092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:30:38.373874 4092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:30:38.373955 4092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:38.374068 4092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:30:38.374117 4092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:38.411969 4092 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41565
I20260926 23:30:38.412034 4219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41565 every 8 connection(s)
I20260926 23:30:38.413573 4092 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:30:38.420985 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 4092
I20260926 23:30:38.421270 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:30:38.421923 388 external_mini_cluster.cc:1428] Setting key fe2d6b162043e5ca26422fd0a2ddffa5
Sep 26 23:30:38 dist-test-slave-gwfx krb5kdc[3986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:38.440227 4222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:38.416888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48831 (local address 127.0.97.62:40159)
0926 23:30:38.417429 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:38.417434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:38.418859 (+ 1425us) server_negotiation.cc:407] Connection header received
0926 23:30:38.421117 (+ 2258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:38.421123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:38.421231 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:38.421341 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:38.423612 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:38.424619 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:38.426514 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:38.427317 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:38.427491 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:38.432483 (+ 4992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:38.432494 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:38.432499 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:38.432557 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:38.436069 (+ 3512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:38.437198 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:38.437205 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:38.437208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:38.437408 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:38.438006 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:38.438017 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:38.438022 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:38.438420 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:38.438573 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:38.439545 (+ 972us) server_negotiation.cc:299] Negotiation successful
0926 23:30:38.439828 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":392,"thread_start_us":257,"threads_started":1}
I20260926 23:30:38.441232 4223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:38.417584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40159 (local address 127.0.97.1:48831)
0926 23:30:38.418600 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:38.418657 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:38.420722 (+ 2065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:38.421687 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:38.421698 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:38.422414 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:38.423330 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:38.423361 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:38.424795 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:38.424800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:38.426315 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:38.426330 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:38.427650 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:38.427661 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:38.427939 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:38.429038 (+ 1099us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:38.429069 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:38.432074 (+ 3005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:38.436336 (+ 4262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:38.436345 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:38.436363 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:38.436971 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:38.437573 (+ 602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:38.437577 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:38.437580 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:38.437859 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:38.438579 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:38.438587 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:38.439082 (+ 495us) client_negotiation.cc:769] Sending connection context
0926 23:30:38.439376 (+ 294us) client_negotiation.cc:240] Negotiation successful
0926 23:30:38.439882 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":402,"threads_started":1}
I20260926 23:30:38.443192 4220 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40159
I20260926 23:30:38.443568 4220 heartbeater.cc:464] Registering TS with master...
I20260926 23:30:38.444372 4220 heartbeater.cc:511] Master 127.0.97.62:40159 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 23:30:38.447484 4031 ts_manager.cc:194] Registered new tserver with Master: 0d75af8d4bd4496ca676c823b2f262b5 (127.0.97.1:41565)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 23:30:38.449389 4031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48831
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:30:38.475064 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40159
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:30:38.851235 4226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:38.851554 4226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:38.851673 4226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:38.859246 4226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:38.859386 4226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:38.859527 4226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:30:38.869738 4226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:38.871297 4226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:38.872740 4226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:38.887487 4231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:38.887704 4232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:38.887933 4234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:38.888684 4226 server_base.cc:1034] running on GCE node
I20260926 23:30:38.889537 4226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:38.890935 4226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:38.892233 4226 hybrid_clock.cc:648] HybridClock initialized: now 1790465438892164 us; error 52 us; skew 500 ppm
I20260926 23:30:38.892743 4226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:38 dist-test-slave-gwfx krb5kdc[3986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:38.903108 4226 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:30:38.905320 4226 webserver.cc:460] Webserver started at http://127.0.97.2:42391/ using document root <none> and password file <none>
I20260926 23:30:38.906291 4226 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:38.906473 4226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:38.906924 4226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:38.910672 4226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-1/data/instance:
uuid: "8de6ba0c8a8e4698828be8a29bb1b83b"
format_stamp: "Formatted at 2026-09-26 23:30:38 on dist-test-slave-gwfx"
server_key: "68bb35c273550c425122c9f933d60a99"
server_key_iv: "c71fe32a63fac3880a7bc78086e5ea28"
server_key_version: "encryptionkey@0"
I20260926 23:30:38.911723 4226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8de6ba0c8a8e4698828be8a29bb1b83b"
format_stamp: "Formatted at 2026-09-26 23:30:38 on dist-test-slave-gwfx"
server_key: "68bb35c273550c425122c9f933d60a99"
server_key_iv: "c71fe32a63fac3880a7bc78086e5ea28"
server_key_version: "encryptionkey@0"
I20260926 23:30:38.919080 4226 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260926 23:30:38.925559 4241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:38.927983 4226 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.000s
I20260926 23:30:38.928237 4226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "8de6ba0c8a8e4698828be8a29bb1b83b"
format_stamp: "Formatted at 2026-09-26 23:30:38 on dist-test-slave-gwfx"
server_key: "68bb35c273550c425122c9f933d60a99"
server_key_iv: "c71fe32a63fac3880a7bc78086e5ea28"
server_key_version: "encryptionkey@0"
I20260926 23:30:38.928516 4226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:38.955575 4226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:38.961259 4226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:38.961617 4226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:38.962922 4226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:30:38.964548 4226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:30:38.964633 4226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:38.964768 4226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:30:38.964824 4226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:39.002724 4226 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45535
I20260926 23:30:39.002795 4353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45535 every 8 connection(s)
I20260926 23:30:39.004361 4226 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:30:39.008864 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 4226
I20260926 23:30:39.009092 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:30:39.009513 388 external_mini_cluster.cc:1428] Setting key 42911fe8597f26687b08e3d319fc20b3
Sep 26 23:30:39 dist-test-slave-gwfx krb5kdc[3986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 23:30:39.034622 4356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:39.007666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45559 (local address 127.0.97.62:40159)
0926 23:30:39.008211 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:39.008216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:39.009698 (+ 1482us) server_negotiation.cc:407] Connection header received
0926 23:30:39.012416 (+ 2718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:39.012420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:39.012501 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:39.012867 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:39.015491 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:39.016531 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:39.019445 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:39.020177 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:39.020390 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:39.026768 (+ 6378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:39.026800 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:39.026804 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:39.026857 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:39.029818 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:39.031396 (+ 1578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:39.031401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:39.031404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:39.031581 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:39.032295 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:39.032300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:39.032303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:39.032709 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:39.032883 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:39.033939 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0926 23:30:39.034288 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":299,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 23:30:39.035940 4357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:39.008405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40159 (local address 127.0.97.2:45559)
0926 23:30:39.009445 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:39.009510 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:39.011956 (+ 2446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:39.013218 (+ 1262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:39.013232 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:39.014219 (+ 987us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:39.015187 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:39.015217 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:39.016816 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:39.016822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:39.019072 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:39.019108 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:39.020646 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:39.020656 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:39.021111 (+ 455us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:39.022380 (+ 1269us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:39.022424 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:39.026200 (+ 3776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:39.030125 (+ 3925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:39.030144 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:39.030174 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:39.031130 (+ 956us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:39.031737 (+ 607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:39.031743 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:39.031748 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:39.032095 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:39.032870 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:39.032879 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:39.033390 (+ 511us) client_negotiation.cc:769] Sending connection context
0926 23:30:39.033736 (+ 346us) client_negotiation.cc:240] Negotiation successful
0926 23:30:39.034331 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":409,"threads_started":1}
I20260926 23:30:39.038251 4354 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40159
I20260926 23:30:39.038827 4354 heartbeater.cc:464] Registering TS with master...
I20260926 23:30:39.040695 4354 heartbeater.cc:511] Master 127.0.97.62:40159 requested a full tablet report, sending...
I20260926 23:30:39.043547 4031 ts_manager.cc:194] Registered new tserver with Master: 8de6ba0c8a8e4698828be8a29bb1b83b (127.0.97.2:45535)
I20260926 23:30:39.044601 4031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45559
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:30:39.067720 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40159
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:30:39.453604 4220 heartbeater.cc:503] Master 127.0.97.62:40159 was elected leader, sending a full tablet report...
W20260926 23:30:39.472011 4360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:39.472504 4360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:39.472678 4360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:39.481146 4360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:39.481307 4360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:39.481434 4360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:30:39.493006 4360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:39.494755 4360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:39.496411 4360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:39.512619 4366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:39.512734 4368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:39.512645 4365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:39.516657 4360 server_base.cc:1034] running on GCE node
I20260926 23:30:39.517562 4360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:39.518877 4360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:39.520128 4360 hybrid_clock.cc:648] HybridClock initialized: now 1790465439520064 us; error 53 us; skew 500 ppm
I20260926 23:30:39.520587 4360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:39 dist-test-slave-gwfx krb5kdc[3986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:39.524724 4360 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:30:39.526867 4360 webserver.cc:460] Webserver started at http://127.0.97.3:44847/ using document root <none> and password file <none>
I20260926 23:30:39.527810 4360 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:39.527949 4360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:39.528317 4360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:39.532099 4360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-2/data/instance:
uuid: "9c2492191a194a47b8a0f50fc0eaf290"
format_stamp: "Formatted at 2026-09-26 23:30:39 on dist-test-slave-gwfx"
server_key: "cc32407b3b069384e07c4b787a693c86"
server_key_iv: "dc710e9d912aa61a3527bd2c149d7e63"
server_key_version: "encryptionkey@0"
I20260926 23:30:39.533119 4360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9c2492191a194a47b8a0f50fc0eaf290"
format_stamp: "Formatted at 2026-09-26 23:30:39 on dist-test-slave-gwfx"
server_key: "cc32407b3b069384e07c4b787a693c86"
server_key_iv: "dc710e9d912aa61a3527bd2c149d7e63"
server_key_version: "encryptionkey@0"
I20260926 23:30:39.539841 4360 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 23:30:39.544220 4375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:39.546005 4360 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260926 23:30:39.546223 4360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "9c2492191a194a47b8a0f50fc0eaf290"
format_stamp: "Formatted at 2026-09-26 23:30:39 on dist-test-slave-gwfx"
server_key: "cc32407b3b069384e07c4b787a693c86"
server_key_iv: "dc710e9d912aa61a3527bd2c149d7e63"
server_key_version: "encryptionkey@0"
I20260926 23:30:39.546461 4360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:39.582278 4360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:39.587862 4360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:39.588178 4360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:39.589471 4360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:30:39.591331 4360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:30:39.591423 4360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:39.591565 4360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:30:39.591622 4360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:39.628883 4360 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35685
I20260926 23:30:39.628965 4487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35685 every 8 connection(s)
I20260926 23:30:39.630419 4360 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:30:39.632949 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 4360
I20260926 23:30:39.633117 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790465400694755-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:30:39.633623 388 external_mini_cluster.cc:1428] Setting key e6186a51112cb9aeca566152504316ac
Sep 26 23:30:39 dist-test-slave-gwfx krb5kdc[3986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:39.655262 4490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:39.633715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44545 (local address 127.0.97.62:40159)
0926 23:30:39.634327 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:39.634333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:39.635824 (+ 1491us) server_negotiation.cc:407] Connection header received
0926 23:30:39.638152 (+ 2328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:39.638156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:39.638257 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:39.638378 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:39.640294 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:39.641225 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:39.643222 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:39.643739 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:39.643877 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:39.648108 (+ 4231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:39.648116 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:39.648123 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:39.648222 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:39.651761 (+ 3539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:39.652656 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:39.652662 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:39.652666 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:39.652806 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:39.653249 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:39.653254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:39.653258 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:39.653570 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:39.653709 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:39.654527 (+ 818us) server_negotiation.cc:299] Negotiation successful
0926 23:30:39.654825 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":292,"threads_started":1}
I20260926 23:30:39.656447 4491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:39.634476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40159 (local address 127.0.97.3:44545)
0926 23:30:39.635604 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:39.635673 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:39.637848 (+ 2175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:39.638764 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:39.638778 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:39.639383 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:39.640066 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:39.640089 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:39.641451 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:39.641455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:39.643023 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:39.643040 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:39.644037 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:39.644043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:39.644315 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:39.645062 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:39.645091 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:39.647690 (+ 2599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:39.651938 (+ 4248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:39.651951 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:39.651971 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:39.652503 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:39.652914 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:39.652919 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:39.652924 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:39.653126 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:39.653700 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:39.653709 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:39.654096 (+ 387us) client_negotiation.cc:769] Sending connection context
0926 23:30:39.654386 (+ 290us) client_negotiation.cc:240] Negotiation successful
0926 23:30:39.654994 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":426,"threads_started":1}
I20260926 23:30:39.658186 4488 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40159
I20260926 23:30:39.658600 4488 heartbeater.cc:464] Registering TS with master...
I20260926 23:30:39.659577 4488 heartbeater.cc:511] Master 127.0.97.62:40159 requested a full tablet report, sending...
I20260926 23:30:39.661597 4031 ts_manager.cc:194] Registered new tserver with Master: 9c2492191a194a47b8a0f50fc0eaf290 (127.0.97.3:35685)
I20260926 23:30:39.662822 4031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44545
I20260926 23:30:39.664484 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 23:30:39 dist-test-slave-gwfx krb5kdc[3986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 23:30:39 dist-test-slave-gwfx krb5kdc[3986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:30:39.721240 4496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:39.701980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46650 (local address 127.0.97.1:41565)
0926 23:30:39.702825 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:39.702835 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:39.702883 (+ 48us) server_negotiation.cc:407] Connection header received
0926 23:30:39.703035 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:39.703043 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:39.703343 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:39.703520 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:39.705070 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:39.706305 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:39.707869 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:39.708649 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:39.708826 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:39.713652 (+ 4826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:39.713680 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:39.713704 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:39.713784 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:39.717665 (+ 3881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:39.718708 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:39.718722 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:39.718726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:39.718930 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:39.719461 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:39.719465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:39.719468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:39.719931 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:39.720234 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:39.720598 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 23:30:39.720907 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":344,"threads_started":1}
Sep 26 23:30:39 dist-test-slave-gwfx krb5kdc[3986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:39.744889 4490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:39.727728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40684 (local address 127.0.97.62:40159)
0926 23:30:39.728024 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:39.728032 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:39.728219 (+ 187us) server_negotiation.cc:407] Connection header received
0926 23:30:39.728414 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:39.728419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:39.728530 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:39.728634 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:39.730203 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:39.731185 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:39.732666 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:39.733205 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:39.733359 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:39.738771 (+ 5412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:39.738781 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:39.738784 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:39.738837 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:39.741742 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:39.742672 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:39.742679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:39.742715 (+ 36us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:39.742875 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:39.743254 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:39.743259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:39.743261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:39.743666 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:39.743919 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:39.744258 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 23:30:39.744527 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
W20260926 23:30:39.745807 4031 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40684
I20260926 23:30:39.747206 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 4092
I20260926 23:30:39.762136 4496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:39.748388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46656 (local address 127.0.97.1:41565)
0926 23:30:39.748593 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:39.748598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:39.748811 (+ 213us) server_negotiation.cc:407] Connection header received
0926 23:30:39.748946 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:39.748950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:39.749074 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:39.749187 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:39.750630 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:39.751506 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:39.752841 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:39.753325 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:39.753451 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:39.756167 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:39.756174 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:39.756178 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:39.756227 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:39.759264 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:39.760141 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:39.760146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:39.760149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:39.760339 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:39.760712 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:39.760716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:39.760718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:39.761017 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:39.761285 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:39.761582 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 23:30:39.761821 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 23:30:39.763090 4215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:30:39.853268 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 4092
I20260926 23:30:39.869436 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 4226
Sep 26 23:30:39 dist-test-slave-gwfx krb5kdc[3986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:30:39.887187 4501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:39.870549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39950 (local address 127.0.97.2:45535)
0926 23:30:39.871247 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:39.871254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:39.871284 (+ 30us) server_negotiation.cc:407] Connection header received
0926 23:30:39.871354 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:39.871361 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:39.871600 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:39.871755 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:39.873006 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:39.874034 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:39.875445 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:39.876027 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:39.876148 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:39.880791 (+ 4643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:39.880805 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:39.880822 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:39.880872 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:39.883893 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:39.884690 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:39.884701 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:39.884704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:39.884894 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:39.885362 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:39.885366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:39.885368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:39.885837 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:39.886136 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:39.886501 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 23:30:39.886832 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":379,"threads_started":1}
I20260926 23:30:39.888743 4349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:30:39.976840 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 4226
I20260926 23:30:39.991042 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 4360
Sep 26 23:30:39 dist-test-slave-gwfx krb5kdc[3986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:30:40.012621 4503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:39.992359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38922 (local address 127.0.97.3:35685)
0926 23:30:39.993099 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:39.993107 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:39.993137 (+ 30us) server_negotiation.cc:407] Connection header received
0926 23:30:39.993251 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:39.993258 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:39.993478 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:39.993616 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:39.995380 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:39.996448 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:39.998072 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:39.998720 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:39.998865 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:40.004033 (+ 5168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:40.004051 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:40.004071 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:40.004118 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:40.007485 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:40.008881 (+ 1396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:40.008895 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:40.008898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:40.009249 (+ 351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:40.010017 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:40.010023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:40.010026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:40.010604 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:40.010949 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:40.011760 (+ 811us) server_negotiation.cc:299] Negotiation successful
0926 23:30:40.012139 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":407,"threads_started":1}
I20260926 23:30:40.014496 4483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:30:40.105523 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 4360
I20260926 23:30:40.119149 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 3999
I20260926 23:30:40.120626 4061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:30:40.229844 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 3999
2026-09-26T23:30:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4381 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:30:40 dist-test-slave-gwfx krb5kdc[4508](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:30:40 dist-test-slave-gwfx krb5kdc[4508](info): set up 2 sockets
Sep 26 23:30:40 dist-test-slave-gwfx krb5kdc[4508](info): commencing operation
krb5kdc: starting...
W20260926 23:30:41.271450 388 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:30:41 dist-test-slave-gwfx krb5kdc[4508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:30:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:30:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:30:41.507846 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46347
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:30:41.897770 4521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:41.898183 4521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:41.898316 4521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:41.906466 4521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:30:41.906628 4521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:41.906706 4521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:41.906800 4521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:30:41.906862 4521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:30:41.917927 4521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:41.919723 4521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:41.921432 4521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:41.935209 4526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:41.935137 4527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:41.935178 4529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:41.936355 4521 server_base.cc:1034] running on GCE node
I20260926 23:30:41.937359 4521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:41.939172 4521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:41.940392 4521 hybrid_clock.cc:648] HybridClock initialized: now 1790465441940327 us; error 49 us; skew 500 ppm
I20260926 23:30:41.940851 4521 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:41 dist-test-slave-gwfx krb5kdc[4508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:41.944373 4521 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:30:41.945861 4521 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:41.945976 4521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:41.946337 4521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:41.950050 4521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "a7a81df492694bb9837f493c948e3297"
format_stamp: "Formatted at 2026-09-26 23:30:41 on dist-test-slave-gwfx"
server_key: "6bf89be960283ae8ec0c07f4d7dc82e4"
server_key_iv: "2f5a3d7c7cfb3df6b1b4198bb5304d7e"
server_key_version: "encryptionkey@0"
I20260926 23:30:41.951076 4521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "a7a81df492694bb9837f493c948e3297"
format_stamp: "Formatted at 2026-09-26 23:30:41 on dist-test-slave-gwfx"
server_key: "6bf89be960283ae8ec0c07f4d7dc82e4"
server_key_iv: "2f5a3d7c7cfb3df6b1b4198bb5304d7e"
server_key_version: "encryptionkey@0"
I20260926 23:30:41.957278 4521 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260926 23:30:41.961884 4535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:41.963914 4521 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260926 23:30:41.964159 4521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "a7a81df492694bb9837f493c948e3297"
format_stamp: "Formatted at 2026-09-26 23:30:41 on dist-test-slave-gwfx"
server_key: "6bf89be960283ae8ec0c07f4d7dc82e4"
server_key_iv: "2f5a3d7c7cfb3df6b1b4198bb5304d7e"
server_key_version: "encryptionkey@0"
I20260926 23:30:41.964435 4521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:41.983232 4521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:41.989109 4521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:41.989535 4521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:42.011704 4521 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46347
I20260926 23:30:42.011730 4586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46347 every 8 connection(s)
I20260926 23:30:42.012854 4521 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:30:42.017668 4587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:42.023273 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 4521
I20260926 23:30:42.023555 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:30:42.024274 388 external_mini_cluster.cc:1428] Setting key 41d2b1c34a0210c2c6262ddefdf6a8ce
I20260926 23:30:42.033155 4587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7a81df492694bb9837f493c948e3297: Bootstrap starting.
I20260926 23:30:42.037936 4587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7a81df492694bb9837f493c948e3297: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:42.039628 4587 log.cc:824] T 00000000000000000000000000000000 P a7a81df492694bb9837f493c948e3297: Log is configured to *not* fsync() on all Append() calls
I20260926 23:30:42.043825 4587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7a81df492694bb9837f493c948e3297: No bootstrap required, opened a new log
Sep 26 23:30:42 dist-test-slave-gwfx krb5kdc[4508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:42.053929 4587 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7a81df492694bb9837f493c948e3297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a81df492694bb9837f493c948e3297" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46347 } }
I20260926 23:30:42.054363 4587 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7a81df492694bb9837f493c948e3297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:42.054476 4587 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7a81df492694bb9837f493c948e3297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7a81df492694bb9837f493c948e3297, State: Initialized, Role: FOLLOWER
I20260926 23:30:42.054679 4590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:42.027205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51098 (local address 127.0.97.62:46347)
0926 23:30:42.028471 (+ 1266us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:42.028491 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:42.028564 (+ 73us) server_negotiation.cc:407] Connection header received
0926 23:30:42.031454 (+ 2890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:42.031500 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:42.032267 (+ 767us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:42.032915 (+ 648us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:42.034724 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:42.036990 (+ 2266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:42.038743 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:42.039977 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:42.040224 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:42.045341 (+ 5117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:42.045351 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:42.045373 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:42.045419 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:42.049114 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:42.050309 (+ 1195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:42.050323 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:42.050332 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:42.050592 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:42.051105 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:42.051109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:42.051112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:42.051939 (+ 827us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:42.052211 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:42.052766 (+ 555us) server_negotiation.cc:299] Negotiation successful
0926 23:30:42.053312 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":923,"thread_start_us":519,"threads_started":1}
I20260926 23:30:42.055249 4587 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7a81df492694bb9837f493c948e3297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a81df492694bb9837f493c948e3297" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46347 } }
I20260926 23:30:42.055505 4587 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7a81df492694bb9837f493c948e3297 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:30:42.055653 4587 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7a81df492694bb9837f493c948e3297 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:30:42.055838 4587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7a81df492694bb9837f493c948e3297 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:42.059285 4587 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7a81df492694bb9837f493c948e3297 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a81df492694bb9837f493c948e3297" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46347 } }
I20260926 23:30:42.059942 4587 leader_election.cc:302] T 00000000000000000000000000000000 P a7a81df492694bb9837f493c948e3297 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7a81df492694bb9837f493c948e3297; no voters:
I20260926 23:30:42.060614 4587 leader_election.cc:288] T 00000000000000000000000000000000 P a7a81df492694bb9837f493c948e3297 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:30:42.060765 4592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7a81df492694bb9837f493c948e3297 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:30:42.061478 4592 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7a81df492694bb9837f493c948e3297 [term 1 LEADER]: Becoming Leader. State: Replica: a7a81df492694bb9837f493c948e3297, State: Running, Role: LEADER
I20260926 23:30:42.062142 4592 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7a81df492694bb9837f493c948e3297 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a81df492694bb9837f493c948e3297" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46347 } }
I20260926 23:30:42.063139 4587 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7a81df492694bb9837f493c948e3297 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:30:42.065593 4593 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7a81df492694bb9837f493c948e3297 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7a81df492694bb9837f493c948e3297" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a81df492694bb9837f493c948e3297" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46347 } } }
I20260926 23:30:42.065968 4593 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7a81df492694bb9837f493c948e3297 [sys.catalog]: This master's current role is: LEADER
I20260926 23:30:42.066431 4594 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7a81df492694bb9837f493c948e3297 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7a81df492694bb9837f493c948e3297. Latest consensus state: current_term: 1 leader_uuid: "a7a81df492694bb9837f493c948e3297" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a81df492694bb9837f493c948e3297" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46347 } } }
I20260926 23:30:42.066807 4594 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7a81df492694bb9837f493c948e3297 [sys.catalog]: This master's current role is: LEADER
I20260926 23:30:42.067122 4599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:30:42.074082 4599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:30:42.087255 4599 catalog_manager.cc:1347] Generated new cluster ID: 90ad6982edb842e4a01d85fba00722f8
I20260926 23:30:42.087424 4599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:30:42.102941 4599 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:30:42.104593 4599 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:30:42.116310 4599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7a81df492694bb9837f493c948e3297: Generated new TSK 0
I20260926 23:30:42.117683 4599 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:30:42.191607 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46347
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:30:42.617740 4613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:42.618145 4613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:42.618291 4613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:42.626636 4613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:42.626868 4613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:42.627045 4613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:30:42.638345 4613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:42.640146 4613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:42.641872 4613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:42.659269 4618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:42.659304 4621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:42.659240 4619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:42.660264 4613 server_base.cc:1034] running on GCE node
I20260926 23:30:42.661074 4613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:42.662360 4613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:42.663609 4613 hybrid_clock.cc:648] HybridClock initialized: now 1790465442663519 us; error 78 us; skew 500 ppm
I20260926 23:30:42.664043 4613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:42 dist-test-slave-gwfx krb5kdc[4508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:42.667924 4613 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:30:42.669530 4613 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:42.669656 4613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:42.670089 4613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:42.673677 4613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-0/data/instance:
uuid: "2cf2f3fa61b34a13913081544013fd7a"
format_stamp: "Formatted at 2026-09-26 23:30:42 on dist-test-slave-gwfx"
server_key: "d6facd55c5eaf89a21544b304145903f"
server_key_iv: "f97b2f327a1bcd304523507e89167093"
server_key_version: "encryptionkey@0"
I20260926 23:30:42.674646 4613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2cf2f3fa61b34a13913081544013fd7a"
format_stamp: "Formatted at 2026-09-26 23:30:42 on dist-test-slave-gwfx"
server_key: "d6facd55c5eaf89a21544b304145903f"
server_key_iv: "f97b2f327a1bcd304523507e89167093"
server_key_version: "encryptionkey@0"
I20260926 23:30:42.680917 4613 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 23:30:42.685300 4627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:42.687156 4613 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:30:42.687363 4613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "2cf2f3fa61b34a13913081544013fd7a"
format_stamp: "Formatted at 2026-09-26 23:30:42 on dist-test-slave-gwfx"
server_key: "d6facd55c5eaf89a21544b304145903f"
server_key_iv: "f97b2f327a1bcd304523507e89167093"
server_key_version: "encryptionkey@0"
I20260926 23:30:42.687649 4613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:42.732151 4613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:42.737634 4613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:42.737952 4613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:42.739147 4613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:30:42.740742 4613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:30:42.740852 4613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:42.740974 4613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:30:42.741047 4613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:42.775620 4613 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45111
I20260926 23:30:42.775689 4739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45111 every 8 connection(s)
I20260926 23:30:42.776718 4613 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:30:42.780875 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 4613
I20260926 23:30:42.781153 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:30:42.781991 388 external_mini_cluster.cc:1428] Setting key fcd0e77fefc0d2b00b7e611a6b6fba15
Sep 26 23:30:42 dist-test-slave-gwfx krb5kdc[4508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:42.802820 4742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:42.779883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36669 (local address 127.0.97.62:46347)
0926 23:30:42.780439 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:42.780516 (+ 77us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:42.781793 (+ 1277us) server_negotiation.cc:407] Connection header received
0926 23:30:42.784368 (+ 2575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:42.784374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:42.784473 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:42.784590 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:42.787425 (+ 2835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:42.788228 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:42.790737 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:42.791180 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:42.791287 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:42.795841 (+ 4554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:42.795847 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:42.795850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:42.795894 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:42.798656 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:42.799781 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:42.799786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:42.799789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:42.799936 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:42.800539 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:42.800544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:42.800546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:42.800898 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:42.801065 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:42.802223 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0926 23:30:42.802464 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":250,"threads_started":1}
I20260926 23:30:42.804605 4743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:42.780560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46347 (local address 127.0.97.1:36669)
0926 23:30:42.781563 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:42.781624 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:42.783997 (+ 2373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:42.784932 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:42.784947 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:42.785901 (+ 954us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:42.787155 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:42.787194 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:42.788450 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:42.788455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:42.790516 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:42.790534 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:42.791436 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:42.791453 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:42.791800 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:42.792686 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:42.792714 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:42.795485 (+ 2771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:42.798854 (+ 3369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:42.798866 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:42.798892 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:42.799606 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:42.800053 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:42.800058 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:42.800061 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:42.800363 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:42.801088 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:42.801099 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:42.801716 (+ 617us) client_negotiation.cc:769] Sending connection context
0926 23:30:42.802091 (+ 375us) client_negotiation.cc:240] Negotiation successful
0926 23:30:42.802835 (+ 744us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":389,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 23:30:42.806653 4740 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46347
I20260926 23:30:42.807054 4740 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 23:30:42.807899 4740 heartbeater.cc:511] Master 127.0.97.62:46347 requested a full tablet report, sending...
I20260926 23:30:42.810873 4552 ts_manager.cc:194] Registered new tserver with Master: 2cf2f3fa61b34a13913081544013fd7a (127.0.97.1:45111)
I20260926 23:30:42.813081 4552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:30:42.839380 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46347
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:30:43.238260 4746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:43.238616 4746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:43.238786 4746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:43.246660 4746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:43.246831 4746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:43.247027 4746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:30:43.258293 4746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:43.260507 4746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:43.262256 4746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:43.277100 4751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:43.277069 4752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:43.277128 4754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:43.278353 4746 server_base.cc:1034] running on GCE node
I20260926 23:30:43.279181 4746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:43.280388 4746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:43.281625 4746 hybrid_clock.cc:648] HybridClock initialized: now 1790465443281561 us; error 61 us; skew 500 ppm
I20260926 23:30:43.282027 4746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:43 dist-test-slave-gwfx krb5kdc[4508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:43.286072 4746 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:30:43.287685 4746 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:43.287818 4746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:43.288193 4746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:43.291919 4746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-1/data/instance:
uuid: "230d1b5af2844860b41377ea32cdba6b"
format_stamp: "Formatted at 2026-09-26 23:30:43 on dist-test-slave-gwfx"
server_key: "41db2862ca4e1380921296701f514942"
server_key_iv: "f5ad1d41d1effc42688dad7c2242b367"
server_key_version: "encryptionkey@0"
I20260926 23:30:43.292928 4746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-1/wal/instance:
uuid: "230d1b5af2844860b41377ea32cdba6b"
format_stamp: "Formatted at 2026-09-26 23:30:43 on dist-test-slave-gwfx"
server_key: "41db2862ca4e1380921296701f514942"
server_key_iv: "f5ad1d41d1effc42688dad7c2242b367"
server_key_version: "encryptionkey@0"
I20260926 23:30:43.299572 4746 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 23:30:43.304489 4760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:43.306450 4746 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 23:30:43.306732 4746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "230d1b5af2844860b41377ea32cdba6b"
format_stamp: "Formatted at 2026-09-26 23:30:43 on dist-test-slave-gwfx"
server_key: "41db2862ca4e1380921296701f514942"
server_key_iv: "f5ad1d41d1effc42688dad7c2242b367"
server_key_version: "encryptionkey@0"
I20260926 23:30:43.307056 4746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:43.323724 4746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:43.329494 4746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:43.329809 4746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:43.331096 4746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:30:43.332903 4746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:30:43.333011 4746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:43.333105 4746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:30:43.333164 4746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:43.368630 4746 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35851
I20260926 23:30:43.368703 4872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35851 every 8 connection(s)
I20260926 23:30:43.369778 4746 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:30:43.375483 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 4746
I20260926 23:30:43.375723 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:30:43.376417 388 external_mini_cluster.cc:1428] Setting key 6bf10248e06439aab838bc5a357b6368
Sep 26 23:30:43 dist-test-slave-gwfx krb5kdc[4508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:43.397794 4875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:43.372706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44631 (local address 127.0.97.62:46347)
0926 23:30:43.373241 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:43.373246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:43.374529 (+ 1283us) server_negotiation.cc:407] Connection header received
0926 23:30:43.377697 (+ 3168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:43.377705 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:43.377830 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:43.377952 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:43.380430 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:43.381244 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:43.384025 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:43.384586 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:43.384730 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:43.389931 (+ 5201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:43.389940 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:43.389946 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:43.390004 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:43.393550 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:43.394635 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:43.394642 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:43.394646 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:43.394895 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:43.395417 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:43.395424 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:43.395427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:43.395877 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:43.396059 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:43.397074 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0926 23:30:43.397353 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":290,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 23:30:43.399091 4876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:43.373310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46347 (local address 127.0.97.2:44631)
0926 23:30:43.374254 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:43.374307 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:43.377147 (+ 2840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:43.378282 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:43.378292 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:43.379015 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:43.380129 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:43.380162 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:43.381460 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:43.381465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:43.383751 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:43.383775 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:43.384947 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:43.384957 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:43.385356 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:43.386403 (+ 1047us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:43.386436 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:43.389447 (+ 3011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:43.393819 (+ 4372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:43.393831 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:43.393851 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:43.394423 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:43.395034 (+ 611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:43.395038 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:43.395040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:43.395280 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:43.396040 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:43.396048 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:43.396586 (+ 538us) client_negotiation.cc:769] Sending connection context
0926 23:30:43.396883 (+ 297us) client_negotiation.cc:240] Negotiation successful
0926 23:30:43.397435 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":414,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 23:30:43.401548 4873 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46347
I20260926 23:30:43.401983 4873 heartbeater.cc:464] Registering TS with master...
I20260926 23:30:43.402931 4873 heartbeater.cc:511] Master 127.0.97.62:46347 requested a full tablet report, sending...
I20260926 23:30:43.405082 4552 ts_manager.cc:194] Registered new tserver with Master: 230d1b5af2844860b41377ea32cdba6b (127.0.97.2:35851)
I20260926 23:30:43.406037 4552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:30:43.427039 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46347
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:30:43.803686 4879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:43.803989 4879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:43.804090 4879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:43.811653 4879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:43.811797 4879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:43.811918 4879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:30:43.817932 4740 heartbeater.cc:503] Master 127.0.97.62:46347 was elected leader, sending a full tablet report...
I20260926 23:30:43.823287 4879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:43.824745 4879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:43.826170 4879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:43.842413 4884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:43.842499 4887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:43.842486 4885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:43.844153 4879 server_base.cc:1034] running on GCE node
I20260926 23:30:43.844941 4879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:43.846223 4879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:43.847605 4879 hybrid_clock.cc:648] HybridClock initialized: now 1790465443847496 us; error 97 us; skew 500 ppm
I20260926 23:30:43.848117 4879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:43 dist-test-slave-gwfx krb5kdc[4508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:43.851948 4879 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:30:43.853813 4879 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:43.853982 4879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:43.854537 4879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:43.857888 4879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-2/data/instance:
uuid: "8dbf35e45bab4691b17277d00344171a"
format_stamp: "Formatted at 2026-09-26 23:30:43 on dist-test-slave-gwfx"
server_key: "1ecd9edf9e2e64104c325784c84a40c7"
server_key_iv: "2f346bd624d2daa4b0be5ef331ebe6d6"
server_key_version: "encryptionkey@0"
I20260926 23:30:43.858886 4879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8dbf35e45bab4691b17277d00344171a"
format_stamp: "Formatted at 2026-09-26 23:30:43 on dist-test-slave-gwfx"
server_key: "1ecd9edf9e2e64104c325784c84a40c7"
server_key_iv: "2f346bd624d2daa4b0be5ef331ebe6d6"
server_key_version: "encryptionkey@0"
I20260926 23:30:43.865046 4879 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260926 23:30:43.869699 4893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:43.871328 4879 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:30:43.871498 4879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "8dbf35e45bab4691b17277d00344171a"
format_stamp: "Formatted at 2026-09-26 23:30:43 on dist-test-slave-gwfx"
server_key: "1ecd9edf9e2e64104c325784c84a40c7"
server_key_iv: "2f346bd624d2daa4b0be5ef331ebe6d6"
server_key_version: "encryptionkey@0"
I20260926 23:30:43.871766 4879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:43.912973 4879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:43.918921 4879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:43.919188 4879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:43.920310 4879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:30:43.922197 4879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:30:43.922328 4879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:43.922444 4879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:30:43.922516 4879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:43.959285 4879 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39341
I20260926 23:30:43.959391 5005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39341 every 8 connection(s)
I20260926 23:30:43.960264 4879 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:30:43.970788 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 4879
I20260926 23:30:43.971035 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:30:43.971621 388 external_mini_cluster.cc:1428] Setting key 34e7b4f5b4044e3a66187daee2606aed
Sep 26 23:30:43 dist-test-slave-gwfx krb5kdc[4508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:43.986449 5008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:43.963608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54753 (local address 127.0.97.62:46347)
0926 23:30:43.964141 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:43.964146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:43.965414 (+ 1268us) server_negotiation.cc:407] Connection header received
0926 23:30:43.967721 (+ 2307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:43.967728 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:43.967830 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:43.967940 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:43.970173 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:43.971480 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:43.973919 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:43.974592 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:43.974772 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:43.979476 (+ 4704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:43.979483 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:43.979486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:43.979539 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:43.982604 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:43.983694 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:43.983701 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:43.983704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:43.983847 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:43.984332 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:43.984336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:43.984338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:43.984669 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:43.984826 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:43.985704 (+ 878us) server_negotiation.cc:299] Negotiation successful
0926 23:30:43.986001 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":263,"threads_started":1}
I20260926 23:30:43.987850 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:43.964114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46347 (local address 127.0.97.3:54753)
0926 23:30:43.965119 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:43.965248 (+ 129us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:43.967331 (+ 2083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:43.968276 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:43.968288 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:43.969029 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:43.969934 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:43.969957 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:43.971789 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:43.971796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:43.973692 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:43.973717 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:43.974928 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:43.974937 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:43.975274 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:43.976165 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:43.976197 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:43.979141 (+ 2944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:43.982855 (+ 3714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:43.982866 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:43.982888 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:43.983540 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:43.983965 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:43.983971 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:43.983975 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:43.984199 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:43.984806 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:43.984815 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:43.985292 (+ 477us) client_negotiation.cc:769] Sending connection context
0926 23:30:43.985570 (+ 278us) client_negotiation.cc:240] Negotiation successful
0926 23:30:43.986239 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":379,"threads_started":1}
I20260926 23:30:43.989423 5006 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46347
I20260926 23:30:43.989802 5006 heartbeater.cc:464] Registering TS with master...
I20260926 23:30:43.990928 5006 heartbeater.cc:511] Master 127.0.97.62:46347 requested a full tablet report, sending...
I20260926 23:30:43.993070 4552 ts_manager.cc:194] Registered new tserver with Master: 8dbf35e45bab4691b17277d00344171a (127.0.97.3:39341)
I20260926 23:30:43.994012 4552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54753
I20260926 23:30:44.002542 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:30:44.025271 5008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:44.008800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51108 (local address 127.0.97.62:46347)
0926 23:30:44.009089 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:44.009097 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:44.009389 (+ 292us) server_negotiation.cc:407] Connection header received
0926 23:30:44.009596 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:44.009601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:44.009691 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:44.009808 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:44.011877 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:44.012976 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:44.014656 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:44.015229 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:44.015432 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:44.018214 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:44.018219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:44.018223 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:44.018270 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:44.021685 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:44.022925 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:44.022931 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:44.022933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:44.023122 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:44.023538 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:44.023542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:44.023544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:44.023996 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:44.024262 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:44.024621 (+ 359us) server_negotiation.cc:299] Negotiation successful
0926 23:30:44.024893 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260926 23:30:44.030431 4552 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:30:44.037793 4552 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:30:44.066740 5022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:44.054648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50480 (local address 127.0.97.3:39341)
0926 23:30:44.055525 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:44.055537 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:44.056762 (+ 1225us) server_negotiation.cc:407] Connection header received
0926 23:30:44.057089 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:44.057099 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:44.057406 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:44.057610 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:30:44.058901 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:44.060575 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:44.063806 (+ 3231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:44.065275 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:44.065433 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:44.065734 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:44.066077 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 23:30:44.066297 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":406,"threads_started":1}
I20260926 23:30:44.066867 5019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:44.054300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39341 (local address 127.0.0.1:50480)
0926 23:30:44.056435 (+ 2135us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:44.056518 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:44.056780 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:44.057925 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:44.057935 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:44.057971 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:30:44.058630 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:44.058646 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:44.060734 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:44.060739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:44.063597 (+ 2858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:44.063612 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:44.065514 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:44.065521 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:44.065761 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:44.065817 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 23:30:44.066040 (+ 223us) client_negotiation.cc:240] Negotiation successful
0926 23:30:44.066303 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1806,"mutex_wait_us":171,"spinlock_wait_cycles":4608,"thread_start_us":1602,"threads_started":1}
I20260926 23:30:44.067516 5021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:44.054760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58282 (local address 127.0.97.1:45111)
0926 23:30:44.056091 (+ 1331us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:44.056099 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:44.056678 (+ 579us) server_negotiation.cc:407] Connection header received
0926 23:30:44.057091 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:44.057099 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:44.057407 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:44.057610 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:30:44.059643 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:44.061249 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:44.063764 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:44.065522 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:44.065688 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:44.066034 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:44.067004 (+ 970us) server_negotiation.cc:299] Negotiation successful
0926 23:30:44.067193 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1143,"thread_start_us":921,"threads_started":1}
I20260926 23:30:44.067968 5020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:44.054300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45111 (local address 127.0.0.1:58282)
0926 23:30:44.056435 (+ 2135us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:44.056518 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:44.056780 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:44.057925 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:44.057935 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:44.057977 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:30:44.059381 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:44.059401 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:44.061384 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:44.061389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:44.063507 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:44.063529 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:44.066198 (+ 2669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:44.066204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:44.066429 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:44.066468 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 23:30:44.066566 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 23:30:44.066725 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1977,"mutex_wait_us":736,"spinlock_wait_cycles":12032,"thread_start_us":2409,"threads_started":1}
I20260926 23:30:44.068917 5023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:44.057365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35851 (local address 127.0.0.1:58272)
0926 23:30:44.059395 (+ 2030us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:44.059417 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:44.059551 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:44.060381 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:44.060387 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:44.060421 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:30:44.060833 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:44.060847 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:44.062736 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:44.062749 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:44.065099 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:44.065116 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:44.067347 (+ 2231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:44.067354 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:44.068116 (+ 762us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:44.068155 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 23:30:44.068247 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 23:30:44.068408 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1878,"mutex_wait_us":102,"thread_start_us":1730,"threads_started":1}
I20260926 23:30:44.069396 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:44.057680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58272 (local address 127.0.97.2:35851)
0926 23:30:44.058892 (+ 1212us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:44.058902 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:44.059671 (+ 769us) server_negotiation.cc:407] Connection header received
0926 23:30:44.059779 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:44.059788 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:44.060100 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:44.060292 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:30:44.061028 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:44.062540 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:44.065310 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:44.066794 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:44.066952 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:44.067168 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:44.068605 (+ 1437us) server_negotiation.cc:299] Negotiation successful
0926 23:30:44.068858 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":951,"thread_start_us":753,"threads_started":1}
I20260926 23:30:44.073201 4675 tablet_service.cc:1467] Processing CreateTablet for tablet 75f8eb38850343239231fc7447e0de0a (DEFAULT_TABLE table=test-table [id=65a2f4bbbb174d8da5acf4d9210ea1d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:30:44.073182 4941 tablet_service.cc:1467] Processing CreateTablet for tablet 75f8eb38850343239231fc7447e0de0a (DEFAULT_TABLE table=test-table [id=65a2f4bbbb174d8da5acf4d9210ea1d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:30:44.074111 4808 tablet_service.cc:1467] Processing CreateTablet for tablet 75f8eb38850343239231fc7447e0de0a (DEFAULT_TABLE table=test-table [id=65a2f4bbbb174d8da5acf4d9210ea1d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:30:44.075157 4941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75f8eb38850343239231fc7447e0de0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:44.075147 4675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75f8eb38850343239231fc7447e0de0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:44.075861 4808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75f8eb38850343239231fc7447e0de0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:44.089486 5025 tablet_bootstrap.cc:492] T 75f8eb38850343239231fc7447e0de0a P 230d1b5af2844860b41377ea32cdba6b: Bootstrap starting.
I20260926 23:30:44.093405 5025 tablet_bootstrap.cc:654] T 75f8eb38850343239231fc7447e0de0a P 230d1b5af2844860b41377ea32cdba6b: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:44.094162 5027 tablet_bootstrap.cc:492] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a: Bootstrap starting.
I20260926 23:30:44.095237 5025 log.cc:824] T 75f8eb38850343239231fc7447e0de0a P 230d1b5af2844860b41377ea32cdba6b: Log is configured to *not* fsync() on all Append() calls
I20260926 23:30:44.097771 5026 tablet_bootstrap.cc:492] T 75f8eb38850343239231fc7447e0de0a P 2cf2f3fa61b34a13913081544013fd7a: Bootstrap starting.
I20260926 23:30:44.098270 5027 tablet_bootstrap.cc:654] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:44.099745 5027 log.cc:824] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a: Log is configured to *not* fsync() on all Append() calls
I20260926 23:30:44.101187 5025 tablet_bootstrap.cc:492] T 75f8eb38850343239231fc7447e0de0a P 230d1b5af2844860b41377ea32cdba6b: No bootstrap required, opened a new log
I20260926 23:30:44.101577 5025 ts_tablet_manager.cc:1397] T 75f8eb38850343239231fc7447e0de0a P 230d1b5af2844860b41377ea32cdba6b: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260926 23:30:44.101855 5026 tablet_bootstrap.cc:654] T 75f8eb38850343239231fc7447e0de0a P 2cf2f3fa61b34a13913081544013fd7a: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:44.103317 5026 log.cc:824] T 75f8eb38850343239231fc7447e0de0a P 2cf2f3fa61b34a13913081544013fd7a: Log is configured to *not* fsync() on all Append() calls
I20260926 23:30:44.103786 5027 tablet_bootstrap.cc:492] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a: No bootstrap required, opened a new log
I20260926 23:30:44.104159 5027 ts_tablet_manager.cc:1397] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260926 23:30:44.111296 5026 tablet_bootstrap.cc:492] T 75f8eb38850343239231fc7447e0de0a P 2cf2f3fa61b34a13913081544013fd7a: No bootstrap required, opened a new log
I20260926 23:30:44.111755 5026 ts_tablet_manager.cc:1397] T 75f8eb38850343239231fc7447e0de0a P 2cf2f3fa61b34a13913081544013fd7a: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20260926 23:30:44.111438 5025 raft_consensus.cc:348] T 75f8eb38850343239231fc7447e0de0a P 230d1b5af2844860b41377ea32cdba6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbf35e45bab4691b17277d00344171a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39341 } } peers { permanent_uuid: "2cf2f3fa61b34a13913081544013fd7a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45111 } } peers { permanent_uuid: "230d1b5af2844860b41377ea32cdba6b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35851 } }
I20260926 23:30:44.112005 5025 raft_consensus.cc:374] T 75f8eb38850343239231fc7447e0de0a P 230d1b5af2844860b41377ea32cdba6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:44.112115 5025 raft_consensus.cc:729] T 75f8eb38850343239231fc7447e0de0a P 230d1b5af2844860b41377ea32cdba6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 230d1b5af2844860b41377ea32cdba6b, State: Initialized, Role: FOLLOWER
I20260926 23:30:44.112973 5025 consensus_queue.cc:260] T 75f8eb38850343239231fc7447e0de0a P 230d1b5af2844860b41377ea32cdba6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbf35e45bab4691b17277d00344171a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39341 } } peers { permanent_uuid: "2cf2f3fa61b34a13913081544013fd7a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45111 } } peers { permanent_uuid: "230d1b5af2844860b41377ea32cdba6b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35851 } }
I20260926 23:30:44.114369 4873 heartbeater.cc:503] Master 127.0.97.62:46347 was elected leader, sending a full tablet report...
I20260926 23:30:44.115280 5025 ts_tablet_manager.cc:1428] T 75f8eb38850343239231fc7447e0de0a P 230d1b5af2844860b41377ea32cdba6b: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260926 23:30:44.115403 5027 raft_consensus.cc:348] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbf35e45bab4691b17277d00344171a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39341 } } peers { permanent_uuid: "2cf2f3fa61b34a13913081544013fd7a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45111 } } peers { permanent_uuid: "230d1b5af2844860b41377ea32cdba6b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35851 } }
I20260926 23:30:44.115960 5027 raft_consensus.cc:374] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:44.116068 5027 raft_consensus.cc:729] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dbf35e45bab4691b17277d00344171a, State: Initialized, Role: FOLLOWER
I20260926 23:30:44.116868 5027 consensus_queue.cc:260] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbf35e45bab4691b17277d00344171a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39341 } } peers { permanent_uuid: "2cf2f3fa61b34a13913081544013fd7a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45111 } } peers { permanent_uuid: "230d1b5af2844860b41377ea32cdba6b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35851 } }
I20260926 23:30:44.118263 5006 heartbeater.cc:503] Master 127.0.97.62:46347 was elected leader, sending a full tablet report...
I20260926 23:30:44.119094 5027 ts_tablet_manager.cc:1428] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260926 23:30:44.122128 4874 tablet.cc:2369] T 75f8eb38850343239231fc7447e0de0a P 230d1b5af2844860b41377ea32cdba6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:30:44.124053 5026 raft_consensus.cc:348] T 75f8eb38850343239231fc7447e0de0a P 2cf2f3fa61b34a13913081544013fd7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbf35e45bab4691b17277d00344171a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39341 } } peers { permanent_uuid: "2cf2f3fa61b34a13913081544013fd7a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45111 } } peers { permanent_uuid: "230d1b5af2844860b41377ea32cdba6b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35851 } }
I20260926 23:30:44.124688 5026 raft_consensus.cc:374] T 75f8eb38850343239231fc7447e0de0a P 2cf2f3fa61b34a13913081544013fd7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:44.124827 5026 raft_consensus.cc:729] T 75f8eb38850343239231fc7447e0de0a P 2cf2f3fa61b34a13913081544013fd7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cf2f3fa61b34a13913081544013fd7a, State: Initialized, Role: FOLLOWER
I20260926 23:30:44.125734 5026 consensus_queue.cc:260] T 75f8eb38850343239231fc7447e0de0a P 2cf2f3fa61b34a13913081544013fd7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbf35e45bab4691b17277d00344171a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39341 } } peers { permanent_uuid: "2cf2f3fa61b34a13913081544013fd7a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45111 } } peers { permanent_uuid: "230d1b5af2844860b41377ea32cdba6b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35851 } }
I20260926 23:30:44.128134 5026 ts_tablet_manager.cc:1428] T 75f8eb38850343239231fc7447e0de0a P 2cf2f3fa61b34a13913081544013fd7a: Time spent starting tablet: real 0.016s user 0.009s sys 0.006s
W20260926 23:30:44.212014 5007 tablet.cc:2369] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:30:44.215602 5032 raft_consensus.cc:482] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:30:44.215946 5032 raft_consensus.cc:504] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbf35e45bab4691b17277d00344171a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39341 } } peers { permanent_uuid: "2cf2f3fa61b34a13913081544013fd7a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45111 } } peers { permanent_uuid: "230d1b5af2844860b41377ea32cdba6b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35851 } }
I20260926 23:30:44.217955 5032 leader_election.cc:288] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cf2f3fa61b34a13913081544013fd7a (127.0.97.1:45111), 230d1b5af2844860b41377ea32cdba6b (127.0.97.2:35851)
I20260926 23:30:44.226145 5021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:44.218551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37557 (local address 127.0.97.1:45111)
0926 23:30:44.218872 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:44.218878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:44.218902 (+ 24us) server_negotiation.cc:407] Connection header received
0926 23:30:44.219097 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:44.219103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:44.219182 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:44.219323 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:30:44.219969 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:44.221033 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:44.223545 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:44.224907 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:44.225185 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:44.225319 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:44.225668 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 23:30:44.225859 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260926 23:30:44.226145 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:44.218301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45111 (local address 127.0.97.3:37557)
0926 23:30:44.218625 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:44.218656 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:44.218909 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:44.219505 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:44.219510 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:44.219557 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:30:44.219773 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:44.219788 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:44.221226 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:44.221233 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:44.223286 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:44.223312 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:44.225209 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:44.225216 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:44.225426 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:44.225456 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 23:30:44.225623 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 23:30:44.225807 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":1}
I20260926 23:30:44.227537 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:44.219225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39653 (local address 127.0.97.2:35851)
0926 23:30:44.219440 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:44.219446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:44.219665 (+ 219us) server_negotiation.cc:407] Connection header received
0926 23:30:44.219843 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:44.219848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:44.219928 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:44.220009 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:30:44.220773 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:44.222211 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:44.225110 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:44.226462 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:44.226626 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:44.226795 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:44.227109 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 23:30:44.227251 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 23:30:44.227614 5034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:44.218838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35851 (local address 127.0.97.3:39653)
0926 23:30:44.219548 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:44.219565 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:44.219710 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:44.220069 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:44.220075 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:44.220101 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:30:44.220515 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:44.220528 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:44.222452 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:44.222459 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:44.224797 (+ 2338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:44.224827 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:44.226726 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:44.226734 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:44.226915 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:44.226950 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 23:30:44.227063 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 23:30:44.227222 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":4,"thread_start_us":378,"threads_started":1}
I20260926 23:30:44.227741 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f8eb38850343239231fc7447e0de0a" candidate_uuid: "8dbf35e45bab4691b17277d00344171a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cf2f3fa61b34a13913081544013fd7a" is_pre_election: true
I20260926 23:30:44.228436 4695 raft_consensus.cc:2454] T 75f8eb38850343239231fc7447e0de0a P 2cf2f3fa61b34a13913081544013fd7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dbf35e45bab4691b17277d00344171a in term 0.
I20260926 23:30:44.228693 4828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f8eb38850343239231fc7447e0de0a" candidate_uuid: "8dbf35e45bab4691b17277d00344171a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "230d1b5af2844860b41377ea32cdba6b" is_pre_election: true
I20260926 23:30:44.229245 4895 leader_election.cc:302] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cf2f3fa61b34a13913081544013fd7a, 8dbf35e45bab4691b17277d00344171a; no voters:
I20260926 23:30:44.229382 4828 raft_consensus.cc:2454] T 75f8eb38850343239231fc7447e0de0a P 230d1b5af2844860b41377ea32cdba6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dbf35e45bab4691b17277d00344171a in term 0.
I20260926 23:30:44.229714 5032 raft_consensus.cc:2781] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:30:44.229854 5032 raft_consensus.cc:482] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:30:44.229961 5032 raft_consensus.cc:3037] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:44.232569 5032 raft_consensus.cc:504] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbf35e45bab4691b17277d00344171a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39341 } } peers { permanent_uuid: "2cf2f3fa61b34a13913081544013fd7a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45111 } } peers { permanent_uuid: "230d1b5af2844860b41377ea32cdba6b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35851 } }
I20260926 23:30:44.233278 5032 leader_election.cc:288] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a [CANDIDATE]: Term 1 election: Requested vote from peers 2cf2f3fa61b34a13913081544013fd7a (127.0.97.1:45111), 230d1b5af2844860b41377ea32cdba6b (127.0.97.2:35851)
I20260926 23:30:44.233971 4828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f8eb38850343239231fc7447e0de0a" candidate_uuid: "8dbf35e45bab4691b17277d00344171a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "230d1b5af2844860b41377ea32cdba6b"
I20260926 23:30:44.233985 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f8eb38850343239231fc7447e0de0a" candidate_uuid: "8dbf35e45bab4691b17277d00344171a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cf2f3fa61b34a13913081544013fd7a"
I20260926 23:30:44.234202 4828 raft_consensus.cc:3037] T 75f8eb38850343239231fc7447e0de0a P 230d1b5af2844860b41377ea32cdba6b [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:44.234251 4695 raft_consensus.cc:3037] T 75f8eb38850343239231fc7447e0de0a P 2cf2f3fa61b34a13913081544013fd7a [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:44.236733 4828 raft_consensus.cc:2454] T 75f8eb38850343239231fc7447e0de0a P 230d1b5af2844860b41377ea32cdba6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dbf35e45bab4691b17277d00344171a in term 1.
I20260926 23:30:44.236873 4695 raft_consensus.cc:2454] T 75f8eb38850343239231fc7447e0de0a P 2cf2f3fa61b34a13913081544013fd7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dbf35e45bab4691b17277d00344171a in term 1.
I20260926 23:30:44.237285 4895 leader_election.cc:302] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 230d1b5af2844860b41377ea32cdba6b, 8dbf35e45bab4691b17277d00344171a; no voters:
I20260926 23:30:44.237643 5032 raft_consensus.cc:2781] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:30:44.238154 5032 raft_consensus.cc:686] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a [term 1 LEADER]: Becoming Leader. State: Replica: 8dbf35e45bab4691b17277d00344171a, State: Running, Role: LEADER
I20260926 23:30:44.238849 5032 consensus_queue.cc:237] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbf35e45bab4691b17277d00344171a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39341 } } peers { permanent_uuid: "2cf2f3fa61b34a13913081544013fd7a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45111 } } peers { permanent_uuid: "230d1b5af2844860b41377ea32cdba6b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35851 } }
I20260926 23:30:44.244419 4552 catalog_manager.cc:5495] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a reported cstate change: term changed from 0 to 1, leader changed from <none> to 8dbf35e45bab4691b17277d00344171a (127.0.97.3). New cstate: current_term: 1 leader_uuid: "8dbf35e45bab4691b17277d00344171a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbf35e45bab4691b17277d00344171a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cf2f3fa61b34a13913081544013fd7a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "230d1b5af2844860b41377ea32cdba6b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35851 } health_report { overall_health: UNKNOWN } } }
I20260926 23:30:44.267869 5022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:44.261009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50488 (local address 127.0.97.3:39341)
0926 23:30:44.261260 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:44.261266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:44.261293 (+ 27us) server_negotiation.cc:407] Connection header received
0926 23:30:44.261480 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:44.261486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:44.261601 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:44.261794 (+ 193us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:30:44.262339 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:44.263518 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:44.265346 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:44.265791 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:44.265920 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:44.266492 (+ 572us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:30:44.267213 (+ 721us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:30:44.267365 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:44.267466 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 23:30:44.267565 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260926 23:30:44.278494 4695 raft_consensus.cc:1261] T 75f8eb38850343239231fc7447e0de0a P 2cf2f3fa61b34a13913081544013fd7a [term 1 FOLLOWER]: Refusing update from remote peer 8dbf35e45bab4691b17277d00344171a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:30:44.278568 4828 raft_consensus.cc:1261] T 75f8eb38850343239231fc7447e0de0a P 230d1b5af2844860b41377ea32cdba6b [term 1 FOLLOWER]: Refusing update from remote peer 8dbf35e45bab4691b17277d00344171a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260926 23:30:44.279474 4741 tablet.cc:2369] T 75f8eb38850343239231fc7447e0de0a P 2cf2f3fa61b34a13913081544013fd7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:30:44.279733 5032 consensus_queue.cc:1035] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf2f3fa61b34a13913081544013fd7a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:30:44.280192 5035 consensus_queue.cc:1035] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a [LEADER]: Connected to new peer: Peer: permanent_uuid: "230d1b5af2844860b41377ea32cdba6b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:30:44.319082 4551 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:30:44.319432 4551 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:30:44.324126 4551 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7a81df492694bb9837f493c948e3297: Sending DeleteTablet for 3 replicas of tablet 75f8eb38850343239231fc7447e0de0a
I20260926 23:30:44.325723 4941 tablet_service.cc:1514] Processing DeleteTablet for tablet 75f8eb38850343239231fc7447e0de0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:30:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50480
I20260926 23:30:44.325816 4808 tablet_service.cc:1514] Processing DeleteTablet for tablet 75f8eb38850343239231fc7447e0de0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:30:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58272
I20260926 23:30:44.327231 4675 tablet_service.cc:1514] Processing DeleteTablet for tablet 75f8eb38850343239231fc7447e0de0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:30:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58282
I20260926 23:30:44.328892 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 4613
I20260926 23:30:44.339160 5049 tablet_replica.cc:330] T 75f8eb38850343239231fc7447e0de0a P 230d1b5af2844860b41377ea32cdba6b: stopping tablet replica
I20260926 23:30:44.339756 5049 raft_consensus.cc:2229] T 75f8eb38850343239231fc7447e0de0a P 230d1b5af2844860b41377ea32cdba6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:30:44.340377 5049 raft_consensus.cc:2258] T 75f8eb38850343239231fc7447e0de0a P 230d1b5af2844860b41377ea32cdba6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:30:44.343398 5049 ts_tablet_manager.cc:1905] T 75f8eb38850343239231fc7447e0de0a P 230d1b5af2844860b41377ea32cdba6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:30:44.344946 5050 tablet_replica.cc:330] T 75f8eb38850343239231fc7447e0de0a P 2cf2f3fa61b34a13913081544013fd7a: stopping tablet replica
I20260926 23:30:44.345855 5050 raft_consensus.cc:2229] T 75f8eb38850343239231fc7447e0de0a P 2cf2f3fa61b34a13913081544013fd7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:30:44.346735 5050 raft_consensus.cc:2258] T 75f8eb38850343239231fc7447e0de0a P 2cf2f3fa61b34a13913081544013fd7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:30:44.347182 5048 tablet_replica.cc:330] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a: stopping tablet replica
I20260926 23:30:44.347752 5048 raft_consensus.cc:2229] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:30:44.348731 5048 raft_consensus.cc:2258] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:30:44.349876 5050 ts_tablet_manager.cc:1905] T 75f8eb38850343239231fc7447e0de0a P 2cf2f3fa61b34a13913081544013fd7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:30:44.352440 5049 ts_tablet_manager.cc:1918] T 75f8eb38850343239231fc7447e0de0a P 230d1b5af2844860b41377ea32cdba6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:30:44.352667 5049 log.cc:1196] T 75f8eb38850343239231fc7447e0de0a P 230d1b5af2844860b41377ea32cdba6b: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-1/wal/wals/75f8eb38850343239231fc7447e0de0a
I20260926 23:30:44.353247 5049 ts_tablet_manager.cc:1939] T 75f8eb38850343239231fc7447e0de0a P 230d1b5af2844860b41377ea32cdba6b: Deleting consensus metadata
I20260926 23:30:44.353682 5048 ts_tablet_manager.cc:1905] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:30:44.355463 4537 catalog_manager.cc:4841] TS 230d1b5af2844860b41377ea32cdba6b (127.0.97.2:35851): tablet 75f8eb38850343239231fc7447e0de0a (table test-table [id=65a2f4bbbb174d8da5acf4d9210ea1d9]) successfully deleted
I20260926 23:30:44.358143 5050 ts_tablet_manager.cc:1918] T 75f8eb38850343239231fc7447e0de0a P 2cf2f3fa61b34a13913081544013fd7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:30:44.358443 5050 log.cc:1196] T 75f8eb38850343239231fc7447e0de0a P 2cf2f3fa61b34a13913081544013fd7a: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-0/wal/wals/75f8eb38850343239231fc7447e0de0a
I20260926 23:30:44.359149 5050 ts_tablet_manager.cc:1939] T 75f8eb38850343239231fc7447e0de0a P 2cf2f3fa61b34a13913081544013fd7a: Deleting consensus metadata
Sep 26 23:30:44 dist-test-slave-gwfx krb5kdc[4508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:30:44.361625 4537 catalog_manager.cc:4841] TS 2cf2f3fa61b34a13913081544013fd7a (127.0.97.1:45111): tablet 75f8eb38850343239231fc7447e0de0a (table test-table [id=65a2f4bbbb174d8da5acf4d9210ea1d9]) successfully deleted
I20260926 23:30:44.361933 5048 ts_tablet_manager.cc:1918] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:30:44.362138 5048 log.cc:1196] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790465400694755-388-0/minicluster-data/ts-2/wal/wals/75f8eb38850343239231fc7447e0de0a
I20260926 23:30:44.362591 5048 ts_tablet_manager.cc:1939] T 75f8eb38850343239231fc7447e0de0a P 8dbf35e45bab4691b17277d00344171a: Deleting consensus metadata
I20260926 23:30:44.364426 4539 catalog_manager.cc:4841] TS 8dbf35e45bab4691b17277d00344171a (127.0.97.3:39341): tablet 75f8eb38850343239231fc7447e0de0a (table test-table [id=65a2f4bbbb174d8da5acf4d9210ea1d9]) successfully deleted
I20260926 23:30:44.369410 5021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:44.331667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58284 (local address 127.0.97.1:45111)
0926 23:30:44.331885 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:44.331891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:44.336722 (+ 4831us) server_negotiation.cc:407] Connection header received
0926 23:30:44.337898 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:44.337907 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:44.338061 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:44.338156 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:44.343654 (+ 5498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:44.344998 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:44.349235 (+ 4237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:44.349968 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:44.350116 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:44.360700 (+ 10584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:44.360725 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:44.360750 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:44.360820 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:44.365094 (+ 4274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:44.366086 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:44.366098 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:44.366102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:44.366297 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:44.366870 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:44.366875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:44.366878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:44.367609 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:44.367977 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:44.368590 (+ 613us) server_negotiation.cc:299] Negotiation successful
0926 23:30:44.368869 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 23:30:44.371076 4735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:30:44.491945 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 4613
I20260926 23:30:44.510331 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 4746
Sep 26 23:30:44 dist-test-slave-gwfx krb5kdc[4508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:30:44.528326 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:44.511216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58280 (local address 127.0.97.2:35851)
0926 23:30:44.511449 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:44.511456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:44.511479 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:30:44.511627 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:44.511633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:44.511739 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:44.511847 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:44.513184 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:44.514376 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:44.515732 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:44.516362 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:44.516500 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:44.520878 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:44.520902 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:44.520929 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:44.520988 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:44.525072 (+ 4084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:44.525946 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:44.525957 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:44.525961 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:44.526134 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:44.526534 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:44.526539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:44.526543 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:44.527113 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:44.527443 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:44.527688 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 23:30:44.527930 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 23:30:44.529112 4868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:30:44.626088 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 4746
I20260926 23:30:44.641743 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 4879
Sep 26 23:30:44 dist-test-slave-gwfx krb5kdc[4508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:30:44.661161 5022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:44.642873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50500 (local address 127.0.97.3:39341)
0926 23:30:44.643093 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:44.643099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:44.643127 (+ 28us) server_negotiation.cc:407] Connection header received
0926 23:30:44.643204 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:44.643210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:44.643305 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:44.643436 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:44.644863 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:44.646128 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:44.647604 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:44.648287 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:44.648449 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:44.653281 (+ 4832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:44.653307 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:44.653342 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:44.653408 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:44.657791 (+ 4383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:44.658743 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:44.658755 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:44.658759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:44.658955 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:44.659330 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:44.659335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:44.659343 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:44.659887 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:44.660251 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:44.660435 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 23:30:44.660698 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 23:30:44.662014 5001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:30:44.795806 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 4879
I20260926 23:30:44.815687 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 4521
I20260926 23:30:44.817131 4582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:30:44.935949 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 4521
2026-09-26T23:30:44Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4705 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T23:30:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:30:44Z Disabled control of system clock
I20260926 23:30:45.012425 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36833
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 23:30:45.411309 5059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:45.411733 5059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:45.411856 5059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:45.420275 5059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:30:45.420444 5059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:45.420485 5059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:45.420569 5059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:30:45.420631 5059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:30:45.431566 5059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36833
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:45.433344 5059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:45.435748 5059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:45.449966 5067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:45.450045 5064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:45.450065 5065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:45.451085 5059 server_base.cc:1034] running on GCE node
I20260926 23:30:45.452099 5059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:45.453991 5059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:45.455236 5059 hybrid_clock.cc:648] HybridClock initialized: now 1790465445455172 us; error 60 us; skew 500 ppm
I20260926 23:30:45.455727 5059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:30:45.458741 5059 webserver.cc:460] Webserver started at http://127.0.97.62:39377/ using document root <none> and password file <none>
I20260926 23:30:45.459646 5059 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:45.459761 5059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:45.460126 5059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:45.463723 5059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "f29b49398619433ebfabf0d15a30e4fa"
format_stamp: "Formatted at 2026-09-26 23:30:45 on dist-test-slave-gwfx"
server_key: "34947098683b981e2d06e66b48576b26"
server_key_iv: "e8b8843889c66c4712170f43e939705d"
server_key_version: "encryptionkey@0"
I20260926 23:30:45.464654 5059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "f29b49398619433ebfabf0d15a30e4fa"
format_stamp: "Formatted at 2026-09-26 23:30:45 on dist-test-slave-gwfx"
server_key: "34947098683b981e2d06e66b48576b26"
server_key_iv: "e8b8843889c66c4712170f43e939705d"
server_key_version: "encryptionkey@0"
I20260926 23:30:45.471405 5059 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260926 23:30:45.476048 5073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:45.478042 5059 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 23:30:45.478293 5059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "f29b49398619433ebfabf0d15a30e4fa"
format_stamp: "Formatted at 2026-09-26 23:30:45 on dist-test-slave-gwfx"
server_key: "34947098683b981e2d06e66b48576b26"
server_key_iv: "e8b8843889c66c4712170f43e939705d"
server_key_version: "encryptionkey@0"
I20260926 23:30:45.478569 5059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:45.485697 5059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:45.486819 5059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:45.487139 5059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:45.508857 5059 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36833
I20260926 23:30:45.508906 5124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36833 every 8 connection(s)
I20260926 23:30:45.510582 5059 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:30:45.515209 5125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:45.519346 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 5059
I20260926 23:30:45.519588 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:30:45.520406 388 external_mini_cluster.cc:1428] Setting key 1ebe5ab24211b234072ccc41627d410c
I20260926 23:30:45.528525 5125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f29b49398619433ebfabf0d15a30e4fa: Bootstrap starting.
I20260926 23:30:45.531788 5128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:45.523436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44962 (local address 127.0.97.62:36833)
0926 23:30:45.524845 (+ 1409us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:45.524861 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:45.524929 (+ 68us) server_negotiation.cc:407] Connection header received
0926 23:30:45.527142 (+ 2213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:45.527176 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:45.527815 (+ 639us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:45.528310 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:45.529637 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:45.529645 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:45.529662 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:30:45.529687 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:45.529759 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:30:45.529779 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:45.530033 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:45.530216 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 23:30:45.530560 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":767,"thread_start_us":439,"threads_started":1}
I20260926 23:30:45.532377 5125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f29b49398619433ebfabf0d15a30e4fa: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:45.533638 5125 log.cc:824] T 00000000000000000000000000000000 P f29b49398619433ebfabf0d15a30e4fa: Log is configured to *not* fsync() on all Append() calls
I20260926 23:30:45.537269 5125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f29b49398619433ebfabf0d15a30e4fa: No bootstrap required, opened a new log
I20260926 23:30:45.545109 5125 raft_consensus.cc:348] T 00000000000000000000000000000000 P f29b49398619433ebfabf0d15a30e4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29b49398619433ebfabf0d15a30e4fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36833 } }
I20260926 23:30:45.545572 5125 raft_consensus.cc:374] T 00000000000000000000000000000000 P f29b49398619433ebfabf0d15a30e4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:45.545667 5125 raft_consensus.cc:729] T 00000000000000000000000000000000 P f29b49398619433ebfabf0d15a30e4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f29b49398619433ebfabf0d15a30e4fa, State: Initialized, Role: FOLLOWER
I20260926 23:30:45.546371 5125 consensus_queue.cc:260] T 00000000000000000000000000000000 P f29b49398619433ebfabf0d15a30e4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29b49398619433ebfabf0d15a30e4fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36833 } }
I20260926 23:30:45.546679 5125 raft_consensus.cc:388] T 00000000000000000000000000000000 P f29b49398619433ebfabf0d15a30e4fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:30:45.546878 5125 raft_consensus.cc:482] T 00000000000000000000000000000000 P f29b49398619433ebfabf0d15a30e4fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:30:45.547031 5125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f29b49398619433ebfabf0d15a30e4fa [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:45.549594 5125 raft_consensus.cc:504] T 00000000000000000000000000000000 P f29b49398619433ebfabf0d15a30e4fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29b49398619433ebfabf0d15a30e4fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36833 } }
I20260926 23:30:45.550179 5125 leader_election.cc:302] T 00000000000000000000000000000000 P f29b49398619433ebfabf0d15a30e4fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f29b49398619433ebfabf0d15a30e4fa; no voters:
I20260926 23:30:45.550958 5125 leader_election.cc:288] T 00000000000000000000000000000000 P f29b49398619433ebfabf0d15a30e4fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:30:45.551216 5130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f29b49398619433ebfabf0d15a30e4fa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:30:45.552599 5130 raft_consensus.cc:686] T 00000000000000000000000000000000 P f29b49398619433ebfabf0d15a30e4fa [term 1 LEADER]: Becoming Leader. State: Replica: f29b49398619433ebfabf0d15a30e4fa, State: Running, Role: LEADER
I20260926 23:30:45.553336 5130 consensus_queue.cc:237] T 00000000000000000000000000000000 P f29b49398619433ebfabf0d15a30e4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29b49398619433ebfabf0d15a30e4fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36833 } }
I20260926 23:30:45.553584 5125 sys_catalog.cc:564] T 00000000000000000000000000000000 P f29b49398619433ebfabf0d15a30e4fa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:30:45.557082 5131 sys_catalog.cc:455] T 00000000000000000000000000000000 P f29b49398619433ebfabf0d15a30e4fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f29b49398619433ebfabf0d15a30e4fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29b49398619433ebfabf0d15a30e4fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36833 } } }
I20260926 23:30:45.557149 5132 sys_catalog.cc:455] T 00000000000000000000000000000000 P f29b49398619433ebfabf0d15a30e4fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader f29b49398619433ebfabf0d15a30e4fa. Latest consensus state: current_term: 1 leader_uuid: "f29b49398619433ebfabf0d15a30e4fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29b49398619433ebfabf0d15a30e4fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36833 } } }
I20260926 23:30:45.557578 5131 sys_catalog.cc:458] T 00000000000000000000000000000000 P f29b49398619433ebfabf0d15a30e4fa [sys.catalog]: This master's current role is: LEADER
I20260926 23:30:45.557672 5132 sys_catalog.cc:458] T 00000000000000000000000000000000 P f29b49398619433ebfabf0d15a30e4fa [sys.catalog]: This master's current role is: LEADER
I20260926 23:30:45.560770 5138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:30:45.568120 5138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:30:45.580336 5138 catalog_manager.cc:1347] Generated new cluster ID: 10dc69cadd5740d8bc3e92105645c019
I20260926 23:30:45.580523 5138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:30:45.596316 5138 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:30:45.597072 5138 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:30:45.616122 5138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f29b49398619433ebfabf0d15a30e4fa: Generated new TSK 0
I20260926 23:30:45.617632 5138 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:30:45.635275 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36833
--builtin_ntp_servers=127.0.97.20:45369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 23:30:45.999818 5149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:46.000139 5149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:46.000226 5149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:46.008144 5149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:46.008299 5149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:46.008417 5149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:30:46.018873 5149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:46.020210 5149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:46.022034 5149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:46.037637 5154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:46.038260 5155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:46.038414 5157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:46.039222 5149 server_base.cc:1034] running on GCE node
I20260926 23:30:46.039958 5149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:46.041059 5149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:46.042344 5149 hybrid_clock.cc:648] HybridClock initialized: now 1790465446042265 us; error 63 us; skew 500 ppm
I20260926 23:30:46.042821 5149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:30:46.045507 5149 webserver.cc:460] Webserver started at http://127.0.97.1:36887/ using document root <none> and password file <none>
I20260926 23:30:46.046259 5149 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:46.046373 5149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:46.046648 5149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:46.049695 5149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-0/data/instance:
uuid: "e7a4a8c357674155a3aef51622d2390b"
format_stamp: "Formatted at 2026-09-26 23:30:46 on dist-test-slave-gwfx"
server_key: "1c288dc8516f4ba44455e6d1a3eedf19"
server_key_iv: "bd9f7ee0760fe2a61b34d6a7fbf875ad"
server_key_version: "encryptionkey@0"
I20260926 23:30:46.050525 5149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e7a4a8c357674155a3aef51622d2390b"
format_stamp: "Formatted at 2026-09-26 23:30:46 on dist-test-slave-gwfx"
server_key: "1c288dc8516f4ba44455e6d1a3eedf19"
server_key_iv: "bd9f7ee0760fe2a61b34d6a7fbf875ad"
server_key_version: "encryptionkey@0"
I20260926 23:30:46.056695 5149 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260926 23:30:46.061241 5163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:46.063189 5149 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 23:30:46.063402 5149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "e7a4a8c357674155a3aef51622d2390b"
format_stamp: "Formatted at 2026-09-26 23:30:46 on dist-test-slave-gwfx"
server_key: "1c288dc8516f4ba44455e6d1a3eedf19"
server_key_iv: "bd9f7ee0760fe2a61b34d6a7fbf875ad"
server_key_version: "encryptionkey@0"
I20260926 23:30:46.063637 5149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:46.071141 5149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:46.072081 5149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:46.072352 5149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:46.073391 5149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:30:46.074985 5149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:30:46.075085 5149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:46.075177 5149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:30:46.075243 5149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:46.111770 5149 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39087
I20260926 23:30:46.111863 5275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39087 every 8 connection(s)
I20260926 23:30:46.113226 5149 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:30:46.118157 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 5149
I20260926 23:30:46.118414 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:30:46.119666 388 external_mini_cluster.cc:1428] Setting key 3602a7e27b45618e6e7fccfb89c4f533
I20260926 23:30:46.123013 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36833
--builtin_ntp_servers=127.0.97.20:45369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 23:30:46.125627 5278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:46.117046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47227 (local address 127.0.97.62:36833)
0926 23:30:46.117605 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:46.117610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:46.120501 (+ 2891us) server_negotiation.cc:407] Connection header received
0926 23:30:46.122780 (+ 2279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:46.122787 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:46.122883 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:46.122992 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:46.124742 (+ 1750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:46.124747 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:46.124750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:30:46.124772 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:46.124824 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:30:46.124838 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:46.124985 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:46.125193 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 23:30:46.125278 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":276,"threads_started":1}
I20260926 23:30:46.126971 5279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:46.118598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36833 (local address 127.0.97.1:47227)
0926 23:30:46.120239 (+ 1641us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:46.120293 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:46.122381 (+ 2088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:46.123371 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:46.123384 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:46.123521 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:46.124353 (+ 832us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:30:46.124356 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:46.124411 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:30:46.124524 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:46.124934 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:46.124946 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:46.124955 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 23:30:46.125286 (+ 331us) client_negotiation.cc:240] Negotiation successful
0926 23:30:46.125655 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"thread_start_us":533,"threads_started":1}
I20260926 23:30:46.129350 5276 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36833
I20260926 23:30:46.129730 5276 heartbeater.cc:464] Registering TS with master...
I20260926 23:30:46.130515 5276 heartbeater.cc:511] Master 127.0.97.62:36833 requested a full tablet report, sending...
I20260926 23:30:46.133561 5090 ts_manager.cc:194] Registered new tserver with Master: e7a4a8c357674155a3aef51622d2390b (127.0.97.1:39087)
W20260926 23:30:46.478927 5280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:46.479324 5280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:46.479451 5280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:46.487556 5280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:46.487689 5280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:46.487815 5280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:30:46.500964 5280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:46.502934 5280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:46.505101 5280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:46.521968 5286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:46.522284 5285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:46.522784 5288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:46.524075 5280 server_base.cc:1034] running on GCE node
I20260926 23:30:46.524823 5280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:46.526058 5280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:46.527294 5280 hybrid_clock.cc:648] HybridClock initialized: now 1790465446527239 us; error 49 us; skew 500 ppm
I20260926 23:30:46.527704 5280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:30:46.530495 5280 webserver.cc:460] Webserver started at http://127.0.97.2:37137/ using document root <none> and password file <none>
I20260926 23:30:46.531397 5280 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:46.531513 5280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:46.531883 5280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:46.535461 5280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-1/data/instance:
uuid: "22cad8e109ab417aad458c18372c56b1"
format_stamp: "Formatted at 2026-09-26 23:30:46 on dist-test-slave-gwfx"
server_key: "25bc14ff0450a6d755a0920f100943da"
server_key_iv: "38ba11384e84ce2e39ccd3409ebedf05"
server_key_version: "encryptionkey@0"
I20260926 23:30:46.536486 5280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-1/wal/instance:
uuid: "22cad8e109ab417aad458c18372c56b1"
format_stamp: "Formatted at 2026-09-26 23:30:46 on dist-test-slave-gwfx"
server_key: "25bc14ff0450a6d755a0920f100943da"
server_key_iv: "38ba11384e84ce2e39ccd3409ebedf05"
server_key_version: "encryptionkey@0"
I20260926 23:30:46.543567 5280 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260926 23:30:46.547906 5294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:46.549556 5280 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 23:30:46.549732 5280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "22cad8e109ab417aad458c18372c56b1"
format_stamp: "Formatted at 2026-09-26 23:30:46 on dist-test-slave-gwfx"
server_key: "25bc14ff0450a6d755a0920f100943da"
server_key_iv: "38ba11384e84ce2e39ccd3409ebedf05"
server_key_version: "encryptionkey@0"
I20260926 23:30:46.549989 5280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:46.556617 5280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:46.557572 5280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:46.557852 5280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:46.559016 5280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:30:46.560809 5280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:30:46.560910 5280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:46.561007 5280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:30:46.561064 5280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:46.595329 5280 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42369
I20260926 23:30:46.595435 5406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42369 every 8 connection(s)
I20260926 23:30:46.596966 5280 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:30:46.597651 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 5280
I20260926 23:30:46.597837 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:30:46.598482 388 external_mini_cluster.cc:1428] Setting key 0f963ed52e7a8cfd7f8ab8253a2369f0
I20260926 23:30:46.601958 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36833
--builtin_ntp_servers=127.0.97.20:45369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 23:30:46.607268 5278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:46.600236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51291 (local address 127.0.97.62:36833)
0926 23:30:46.600418 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:46.600423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:46.602634 (+ 2211us) server_negotiation.cc:407] Connection header received
0926 23:30:46.604721 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:46.604727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:46.604806 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:46.604915 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:46.606445 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:46.606451 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:46.606455 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:30:46.606477 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:46.606517 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:30:46.606533 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:46.606677 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:46.606888 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 23:30:46.606965 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 23:30:46.608480 5409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:46.601113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36833 (local address 127.0.97.2:51291)
0926 23:30:46.602396 (+ 1283us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:46.602450 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:46.604377 (+ 1927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:46.605144 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:46.605153 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:46.605273 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:46.606104 (+ 831us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:30:46.606107 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:46.606154 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:30:46.606263 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:46.606645 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:46.606654 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:46.606660 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 23:30:46.606965 (+ 305us) client_negotiation.cc:240] Negotiation successful
0926 23:30:46.607244 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":374,"threads_started":1}
I20260926 23:30:46.610095 5407 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36833
I20260926 23:30:46.610428 5407 heartbeater.cc:464] Registering TS with master...
I20260926 23:30:46.611168 5407 heartbeater.cc:511] Master 127.0.97.62:36833 requested a full tablet report, sending...
I20260926 23:30:46.612911 5090 ts_manager.cc:194] Registered new tserver with Master: 22cad8e109ab417aad458c18372c56b1 (127.0.97.2:42369)
W20260926 23:30:46.991763 5410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:46.992177 5410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:46.992306 5410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:47.000766 5410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:47.000945 5410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:47.001107 5410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:30:47.011708 5410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:47.013571 5410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:47.015983 5410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:47.033160 5415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:47.033667 5416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:47.033701 5418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:47.034225 5410 server_base.cc:1034] running on GCE node
I20260926 23:30:47.035209 5410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:47.036475 5410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:47.037701 5410 hybrid_clock.cc:648] HybridClock initialized: now 1790465447037631 us; error 56 us; skew 500 ppm
I20260926 23:30:47.038090 5410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:30:47.040922 5410 webserver.cc:460] Webserver started at http://127.0.97.3:35041/ using document root <none> and password file <none>
I20260926 23:30:47.041792 5410 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:47.041894 5410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:47.042254 5410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:47.045970 5410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-2/data/instance:
uuid: "d4e08ed2c8ce41618214a0ec9b59d40b"
format_stamp: "Formatted at 2026-09-26 23:30:47 on dist-test-slave-gwfx"
server_key: "d7794f1008a9410143a10d20c6f43384"
server_key_iv: "162d40011ced33dded38348ac8ff0e46"
server_key_version: "encryptionkey@0"
I20260926 23:30:47.047079 5410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d4e08ed2c8ce41618214a0ec9b59d40b"
format_stamp: "Formatted at 2026-09-26 23:30:47 on dist-test-slave-gwfx"
server_key: "d7794f1008a9410143a10d20c6f43384"
server_key_iv: "162d40011ced33dded38348ac8ff0e46"
server_key_version: "encryptionkey@0"
I20260926 23:30:47.053664 5410 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260926 23:30:47.058352 5424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:47.060127 5410 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:30:47.060293 5410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "d4e08ed2c8ce41618214a0ec9b59d40b"
format_stamp: "Formatted at 2026-09-26 23:30:47 on dist-test-slave-gwfx"
server_key: "d7794f1008a9410143a10d20c6f43384"
server_key_iv: "162d40011ced33dded38348ac8ff0e46"
server_key_version: "encryptionkey@0"
I20260926 23:30:47.060592 5410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:47.067597 5410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:47.068784 5410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:47.069085 5410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:47.070254 5410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:30:47.072103 5410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:30:47.072203 5410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:47.072324 5410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:30:47.072399 5410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:47.107791 5410 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45797
I20260926 23:30:47.107829 5536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45797 every 8 connection(s)
I20260926 23:30:47.109620 5410 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:30:47.118235 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 5410
I20260926 23:30:47.118426 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:30:47.119004 388 external_mini_cluster.cc:1428] Setting key fd53653a22836b2b698b270aecde19ae
I20260926 23:30:47.121455 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:47.112867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44725 (local address 127.0.97.62:36833)
0926 23:30:47.117989 (+ 5122us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:47.117996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:47.118029 (+ 33us) server_negotiation.cc:407] Connection header received
0926 23:30:47.118671 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:47.118675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:47.118773 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:47.118870 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:47.120613 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:47.120620 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:47.120625 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:30:47.120648 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:47.120693 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:30:47.120705 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:47.120809 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:47.121017 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 23:30:47.121093 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4959,"thread_start_us":4828,"threads_started":1}
I20260926 23:30:47.122738 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:47.113660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36833 (local address 127.0.97.3:44725)
0926 23:30:47.115331 (+ 1671us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:47.115408 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:47.118148 (+ 2740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:47.119276 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:47.119291 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:47.119479 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:47.120253 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:30:47.120256 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:47.120310 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:30:47.120434 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:47.120799 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:47.120811 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:47.120820 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 23:30:47.121129 (+ 309us) client_negotiation.cc:240] Negotiation successful
0926 23:30:47.121503 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"thread_start_us":420,"threads_started":1}
I20260926 23:30:47.124387 5537 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36833
I20260926 23:30:47.124774 5537 heartbeater.cc:464] Registering TS with master...
I20260926 23:30:47.125495 5537 heartbeater.cc:511] Master 127.0.97.62:36833 requested a full tablet report, sending...
I20260926 23:30:47.127512 5090 ts_manager.cc:194] Registered new tserver with Master: d4e08ed2c8ce41618214a0ec9b59d40b (127.0.97.3:45797)
I20260926 23:30:47.135674 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:30:47.137872 5276 heartbeater.cc:503] Master 127.0.97.62:36833 was elected leader, sending a full tablet report...
I20260926 23:30:47.143862 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:47.140834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40946 (local address 127.0.97.62:36833)
0926 23:30:47.141016 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:47.141021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:47.141559 (+ 538us) server_negotiation.cc:407] Connection header received
0926 23:30:47.141734 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:47.141738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:47.141799 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:47.141879 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:47.143039 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:47.143044 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:47.143047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:30:47.143065 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:47.143103 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:30:47.143117 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:47.143231 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:47.143376 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 23:30:47.143439 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 23:30:47.148901 5090 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:30:47.158365 5090 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:30:47.180145 5552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:47.174528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56470 (local address 127.0.97.2:42369)
0926 23:30:47.175258 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:47.175271 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:47.176161 (+ 890us) server_negotiation.cc:407] Connection header received
0926 23:30:47.176367 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:47.176376 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:47.176616 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:47.177827 (+ 1211us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:47.179061 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:47.179071 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:47.179089 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:30:47.179112 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:47.179161 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:30:47.179176 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:47.179319 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:47.179641 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 23:30:47.179767 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":340,"threads_started":1}
I20260926 23:30:47.180315 5550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:47.174104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42369 (local address 127.0.0.1:56470)
0926 23:30:47.175969 (+ 1865us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:47.176028 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:47.176211 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:47.176815 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:47.176823 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:47.176947 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:47.178282 (+ 1335us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:30:47.178286 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:47.178324 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:30:47.178435 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:47.179285 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:47.179297 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:47.179309 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 23:30:47.179559 (+ 250us) client_negotiation.cc:240] Negotiation successful
0926 23:30:47.179735 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1600,"thread_start_us":1438,"threads_started":1}
I20260926 23:30:47.182857 5553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:47.174733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57760 (local address 127.0.97.3:45797)
0926 23:30:47.175639 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:47.175647 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:47.176836 (+ 1189us) server_negotiation.cc:407] Connection header received
0926 23:30:47.177087 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:47.177095 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:47.177312 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:47.177850 (+ 538us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:47.181632 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:47.181643 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:47.181659 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:30:47.181683 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:47.181764 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:30:47.181787 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:47.181943 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:47.182222 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 23:30:47.182358 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":522,"threads_started":1}
I20260926 23:30:47.182900 5555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:47.177486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50676 (local address 127.0.97.1:39087)
0926 23:30:47.178992 (+ 1506us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:47.179002 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:47.179947 (+ 945us) server_negotiation.cc:407] Connection header received
0926 23:30:47.180152 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:47.180159 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:47.180716 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:47.180888 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:47.182021 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:47.182028 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:47.182041 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:30:47.182084 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:47.182143 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:30:47.182164 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:47.182329 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:47.182387 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 23:30:47.182469 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1321,"thread_start_us":1121,"threads_started":1}
I20260926 23:30:47.182906 5554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:47.176874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39087 (local address 127.0.0.1:50676)
0926 23:30:47.179816 (+ 2942us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:47.179831 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:47.180024 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:47.180870 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:47.180874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:47.180896 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:47.181549 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:30:47.181552 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:47.181567 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:30:47.181608 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:47.182252 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:47.182257 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:47.182262 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 23:30:47.182398 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 23:30:47.182457 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2840,"thread_start_us":2632,"threads_started":1}
I20260926 23:30:47.184051 5551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:47.174301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45797 (local address 127.0.0.1:57760)
0926 23:30:47.176712 (+ 2411us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:47.176729 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:47.176921 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:47.179165 (+ 2244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:47.179171 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:47.179204 (+ 33us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:47.181348 (+ 2144us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:30:47.181351 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:47.181370 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:30:47.181416 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:47.182032 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:47.182037 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:47.182041 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 23:30:47.183584 (+ 1543us) client_negotiation.cc:240] Negotiation successful
0926 23:30:47.183630 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2276,"thread_start_us":2148,"threads_started":1}
I20260926 23:30:47.186257 5342 tablet_service.cc:1467] Processing CreateTablet for tablet 90c01549b7944bf5bae84995ee018681 (DEFAULT_TABLE table=test-table [id=578f18193b494470bc19c13624700031]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:30:47.187064 5211 tablet_service.cc:1467] Processing CreateTablet for tablet 90c01549b7944bf5bae84995ee018681 (DEFAULT_TABLE table=test-table [id=578f18193b494470bc19c13624700031]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:30:47.188120 5342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90c01549b7944bf5bae84995ee018681. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:47.188494 5211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90c01549b7944bf5bae84995ee018681. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:47.188323 5472 tablet_service.cc:1467] Processing CreateTablet for tablet 90c01549b7944bf5bae84995ee018681 (DEFAULT_TABLE table=test-table [id=578f18193b494470bc19c13624700031]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:30:47.189800 5472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90c01549b7944bf5bae84995ee018681. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:47.203088 5556 tablet_bootstrap.cc:492] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1: Bootstrap starting.
I20260926 23:30:47.205986 5557 tablet_bootstrap.cc:492] T 90c01549b7944bf5bae84995ee018681 P e7a4a8c357674155a3aef51622d2390b: Bootstrap starting.
I20260926 23:30:47.208487 5558 tablet_bootstrap.cc:492] T 90c01549b7944bf5bae84995ee018681 P d4e08ed2c8ce41618214a0ec9b59d40b: Bootstrap starting.
I20260926 23:30:47.210161 5556 tablet_bootstrap.cc:654] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:47.210819 5557 tablet_bootstrap.cc:654] T 90c01549b7944bf5bae84995ee018681 P e7a4a8c357674155a3aef51622d2390b: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:47.211984 5556 log.cc:824] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1: Log is configured to *not* fsync() on all Append() calls
I20260926 23:30:47.212163 5557 log.cc:824] T 90c01549b7944bf5bae84995ee018681 P e7a4a8c357674155a3aef51622d2390b: Log is configured to *not* fsync() on all Append() calls
I20260926 23:30:47.212661 5558 tablet_bootstrap.cc:654] T 90c01549b7944bf5bae84995ee018681 P d4e08ed2c8ce41618214a0ec9b59d40b: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:47.214231 5558 log.cc:824] T 90c01549b7944bf5bae84995ee018681 P d4e08ed2c8ce41618214a0ec9b59d40b: Log is configured to *not* fsync() on all Append() calls
I20260926 23:30:47.216622 5556 tablet_bootstrap.cc:492] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1: No bootstrap required, opened a new log
I20260926 23:30:47.216622 5557 tablet_bootstrap.cc:492] T 90c01549b7944bf5bae84995ee018681 P e7a4a8c357674155a3aef51622d2390b: No bootstrap required, opened a new log
I20260926 23:30:47.217072 5557 ts_tablet_manager.cc:1397] T 90c01549b7944bf5bae84995ee018681 P e7a4a8c357674155a3aef51622d2390b: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260926 23:30:47.217073 5556 ts_tablet_manager.cc:1397] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260926 23:30:47.218326 5558 tablet_bootstrap.cc:492] T 90c01549b7944bf5bae84995ee018681 P d4e08ed2c8ce41618214a0ec9b59d40b: No bootstrap required, opened a new log
I20260926 23:30:47.218801 5558 ts_tablet_manager.cc:1397] T 90c01549b7944bf5bae84995ee018681 P d4e08ed2c8ce41618214a0ec9b59d40b: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260926 23:30:47.227612 5558 raft_consensus.cc:348] T 90c01549b7944bf5bae84995ee018681 P d4e08ed2c8ce41618214a0ec9b59d40b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22cad8e109ab417aad458c18372c56b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42369 } } peers { permanent_uuid: "d4e08ed2c8ce41618214a0ec9b59d40b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45797 } } peers { permanent_uuid: "e7a4a8c357674155a3aef51622d2390b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39087 } }
I20260926 23:30:47.228585 5558 raft_consensus.cc:374] T 90c01549b7944bf5bae84995ee018681 P d4e08ed2c8ce41618214a0ec9b59d40b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:47.228787 5558 raft_consensus.cc:729] T 90c01549b7944bf5bae84995ee018681 P d4e08ed2c8ce41618214a0ec9b59d40b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e08ed2c8ce41618214a0ec9b59d40b, State: Initialized, Role: FOLLOWER
I20260926 23:30:47.229343 5556 raft_consensus.cc:348] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22cad8e109ab417aad458c18372c56b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42369 } } peers { permanent_uuid: "d4e08ed2c8ce41618214a0ec9b59d40b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45797 } } peers { permanent_uuid: "e7a4a8c357674155a3aef51622d2390b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39087 } }
I20260926 23:30:47.229334 5557 raft_consensus.cc:348] T 90c01549b7944bf5bae84995ee018681 P e7a4a8c357674155a3aef51622d2390b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22cad8e109ab417aad458c18372c56b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42369 } } peers { permanent_uuid: "d4e08ed2c8ce41618214a0ec9b59d40b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45797 } } peers { permanent_uuid: "e7a4a8c357674155a3aef51622d2390b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39087 } }
I20260926 23:30:47.229874 5556 raft_consensus.cc:374] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:47.229882 5557 raft_consensus.cc:374] T 90c01549b7944bf5bae84995ee018681 P e7a4a8c357674155a3aef51622d2390b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:47.229722 5558 consensus_queue.cc:260] T 90c01549b7944bf5bae84995ee018681 P d4e08ed2c8ce41618214a0ec9b59d40b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22cad8e109ab417aad458c18372c56b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42369 } } peers { permanent_uuid: "d4e08ed2c8ce41618214a0ec9b59d40b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45797 } } peers { permanent_uuid: "e7a4a8c357674155a3aef51622d2390b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39087 } }
I20260926 23:30:47.229990 5557 raft_consensus.cc:729] T 90c01549b7944bf5bae84995ee018681 P e7a4a8c357674155a3aef51622d2390b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7a4a8c357674155a3aef51622d2390b, State: Initialized, Role: FOLLOWER
I20260926 23:30:47.230041 5556 raft_consensus.cc:729] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22cad8e109ab417aad458c18372c56b1, State: Initialized, Role: FOLLOWER
I20260926 23:30:47.230746 5556 consensus_queue.cc:260] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22cad8e109ab417aad458c18372c56b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42369 } } peers { permanent_uuid: "d4e08ed2c8ce41618214a0ec9b59d40b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45797 } } peers { permanent_uuid: "e7a4a8c357674155a3aef51622d2390b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39087 } }
I20260926 23:30:47.230746 5557 consensus_queue.cc:260] T 90c01549b7944bf5bae84995ee018681 P e7a4a8c357674155a3aef51622d2390b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22cad8e109ab417aad458c18372c56b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42369 } } peers { permanent_uuid: "d4e08ed2c8ce41618214a0ec9b59d40b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45797 } } peers { permanent_uuid: "e7a4a8c357674155a3aef51622d2390b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39087 } }
I20260926 23:30:47.231930 5537 heartbeater.cc:503] Master 127.0.97.62:36833 was elected leader, sending a full tablet report...
I20260926 23:30:47.232720 5558 ts_tablet_manager.cc:1428] T 90c01549b7944bf5bae84995ee018681 P d4e08ed2c8ce41618214a0ec9b59d40b: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260926 23:30:47.236826 5557 ts_tablet_manager.cc:1428] T 90c01549b7944bf5bae84995ee018681 P e7a4a8c357674155a3aef51622d2390b: Time spent starting tablet: real 0.020s user 0.014s sys 0.004s
I20260926 23:30:47.238556 5407 heartbeater.cc:503] Master 127.0.97.62:36833 was elected leader, sending a full tablet report...
I20260926 23:30:47.240624 5556 ts_tablet_manager.cc:1428] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1: Time spent starting tablet: real 0.023s user 0.011s sys 0.008s
I20260926 23:30:47.247231 5564 raft_consensus.cc:482] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:30:47.247627 5564 raft_consensus.cc:504] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22cad8e109ab417aad458c18372c56b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42369 } } peers { permanent_uuid: "d4e08ed2c8ce41618214a0ec9b59d40b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45797 } } peers { permanent_uuid: "e7a4a8c357674155a3aef51622d2390b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39087 } }
I20260926 23:30:47.249758 5564 leader_election.cc:288] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4e08ed2c8ce41618214a0ec9b59d40b (127.0.97.3:45797), e7a4a8c357674155a3aef51622d2390b (127.0.97.1:39087)
I20260926 23:30:47.253775 5553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:47.250379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57549 (local address 127.0.97.3:45797)
0926 23:30:47.250581 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:47.250586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:47.251364 (+ 778us) server_negotiation.cc:407] Connection header received
0926 23:30:47.251467 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:47.251474 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:47.251554 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:47.251676 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:47.253062 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:47.253069 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:47.253075 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:30:47.253101 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:47.253147 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:30:47.253163 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:47.253301 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:47.253387 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 23:30:47.253467 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 23:30:47.254014 5565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:47.250174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45797 (local address 127.0.97.2:57549)
0926 23:30:47.251059 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:47.251082 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:47.251244 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:47.251877 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:47.251881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:47.251904 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:47.252811 (+ 907us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:30:47.252815 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:47.252832 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:30:47.252876 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:47.253262 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:47.253267 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:47.253272 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 23:30:47.253426 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 23:30:47.253475 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":498,"threads_started":1}
I20260926 23:30:47.254035 5555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:47.251590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42017 (local address 127.0.97.1:39087)
0926 23:30:47.251786 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:47.251792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:47.252036 (+ 244us) server_negotiation.cc:407] Connection header received
0926 23:30:47.252151 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:47.252156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:47.252225 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:47.252332 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:47.253396 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:47.253402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:47.253406 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:30:47.253425 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:47.253464 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:30:47.253480 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:47.253601 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:47.253707 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 23:30:47.253768 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 23:30:47.254299 5566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:47.251287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39087 (local address 127.0.97.2:42017)
0926 23:30:47.251933 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:47.251948 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:47.252052 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:47.252470 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:47.252475 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:47.252499 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:47.253163 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:30:47.253165 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:47.253179 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:30:47.253216 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:47.253583 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:47.253589 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:47.253594 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 23:30:47.253733 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 23:30:47.253787 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":314,"threads_started":1}
I20260926 23:30:47.255414 5492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c01549b7944bf5bae84995ee018681" candidate_uuid: "22cad8e109ab417aad458c18372c56b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4e08ed2c8ce41618214a0ec9b59d40b" is_pre_election: true
I20260926 23:30:47.255445 5231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c01549b7944bf5bae84995ee018681" candidate_uuid: "22cad8e109ab417aad458c18372c56b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7a4a8c357674155a3aef51622d2390b" is_pre_election: true
I20260926 23:30:47.256047 5492 raft_consensus.cc:2454] T 90c01549b7944bf5bae84995ee018681 P d4e08ed2c8ce41618214a0ec9b59d40b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22cad8e109ab417aad458c18372c56b1 in term 0.
I20260926 23:30:47.256044 5231 raft_consensus.cc:2454] T 90c01549b7944bf5bae84995ee018681 P e7a4a8c357674155a3aef51622d2390b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22cad8e109ab417aad458c18372c56b1 in term 0.
I20260926 23:30:47.256773 5298 leader_election.cc:302] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22cad8e109ab417aad458c18372c56b1, d4e08ed2c8ce41618214a0ec9b59d40b; no voters:
I20260926 23:30:47.257222 5564 raft_consensus.cc:2781] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:30:47.257364 5564 raft_consensus.cc:482] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:30:47.257463 5564 raft_consensus.cc:3037] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:47.260257 5564 raft_consensus.cc:504] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22cad8e109ab417aad458c18372c56b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42369 } } peers { permanent_uuid: "d4e08ed2c8ce41618214a0ec9b59d40b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45797 } } peers { permanent_uuid: "e7a4a8c357674155a3aef51622d2390b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39087 } }
I20260926 23:30:47.260994 5564 leader_election.cc:288] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1 [CANDIDATE]: Term 1 election: Requested vote from peers d4e08ed2c8ce41618214a0ec9b59d40b (127.0.97.3:45797), e7a4a8c357674155a3aef51622d2390b (127.0.97.1:39087)
I20260926 23:30:47.261677 5492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c01549b7944bf5bae84995ee018681" candidate_uuid: "22cad8e109ab417aad458c18372c56b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4e08ed2c8ce41618214a0ec9b59d40b"
I20260926 23:30:47.261766 5231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c01549b7944bf5bae84995ee018681" candidate_uuid: "22cad8e109ab417aad458c18372c56b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7a4a8c357674155a3aef51622d2390b"
I20260926 23:30:47.261945 5492 raft_consensus.cc:3037] T 90c01549b7944bf5bae84995ee018681 P d4e08ed2c8ce41618214a0ec9b59d40b [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:47.261999 5231 raft_consensus.cc:3037] T 90c01549b7944bf5bae84995ee018681 P e7a4a8c357674155a3aef51622d2390b [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:47.264734 5231 raft_consensus.cc:2454] T 90c01549b7944bf5bae84995ee018681 P e7a4a8c357674155a3aef51622d2390b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22cad8e109ab417aad458c18372c56b1 in term 1.
I20260926 23:30:47.264748 5492 raft_consensus.cc:2454] T 90c01549b7944bf5bae84995ee018681 P d4e08ed2c8ce41618214a0ec9b59d40b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22cad8e109ab417aad458c18372c56b1 in term 1.
I20260926 23:30:47.265300 5298 leader_election.cc:302] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22cad8e109ab417aad458c18372c56b1, e7a4a8c357674155a3aef51622d2390b; no voters:
I20260926 23:30:47.265725 5564 raft_consensus.cc:2781] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:30:47.266271 5564 raft_consensus.cc:686] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1 [term 1 LEADER]: Becoming Leader. State: Replica: 22cad8e109ab417aad458c18372c56b1, State: Running, Role: LEADER
I20260926 23:30:47.266937 5564 consensus_queue.cc:237] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22cad8e109ab417aad458c18372c56b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42369 } } peers { permanent_uuid: "d4e08ed2c8ce41618214a0ec9b59d40b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45797 } } peers { permanent_uuid: "e7a4a8c357674155a3aef51622d2390b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39087 } }
I20260926 23:30:47.272694 5090 catalog_manager.cc:5495] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22cad8e109ab417aad458c18372c56b1 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "22cad8e109ab417aad458c18372c56b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22cad8e109ab417aad458c18372c56b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4e08ed2c8ce41618214a0ec9b59d40b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7a4a8c357674155a3aef51622d2390b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39087 } health_report { overall_health: UNKNOWN } } }
I20260926 23:30:47.290342 5552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:47.287720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56478 (local address 127.0.97.2:42369)
0926 23:30:47.287904 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:47.287909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:47.287928 (+ 19us) server_negotiation.cc:407] Connection header received
0926 23:30:47.288047 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:47.288051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:47.288196 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:47.288299 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:47.289646 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:47.289652 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:47.289657 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:30:47.289683 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:47.289733 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:30:47.289753 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:47.289905 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:47.290008 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 23:30:47.290071 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 23:30:47.301443 5492 raft_consensus.cc:1261] T 90c01549b7944bf5bae84995ee018681 P d4e08ed2c8ce41618214a0ec9b59d40b [term 1 FOLLOWER]: Refusing update from remote peer 22cad8e109ab417aad458c18372c56b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:30:47.301492 5231 raft_consensus.cc:1261] T 90c01549b7944bf5bae84995ee018681 P e7a4a8c357674155a3aef51622d2390b [term 1 FOLLOWER]: Refusing update from remote peer 22cad8e109ab417aad458c18372c56b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:30:47.302917 5564 consensus_queue.cc:1035] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4e08ed2c8ce41618214a0ec9b59d40b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:30:47.303361 5567 consensus_queue.cc:1035] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7a4a8c357674155a3aef51622d2390b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:30:47.345177 5090 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:30:47.345583 5090 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:30:47.349874 5090 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f29b49398619433ebfabf0d15a30e4fa: Sending DeleteTablet for 3 replicas of tablet 90c01549b7944bf5bae84995ee018681
I20260926 23:30:47.351341 5342 tablet_service.cc:1514] Processing DeleteTablet for tablet 90c01549b7944bf5bae84995ee018681 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:30:47 UTC) from {username='slave'} at 127.0.0.1:56470
I20260926 23:30:47.351575 5472 tablet_service.cc:1514] Processing DeleteTablet for tablet 90c01549b7944bf5bae84995ee018681 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:30:47 UTC) from {username='slave'} at 127.0.0.1:57760
I20260926 23:30:47.351792 5211 tablet_service.cc:1514] Processing DeleteTablet for tablet 90c01549b7944bf5bae84995ee018681 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:30:47 UTC) from {username='slave'} at 127.0.0.1:50676
I20260926 23:30:47.357043 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 5149
I20260926 23:30:47.365352 5582 tablet_replica.cc:330] T 90c01549b7944bf5bae84995ee018681 P e7a4a8c357674155a3aef51622d2390b: stopping tablet replica
I20260926 23:30:47.366873 5582 raft_consensus.cc:2229] T 90c01549b7944bf5bae84995ee018681 P e7a4a8c357674155a3aef51622d2390b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:30:47.367710 5582 raft_consensus.cc:2258] T 90c01549b7944bf5bae84995ee018681 P e7a4a8c357674155a3aef51622d2390b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:30:47.370612 5582 ts_tablet_manager.cc:1905] T 90c01549b7944bf5bae84995ee018681 P e7a4a8c357674155a3aef51622d2390b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:30:47.379580 5582 ts_tablet_manager.cc:1918] T 90c01549b7944bf5bae84995ee018681 P e7a4a8c357674155a3aef51622d2390b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:30:47.379967 5582 log.cc:1196] T 90c01549b7944bf5bae84995ee018681 P e7a4a8c357674155a3aef51622d2390b: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-0/wal/wals/90c01549b7944bf5bae84995ee018681
I20260926 23:30:47.380662 5582 ts_tablet_manager.cc:1939] T 90c01549b7944bf5bae84995ee018681 P e7a4a8c357674155a3aef51622d2390b: Deleting consensus metadata
I20260926 23:30:47.382973 5555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:47.358820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50682 (local address 127.0.97.1:39087)
0926 23:30:47.359059 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:47.359067 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:47.379741 (+ 20674us) server_negotiation.cc:407] Connection header received
0926 23:30:47.380399 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:47.380405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:47.380552 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:47.380641 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:47.382048 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:47.382073 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:47.382078 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:30:47.382101 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:47.382148 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:30:47.382166 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:47.382283 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:47.382537 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 23:30:47.382610 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 23:30:47.383503 5077 catalog_manager.cc:4841] TS e7a4a8c357674155a3aef51622d2390b (127.0.97.1:39087): tablet 90c01549b7944bf5bae84995ee018681 (table test-table [id=578f18193b494470bc19c13624700031]) successfully deleted
I20260926 23:30:47.384871 5271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:30:47.385473 5580 tablet_replica.cc:330] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1: stopping tablet replica
I20260926 23:30:47.385690 5581 tablet_replica.cc:330] T 90c01549b7944bf5bae84995ee018681 P d4e08ed2c8ce41618214a0ec9b59d40b: stopping tablet replica
I20260926 23:30:47.386108 5580 raft_consensus.cc:2229] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:30:47.386193 5581 raft_consensus.cc:2229] T 90c01549b7944bf5bae84995ee018681 P d4e08ed2c8ce41618214a0ec9b59d40b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:30:47.386842 5581 raft_consensus.cc:2258] T 90c01549b7944bf5bae84995ee018681 P d4e08ed2c8ce41618214a0ec9b59d40b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:30:47.387451 5580 raft_consensus.cc:2258] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:30:47.389827 5581 ts_tablet_manager.cc:1905] T 90c01549b7944bf5bae84995ee018681 P d4e08ed2c8ce41618214a0ec9b59d40b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:30:47.389840 5580 ts_tablet_manager.cc:1905] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:30:47.398074 5581 ts_tablet_manager.cc:1918] T 90c01549b7944bf5bae84995ee018681 P d4e08ed2c8ce41618214a0ec9b59d40b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:30:47.398330 5581 log.cc:1196] T 90c01549b7944bf5bae84995ee018681 P d4e08ed2c8ce41618214a0ec9b59d40b: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-2/wal/wals/90c01549b7944bf5bae84995ee018681
I20260926 23:30:47.398463 5580 ts_tablet_manager.cc:1918] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:30:47.398723 5580 log.cc:1196] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790465400694755-388-0/minicluster-data/ts-1/wal/wals/90c01549b7944bf5bae84995ee018681
I20260926 23:30:47.398993 5581 ts_tablet_manager.cc:1939] T 90c01549b7944bf5bae84995ee018681 P d4e08ed2c8ce41618214a0ec9b59d40b: Deleting consensus metadata
I20260926 23:30:47.399513 5580 ts_tablet_manager.cc:1939] T 90c01549b7944bf5bae84995ee018681 P 22cad8e109ab417aad458c18372c56b1: Deleting consensus metadata
I20260926 23:30:47.401547 5077 catalog_manager.cc:4841] TS d4e08ed2c8ce41618214a0ec9b59d40b (127.0.97.3:45797): tablet 90c01549b7944bf5bae84995ee018681 (table test-table [id=578f18193b494470bc19c13624700031]) successfully deleted
I20260926 23:30:47.401935 5075 catalog_manager.cc:4841] TS 22cad8e109ab417aad458c18372c56b1 (127.0.97.2:42369): tablet 90c01549b7944bf5bae84995ee018681 (table test-table [id=578f18193b494470bc19c13624700031]) successfully deleted
I20260926 23:30:47.488581 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 5149
I20260926 23:30:47.505326 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 5280
I20260926 23:30:47.508754 5552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:47.506302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56494 (local address 127.0.97.2:42369)
0926 23:30:47.506520 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:47.506527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:47.506562 (+ 35us) server_negotiation.cc:407] Connection header received
0926 23:30:47.506755 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:47.506771 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:47.506848 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:47.506972 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:47.507966 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:47.507973 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:47.507979 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:30:47.508005 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:47.508050 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:30:47.508068 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:47.508226 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:47.508380 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 23:30:47.508447 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 23:30:47.509892 5402 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:30:47.639492 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 5280
I20260926 23:30:47.656702 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 5410
I20260926 23:30:47.660213 5553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:47.657631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57762 (local address 127.0.97.3:45797)
0926 23:30:47.657824 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:47.657830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:47.657957 (+ 127us) server_negotiation.cc:407] Connection header received
0926 23:30:47.658172 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:47.658176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:47.658270 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:47.658366 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:47.659472 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:47.659477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:47.659480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:30:47.659500 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:47.659549 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:30:47.659568 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:47.659716 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:47.659806 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 23:30:47.659883 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 23:30:47.661021 5532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:30:47.755312 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 5410
I20260926 23:30:47.770376 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 5059
I20260926 23:30:47.771817 5120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:30:47.889920 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 5059
2026-09-26T23:30:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2951 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 23:30:47.931857 388 mini_oidc.cc:152] Starting JWKS server
I20260926 23:30:47.934859 388 webserver.cc:460] Webserver started at https://127.0.0.1:37789/ using document root <none> and password file <none>
I20260926 23:30:47.935204 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 23:30:47.935951 388 webserver.cc:460] Webserver started at http://127.0.0.1:34363/ using document root <none> and password file <none>
2026-09-26T23:30:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:30:47Z Disabled control of system clock
I20260926 23:30:47.968230 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46557
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46557
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37789/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790465400694755-388-0/testchainca.cert with env {}
W20260926 23:30:48.371500 5593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:48.371999 5593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:48.372123 5593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:48.380539 5593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:30:48.380716 5593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:48.380755 5593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:48.380825 5593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:30:48.380873 5593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 23:30:48.380941 5593 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 23:30:48.380982 5593 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37789/jwks.json
I20260926 23:30:48.392747 5593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46557
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37789/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790465400694755-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:48.394714 5593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:48.396348 5593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:48.410270 5601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:48.410418 5599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:48.410549 5598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:48.411641 5593 server_base.cc:1034] running on GCE node
I20260926 23:30:48.412668 5593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:48.414582 5593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:48.415833 5593 hybrid_clock.cc:648] HybridClock initialized: now 1790465448415769 us; error 63 us; skew 500 ppm
I20260926 23:30:48.416272 5593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:30:48.419286 5593 webserver.cc:460] Webserver started at http://127.0.97.62:34817/ using document root <none> and password file <none>
I20260926 23:30:48.420166 5593 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:48.420287 5593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:48.420631 5593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:48.425908 5593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "7b3561b62ac04b9ba56c1b79323592ae"
format_stamp: "Formatted at 2026-09-26 23:30:48 on dist-test-slave-gwfx"
server_key: "25133d84577488de0e2aaa8852f16016"
server_key_iv: "a5553bdc56d1bcbd9738bd67079a6d0d"
server_key_version: "encryptionkey@0"
I20260926 23:30:48.427184 5593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "7b3561b62ac04b9ba56c1b79323592ae"
format_stamp: "Formatted at 2026-09-26 23:30:48 on dist-test-slave-gwfx"
server_key: "25133d84577488de0e2aaa8852f16016"
server_key_iv: "a5553bdc56d1bcbd9738bd67079a6d0d"
server_key_version: "encryptionkey@0"
I20260926 23:30:48.434119 5593 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 23:30:48.438710 5607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:48.440698 5593 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 23:30:48.440948 5593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "7b3561b62ac04b9ba56c1b79323592ae"
format_stamp: "Formatted at 2026-09-26 23:30:48 on dist-test-slave-gwfx"
server_key: "25133d84577488de0e2aaa8852f16016"
server_key_iv: "a5553bdc56d1bcbd9738bd67079a6d0d"
server_key_version: "encryptionkey@0"
I20260926 23:30:48.441223 5593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:48.465832 5593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:48.466977 5593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:48.467298 5593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:48.487222 5593 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46557
I20260926 23:30:48.487248 5658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46557 every 8 connection(s)
I20260926 23:30:48.488859 5593 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:30:48.494591 5659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:48.494791 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 5593
I20260926 23:30:48.495010 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:30:48.495853 388 external_mini_cluster.cc:1428] Setting key 0f3917ae7d5ea2f4240080a278db4a3c
I20260926 23:30:48.512625 5659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b3561b62ac04b9ba56c1b79323592ae: Bootstrap starting.
I20260926 23:30:48.515846 5662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:48.498439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36482 (local address 127.0.97.62:46557)
0926 23:30:48.502159 (+ 3720us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:48.502188 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:48.502258 (+ 70us) server_negotiation.cc:407] Connection header received
0926 23:30:48.504144 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:48.504187 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:48.504991 (+ 804us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:48.505533 (+ 542us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:48.506135 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:48.508147 (+ 2012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:48.509730 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:48.510922 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:48.511120 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:48.512459 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:48.512471 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:48.512495 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:30:48.513330 (+ 835us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:48.513457 (+ 127us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:30:48.513489 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:48.513599 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:48.513861 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 23:30:48.514272 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3048,"thread_start_us":2718,"threads_started":1}
I20260926 23:30:48.517143 5659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b3561b62ac04b9ba56c1b79323592ae: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:48.518332 5659 log.cc:824] T 00000000000000000000000000000000 P 7b3561b62ac04b9ba56c1b79323592ae: Log is configured to *not* fsync() on all Append() calls
I20260926 23:30:48.520517 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:30:48.522076 5659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b3561b62ac04b9ba56c1b79323592ae: No bootstrap required, opened a new log
I20260926 23:30:48.530735 5659 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b3561b62ac04b9ba56c1b79323592ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b3561b62ac04b9ba56c1b79323592ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46557 } }
I20260926 23:30:48.531265 5659 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b3561b62ac04b9ba56c1b79323592ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:48.531384 5659 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b3561b62ac04b9ba56c1b79323592ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b3561b62ac04b9ba56c1b79323592ae, State: Initialized, Role: FOLLOWER
I20260926 23:30:48.532114 5659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b3561b62ac04b9ba56c1b79323592ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b3561b62ac04b9ba56c1b79323592ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46557 } }
I20260926 23:30:48.532369 5659 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b3561b62ac04b9ba56c1b79323592ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:30:48.532503 5659 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b3561b62ac04b9ba56c1b79323592ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:30:48.532667 5659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b3561b62ac04b9ba56c1b79323592ae [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:48.535375 5659 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b3561b62ac04b9ba56c1b79323592ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b3561b62ac04b9ba56c1b79323592ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46557 } }
I20260926 23:30:48.536006 5659 leader_election.cc:302] T 00000000000000000000000000000000 P 7b3561b62ac04b9ba56c1b79323592ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b3561b62ac04b9ba56c1b79323592ae; no voters:
I20260926 23:30:48.536703 5659 leader_election.cc:288] T 00000000000000000000000000000000 P 7b3561b62ac04b9ba56c1b79323592ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:30:48.536839 5664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b3561b62ac04b9ba56c1b79323592ae [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:30:48.537575 5664 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b3561b62ac04b9ba56c1b79323592ae [term 1 LEADER]: Becoming Leader. State: Replica: 7b3561b62ac04b9ba56c1b79323592ae, State: Running, Role: LEADER
I20260926 23:30:48.538195 5664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b3561b62ac04b9ba56c1b79323592ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b3561b62ac04b9ba56c1b79323592ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46557 } }
I20260926 23:30:48.539314 5659 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b3561b62ac04b9ba56c1b79323592ae [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:30:48.541764 5665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b3561b62ac04b9ba56c1b79323592ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b3561b62ac04b9ba56c1b79323592ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b3561b62ac04b9ba56c1b79323592ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46557 } } }
I20260926 23:30:48.542312 5665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b3561b62ac04b9ba56c1b79323592ae [sys.catalog]: This master's current role is: LEADER
I20260926 23:30:48.542107 5666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b3561b62ac04b9ba56c1b79323592ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b3561b62ac04b9ba56c1b79323592ae. Latest consensus state: current_term: 1 leader_uuid: "7b3561b62ac04b9ba56c1b79323592ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b3561b62ac04b9ba56c1b79323592ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46557 } } }
I20260926 23:30:48.542500 5666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b3561b62ac04b9ba56c1b79323592ae [sys.catalog]: This master's current role is: LEADER
I20260926 23:30:48.543421 5672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:30:48.549803 5672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:30:48.561851 5672 catalog_manager.cc:1347] Generated new cluster ID: f8d5b87d30e64003b9f97e57b7fecec5
I20260926 23:30:48.561996 5672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:30:48.580747 5672 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:30:48.583012 5672 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:30:48.599061 5672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b3561b62ac04b9ba56c1b79323592ae: Generated new TSK 0
I20260926 23:30:48.600580 5672 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 23:30:48.647042 5689 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 23:30:48.647275 5689 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 23:30:48.648201 5662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:48.617738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36496 (local address 127.0.97.62:46557)
0926 23:30:48.618007 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:48.618015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:48.618486 (+ 471us) server_negotiation.cc:407] Connection header received
0926 23:30:48.618897 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:48.618902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:48.618991 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:48.619143 (+ 152us) server_negotiation.cc:226] Negotiated authn=JWT
0926 23:30:48.619802 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:48.620632 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:48.622895 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:48.623423 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:48.623528 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:48.624313 (+ 785us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 23:30:48.647551 (+ 23238us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37789/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 23:30:48.647946 (+ 395us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36496: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37789/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":75}
W20260926 23:30:48.648334 5662 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36496: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37789/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 23:30:48.650532 5688 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:30:48.617570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46557 (local address 127.0.0.1:36496)
0926 23:30:48.618345 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:48.618378 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:48.618608 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:48.619197 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:48.619203 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:48.619217 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0926 23:30:48.619663 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:48.619675 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:48.620779 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:48.620785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:48.622730 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:48.622751 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:48.623548 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:48.623631 (+ 83us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:48.623830 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:48.623933 (+ 103us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 23:30:48.647916 (+ 23983us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37789/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 23:30:48.648347 (+ 431us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:46557: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37789/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":413,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:46557: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37789/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 23:30:48.652520 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 5593
I20260926 23:30:48.653973 5654 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:30:48.755705 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 5593
2026-09-26T23:30:48Z chronyd exiting
I20260926 23:30:48.806406 388 test_util.cc:175] -----------------------------------------------
I20260926 23:30:48.806615 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790465400694755-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (879 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 23:30:48.811000 388 mini_oidc.cc:152] Starting JWKS server
I20260926 23:30:48.813304 388 webserver.cc:460] Webserver started at https://127.0.0.1:38715/ using document root <none> and password file <none>
I20260926 23:30:48.813585 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 23:30:48.814366 388 webserver.cc:460] Webserver started at http://127.0.0.1:40163/ using document root <none> and password file <none>
2026-09-26T23:30:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:30:48Z Disabled control of system clock
I20260926 23:30:48.846612 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45241
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45241
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38715/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790465400694755-388-0/testchainca.cert with env {}
W20260926 23:30:49.246769 5696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:49.247208 5696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:49.247356 5696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:49.255899 5696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:30:49.256064 5696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:49.256131 5696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:49.256215 5696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:30:49.256278 5696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 23:30:49.256371 5696 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 23:30:49.256424 5696 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38715/jwks.json
I20260926 23:30:49.268101 5696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45241
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38715/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790465400694755-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:49.270200 5696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:49.272171 5696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:49.286743 5702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:49.287024 5704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:49.287389 5701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:49.288723 5696 server_base.cc:1034] running on GCE node
I20260926 23:30:49.289743 5696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:49.291774 5696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:49.293032 5696 hybrid_clock.cc:648] HybridClock initialized: now 1790465449292975 us; error 52 us; skew 500 ppm
I20260926 23:30:49.293563 5696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:30:49.304843 5696 webserver.cc:460] Webserver started at http://127.0.97.62:33559/ using document root <none> and password file <none>
I20260926 23:30:49.305876 5696 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:49.306022 5696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:49.306411 5696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:49.310571 5696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "7cf45df4e1a5449da17a1c013b576cc9"
format_stamp: "Formatted at 2026-09-26 23:30:49 on dist-test-slave-gwfx"
server_key: "8edf0f9ef3a2cf9fb2b384c30bd981ae"
server_key_iv: "43a800311962b859216696b7434af03e"
server_key_version: "encryptionkey@0"
I20260926 23:30:49.311774 5696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "7cf45df4e1a5449da17a1c013b576cc9"
format_stamp: "Formatted at 2026-09-26 23:30:49 on dist-test-slave-gwfx"
server_key: "8edf0f9ef3a2cf9fb2b384c30bd981ae"
server_key_iv: "43a800311962b859216696b7434af03e"
server_key_version: "encryptionkey@0"
I20260926 23:30:49.319216 5696 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260926 23:30:49.324054 5710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:49.325994 5696 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 23:30:49.326273 5696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "7cf45df4e1a5449da17a1c013b576cc9"
format_stamp: "Formatted at 2026-09-26 23:30:49 on dist-test-slave-gwfx"
server_key: "8edf0f9ef3a2cf9fb2b384c30bd981ae"
server_key_iv: "43a800311962b859216696b7434af03e"
server_key_version: "encryptionkey@0"
I20260926 23:30:49.326551 5696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:49.354599 5696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:49.355785 5696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:49.356133 5696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:49.376240 5696 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45241
I20260926 23:30:49.376264 5761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45241 every 8 connection(s)
I20260926 23:30:49.377861 5696 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:30:49.382488 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 5696
I20260926 23:30:49.382594 5762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:49.382737 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:30:49.383581 388 external_mini_cluster.cc:1428] Setting key a4f525b4d988e5b59899aee921f3ab84
I20260926 23:30:49.398411 5762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cf45df4e1a5449da17a1c013b576cc9: Bootstrap starting.
I20260926 23:30:49.398819 5765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:49.385780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45218 (local address 127.0.97.62:45241)
0926 23:30:49.386973 (+ 1193us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:49.386986 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:49.387038 (+ 52us) server_negotiation.cc:407] Connection header received
0926 23:30:49.388638 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:49.388663 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:49.389242 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:49.389686 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:49.390344 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:49.392345 (+ 2001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:49.393758 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:49.394528 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:49.394738 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:49.395987 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:49.396001 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:49.396025 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:30:49.396713 (+ 688us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:49.396815 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:30:49.396839 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:49.396959 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:49.397179 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 23:30:49.397570 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":376,"threads_started":1}
I20260926 23:30:49.402876 5762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cf45df4e1a5449da17a1c013b576cc9: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:49.403630 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:30:49.404253 5762 log.cc:824] T 00000000000000000000000000000000 P 7cf45df4e1a5449da17a1c013b576cc9: Log is configured to *not* fsync() on all Append() calls
I20260926 23:30:49.408114 5762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cf45df4e1a5449da17a1c013b576cc9: No bootstrap required, opened a new log
I20260926 23:30:49.416338 5762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cf45df4e1a5449da17a1c013b576cc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf45df4e1a5449da17a1c013b576cc9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45241 } }
I20260926 23:30:49.416816 5762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cf45df4e1a5449da17a1c013b576cc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:49.416930 5762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cf45df4e1a5449da17a1c013b576cc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cf45df4e1a5449da17a1c013b576cc9, State: Initialized, Role: FOLLOWER
I20260926 23:30:49.417701 5762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cf45df4e1a5449da17a1c013b576cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf45df4e1a5449da17a1c013b576cc9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45241 } }
I20260926 23:30:49.417958 5762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cf45df4e1a5449da17a1c013b576cc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:30:49.418061 5762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cf45df4e1a5449da17a1c013b576cc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:30:49.418201 5762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cf45df4e1a5449da17a1c013b576cc9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:49.420580 5762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cf45df4e1a5449da17a1c013b576cc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf45df4e1a5449da17a1c013b576cc9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45241 } }
I20260926 23:30:49.421170 5762 leader_election.cc:302] T 00000000000000000000000000000000 P 7cf45df4e1a5449da17a1c013b576cc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cf45df4e1a5449da17a1c013b576cc9; no voters:
I20260926 23:30:49.421802 5762 leader_election.cc:288] T 00000000000000000000000000000000 P 7cf45df4e1a5449da17a1c013b576cc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:30:49.421975 5767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cf45df4e1a5449da17a1c013b576cc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:30:49.422721 5767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cf45df4e1a5449da17a1c013b576cc9 [term 1 LEADER]: Becoming Leader. State: Replica: 7cf45df4e1a5449da17a1c013b576cc9, State: Running, Role: LEADER
I20260926 23:30:49.423375 5767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cf45df4e1a5449da17a1c013b576cc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf45df4e1a5449da17a1c013b576cc9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45241 } }
I20260926 23:30:49.424410 5762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cf45df4e1a5449da17a1c013b576cc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:30:49.427069 5768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cf45df4e1a5449da17a1c013b576cc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cf45df4e1a5449da17a1c013b576cc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf45df4e1a5449da17a1c013b576cc9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45241 } } }
I20260926 23:30:49.426952 5769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cf45df4e1a5449da17a1c013b576cc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cf45df4e1a5449da17a1c013b576cc9. Latest consensus state: current_term: 1 leader_uuid: "7cf45df4e1a5449da17a1c013b576cc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf45df4e1a5449da17a1c013b576cc9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45241 } } }
I20260926 23:30:49.427481 5769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cf45df4e1a5449da17a1c013b576cc9 [sys.catalog]: This master's current role is: LEADER
I20260926 23:30:49.427479 5768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cf45df4e1a5449da17a1c013b576cc9 [sys.catalog]: This master's current role is: LEADER
I20260926 23:30:49.431334 5775 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:30:49.436911 5775 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:30:49.448187 5775 catalog_manager.cc:1347] Generated new cluster ID: 703797041abd4cefb649dddcb296d195
I20260926 23:30:49.448342 5775 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:30:49.471022 5775 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:30:49.472643 5775 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:30:49.479924 5775 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cf45df4e1a5449da17a1c013b576cc9: Generated new TSK 0
I20260926 23:30:49.480976 5775 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 23:30:49.536463 5792 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 23:30:49.536638 5792 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 23:30:49.537703 5765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:49.508730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45226 (local address 127.0.97.62:45241)
0926 23:30:49.508989 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:49.508996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:49.509439 (+ 443us) server_negotiation.cc:407] Connection header received
0926 23:30:49.509707 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:49.509711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:49.509829 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:49.510008 (+ 179us) server_negotiation.cc:226] Negotiated authn=JWT
0926 23:30:49.510588 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:49.511535 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:49.513663 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:49.514191 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:49.514304 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:49.515017 (+ 713us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 23:30:49.537058 (+ 22041us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38715/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 23:30:49.537430 (+ 372us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45226: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38715/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":95}
W20260926 23:30:49.537847 5765 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45226: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38715/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 23:30:49.538225 5791 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:30:49.508519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45241 (local address 127.0.0.1:45226)
0926 23:30:49.509277 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:49.509300 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:49.509491 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:49.510033 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:49.510039 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:49.510052 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0926 23:30:49.510379 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:49.510391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:49.511688 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:49.511694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:49.513495 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:49.513511 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:49.514318 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:49.514378 (+ 60us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:49.514590 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:49.514679 (+ 89us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 23:30:49.537381 (+ 22702us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38715/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 23:30:49.537741 (+ 360us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:45241: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38715/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":404,"threads_started":1}
I20260926 23:30:49.539580 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 5696
I20260926 23:30:49.541247 5757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:30:49.639405 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 5696
2026-09-26T23:30:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (885 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 23:30:49.695959 388 mini_oidc.cc:152] Starting JWKS server
I20260926 23:30:49.698065 388 webserver.cc:460] Webserver started at https://127.0.0.1:40371/ using document root <none> and password file <none>
I20260926 23:30:49.698359 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 23:30:49.699131 388 webserver.cc:460] Webserver started at http://127.0.0.1:41339/ using document root <none> and password file <none>
2026-09-26T23:30:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:30:49Z Disabled control of system clock
I20260926 23:30:49.728991 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45053
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45053
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40371/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 23:30:50.133311 5799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:50.133736 5799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:50.133870 5799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:50.142146 5799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:30:50.142316 5799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:50.142356 5799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:50.142421 5799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:30:50.142473 5799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 23:30:50.142570 5799 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 23:30:50.142620 5799 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40371/jwks.json
I20260926 23:30:50.153126 5799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45053
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40371/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:50.154925 5799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:50.157940 5799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:50.173446 5807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:50.173655 5804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:50.173628 5805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:50.175087 5799 server_base.cc:1034] running on GCE node
I20260926 23:30:50.176168 5799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:50.178032 5799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:50.179319 5799 hybrid_clock.cc:648] HybridClock initialized: now 1790465450179234 us; error 64 us; skew 500 ppm
I20260926 23:30:50.179828 5799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:30:50.183130 5799 webserver.cc:460] Webserver started at http://127.0.97.62:36361/ using document root <none> and password file <none>
I20260926 23:30:50.184046 5799 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:50.184173 5799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:50.184580 5799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:50.188295 5799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "40ee36cd529548f5bed4843f4f5df009"
format_stamp: "Formatted at 2026-09-26 23:30:50 on dist-test-slave-gwfx"
server_key: "9987f19fcd7e0cf25bbbd2b8ddf8edb3"
server_key_iv: "c1d50846494865dc3285aadd83bdab43"
server_key_version: "encryptionkey@0"
I20260926 23:30:50.189352 5799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "40ee36cd529548f5bed4843f4f5df009"
format_stamp: "Formatted at 2026-09-26 23:30:50 on dist-test-slave-gwfx"
server_key: "9987f19fcd7e0cf25bbbd2b8ddf8edb3"
server_key_iv: "c1d50846494865dc3285aadd83bdab43"
server_key_version: "encryptionkey@0"
I20260926 23:30:50.196386 5799 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260926 23:30:50.201560 5813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:50.203763 5799 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260926 23:30:50.204042 5799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "40ee36cd529548f5bed4843f4f5df009"
format_stamp: "Formatted at 2026-09-26 23:30:50 on dist-test-slave-gwfx"
server_key: "9987f19fcd7e0cf25bbbd2b8ddf8edb3"
server_key_iv: "c1d50846494865dc3285aadd83bdab43"
server_key_version: "encryptionkey@0"
I20260926 23:30:50.204324 5799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:50.238307 5799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:50.239504 5799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:50.239820 5799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:50.261775 5799 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45053
I20260926 23:30:50.261822 5864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45053 every 8 connection(s)
I20260926 23:30:50.263628 5799 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:30:50.265771 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 5799
I20260926 23:30:50.266005 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:30:50.266868 388 external_mini_cluster.cc:1428] Setting key b3addbb5e75426d87191f892f7d2c799
I20260926 23:30:50.269212 5865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:50.285017 5865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40ee36cd529548f5bed4843f4f5df009: Bootstrap starting.
I20260926 23:30:50.285333 5868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:50.269376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37000 (local address 127.0.97.62:45053)
0926 23:30:50.270707 (+ 1331us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:50.270727 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:50.270792 (+ 65us) server_negotiation.cc:407] Connection header received
0926 23:30:50.272615 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:50.272644 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:50.273366 (+ 722us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:50.273849 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:50.274798 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:50.276628 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:50.278848 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:50.279875 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:50.280083 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:50.281679 (+ 1596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:50.281690 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:50.281718 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:30:50.282639 (+ 921us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:50.282862 (+ 223us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:30:50.282904 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:50.283040 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:50.283370 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 23:30:50.283876 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":444,"threads_started":1}
I20260926 23:30:50.289081 5865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40ee36cd529548f5bed4843f4f5df009: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:50.290606 5865 log.cc:824] T 00000000000000000000000000000000 P 40ee36cd529548f5bed4843f4f5df009: Log is configured to *not* fsync() on all Append() calls
I20260926 23:30:50.290957 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:30:50.295286 5865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40ee36cd529548f5bed4843f4f5df009: No bootstrap required, opened a new log
I20260926 23:30:50.302963 5865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40ee36cd529548f5bed4843f4f5df009 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ee36cd529548f5bed4843f4f5df009" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45053 } }
I20260926 23:30:50.303385 5865 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40ee36cd529548f5bed4843f4f5df009 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:50.303449 5865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40ee36cd529548f5bed4843f4f5df009 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40ee36cd529548f5bed4843f4f5df009, State: Initialized, Role: FOLLOWER
I20260926 23:30:50.304111 5865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40ee36cd529548f5bed4843f4f5df009 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ee36cd529548f5bed4843f4f5df009" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45053 } }
I20260926 23:30:50.304386 5865 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40ee36cd529548f5bed4843f4f5df009 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:30:50.304536 5865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40ee36cd529548f5bed4843f4f5df009 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:30:50.304710 5865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40ee36cd529548f5bed4843f4f5df009 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:50.307052 5865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40ee36cd529548f5bed4843f4f5df009 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ee36cd529548f5bed4843f4f5df009" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45053 } }
I20260926 23:30:50.307657 5865 leader_election.cc:302] T 00000000000000000000000000000000 P 40ee36cd529548f5bed4843f4f5df009 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40ee36cd529548f5bed4843f4f5df009; no voters:
I20260926 23:30:50.308362 5865 leader_election.cc:288] T 00000000000000000000000000000000 P 40ee36cd529548f5bed4843f4f5df009 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:30:50.308610 5870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40ee36cd529548f5bed4843f4f5df009 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:30:50.309777 5870 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40ee36cd529548f5bed4843f4f5df009 [term 1 LEADER]: Becoming Leader. State: Replica: 40ee36cd529548f5bed4843f4f5df009, State: Running, Role: LEADER
I20260926 23:30:50.311007 5865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40ee36cd529548f5bed4843f4f5df009 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:30:50.310902 5870 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40ee36cd529548f5bed4843f4f5df009 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ee36cd529548f5bed4843f4f5df009" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45053 } }
I20260926 23:30:50.315441 5871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40ee36cd529548f5bed4843f4f5df009 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40ee36cd529548f5bed4843f4f5df009" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ee36cd529548f5bed4843f4f5df009" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45053 } } }
I20260926 23:30:50.315548 5872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40ee36cd529548f5bed4843f4f5df009 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40ee36cd529548f5bed4843f4f5df009. Latest consensus state: current_term: 1 leader_uuid: "40ee36cd529548f5bed4843f4f5df009" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ee36cd529548f5bed4843f4f5df009" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45053 } } }
I20260926 23:30:50.316931 5871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40ee36cd529548f5bed4843f4f5df009 [sys.catalog]: This master's current role is: LEADER
I20260926 23:30:50.317121 5872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40ee36cd529548f5bed4843f4f5df009 [sys.catalog]: This master's current role is: LEADER
I20260926 23:30:50.318986 5880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:30:50.326336 5880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:30:50.337782 5880 catalog_manager.cc:1347] Generated new cluster ID: aaf70f819a524249bf8422a239597ff5
I20260926 23:30:50.337934 5880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:30:50.347751 5880 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:30:50.349417 5880 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:30:50.365059 5880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40ee36cd529548f5bed4843f4f5df009: Generated new TSK 0
I20260926 23:30:50.366344 5880 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:30:50.400426 5868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:50.391484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37012 (local address 127.0.97.62:45053)
0926 23:30:50.391760 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:50.391767 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:50.391968 (+ 201us) server_negotiation.cc:407] Connection header received
0926 23:30:50.392234 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:50.392241 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:50.392362 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:50.392622 (+ 260us) server_negotiation.cc:226] Negotiated authn=JWT
0926 23:30:50.393257 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:50.394479 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:50.396183 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:50.396803 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:50.396944 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:50.397563 (+ 619us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 23:30:50.399751 (+ 2188us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40371/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 23:30:50.400106 (+ 355us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37012: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40371/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":113}
W20260926 23:30:50.400764 5894 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:30:50.391235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45053 (local address 127.0.0.1:37012)
0926 23:30:50.391856 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:50.391874 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:50.392011 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:50.392722 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:50.392726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:50.392738 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0926 23:30:50.393033 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:50.393042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:50.394784 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:50.394789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:50.395940 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:50.395956 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:50.396963 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:50.397058 (+ 95us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:50.397205 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:50.397247 (+ 42us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 23:30:50.399992 (+ 2745us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40371/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 23:30:50.400333 (+ 341us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:45053: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40371/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":319,"threads_started":1}
W20260926 23:30:50.400867 5868 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37012: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40371/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260926 23:30:50.401787 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 5799
I20260926 23:30:50.403288 5860 generic_service.cc:196] Checking for leaks (request via RPC)
E20260926 23:30:50.413491 5895 webserver.cc:533] Webserver: sslize failed:
E20260926 23:30:50.413658 5895 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 23:30:50.501423 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 5799
2026-09-26T23:30:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (859 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T23:30:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:30:50Z Disabled control of system clock
I20260926 23:30:50.584726 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44011
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44011
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790465400694755-388-0/testchainca.cert with env {}
W20260926 23:30:50.960729 5900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:50.961066 5900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:50.961226 5900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:50.969175 5900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:30:50.969332 5900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:50.969373 5900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:50.969424 5900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:30:50.969475 5900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 23:30:50.969547 5900 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 23:30:50.969592 5900 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 23:30:50.980381 5900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44011
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790465400694755-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:50.981746 5900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:50.983111 5900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:50.996737 5908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:50.996757 5905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:50.997028 5906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:50.997778 5900 server_base.cc:1034] running on GCE node
I20260926 23:30:50.998925 5900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:51.000802 5900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:51.002331 5900 hybrid_clock.cc:648] HybridClock initialized: now 1790465451002263 us; error 56 us; skew 500 ppm
I20260926 23:30:51.002940 5900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:30:51.005965 5900 webserver.cc:460] Webserver started at http://127.0.97.62:42307/ using document root <none> and password file <none>
I20260926 23:30:51.007217 5900 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:51.007520 5900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:51.007948 5900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:51.012077 5900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "10fae078abb14af78040acba6b8705aa"
format_stamp: "Formatted at 2026-09-26 23:30:51 on dist-test-slave-gwfx"
server_key: "63b1d4e4e24f8d8d65e9f03014f4770b"
server_key_iv: "cf8593fba0280d04fe3d83aa8c929b95"
server_key_version: "encryptionkey@0"
I20260926 23:30:51.013077 5900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "10fae078abb14af78040acba6b8705aa"
format_stamp: "Formatted at 2026-09-26 23:30:51 on dist-test-slave-gwfx"
server_key: "63b1d4e4e24f8d8d65e9f03014f4770b"
server_key_iv: "cf8593fba0280d04fe3d83aa8c929b95"
server_key_version: "encryptionkey@0"
I20260926 23:30:51.020226 5900 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260926 23:30:51.025276 5914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:51.027576 5900 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 23:30:51.027818 5900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "10fae078abb14af78040acba6b8705aa"
format_stamp: "Formatted at 2026-09-26 23:30:51 on dist-test-slave-gwfx"
server_key: "63b1d4e4e24f8d8d65e9f03014f4770b"
server_key_iv: "cf8593fba0280d04fe3d83aa8c929b95"
server_key_version: "encryptionkey@0"
I20260926 23:30:51.028107 5900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:51.054014 5900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:51.055027 5900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:51.055328 5900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:51.077397 5900 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44011
I20260926 23:30:51.077461 5965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44011 every 8 connection(s)
I20260926 23:30:51.079222 5900 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:30:51.084702 5966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:51.089900 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 5900
I20260926 23:30:51.090085 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:30:51.090824 388 external_mini_cluster.cc:1428] Setting key 499bfecec865a7a74fc3da1a3ede5d21
I20260926 23:30:51.100596 5966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10fae078abb14af78040acba6b8705aa: Bootstrap starting.
I20260926 23:30:51.106526 5966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10fae078abb14af78040acba6b8705aa: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:51.108297 5966 log.cc:824] T 00000000000000000000000000000000 P 10fae078abb14af78040acba6b8705aa: Log is configured to *not* fsync() on all Append() calls
I20260926 23:30:51.111869 5969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:51.093673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45310 (local address 127.0.97.62:44011)
0926 23:30:51.094982 (+ 1309us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:51.095000 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:51.095066 (+ 66us) server_negotiation.cc:407] Connection header received
0926 23:30:51.098122 (+ 3056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:51.098174 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:51.099276 (+ 1102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:51.099775 (+ 499us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:51.100711 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:51.102406 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:51.105038 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:51.105988 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:51.106214 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:51.108015 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:51.108026 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:51.108046 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:30:51.108982 (+ 936us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:51.109205 (+ 223us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:30:51.109255 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:51.109616 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:51.109816 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 23:30:51.110277 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":427,"threads_started":1}
I20260926 23:30:51.112993 5966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10fae078abb14af78040acba6b8705aa: No bootstrap required, opened a new log
I20260926 23:30:51.117364 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:30:51.123200 5966 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10fae078abb14af78040acba6b8705aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10fae078abb14af78040acba6b8705aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44011 } }
I20260926 23:30:51.123875 5966 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10fae078abb14af78040acba6b8705aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:51.124015 5966 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10fae078abb14af78040acba6b8705aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10fae078abb14af78040acba6b8705aa, State: Initialized, Role: FOLLOWER
I20260926 23:30:51.124899 5966 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10fae078abb14af78040acba6b8705aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10fae078abb14af78040acba6b8705aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44011 } }
I20260926 23:30:51.125223 5966 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10fae078abb14af78040acba6b8705aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:30:51.125380 5966 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10fae078abb14af78040acba6b8705aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:30:51.125566 5966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10fae078abb14af78040acba6b8705aa [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:51.128393 5966 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10fae078abb14af78040acba6b8705aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10fae078abb14af78040acba6b8705aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44011 } }
I20260926 23:30:51.129138 5966 leader_election.cc:302] T 00000000000000000000000000000000 P 10fae078abb14af78040acba6b8705aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10fae078abb14af78040acba6b8705aa; no voters:
I20260926 23:30:51.130076 5966 leader_election.cc:288] T 00000000000000000000000000000000 P 10fae078abb14af78040acba6b8705aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:30:51.130297 5971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10fae078abb14af78040acba6b8705aa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:30:51.131726 5971 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10fae078abb14af78040acba6b8705aa [term 1 LEADER]: Becoming Leader. State: Replica: 10fae078abb14af78040acba6b8705aa, State: Running, Role: LEADER
I20260926 23:30:51.132675 5971 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10fae078abb14af78040acba6b8705aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10fae078abb14af78040acba6b8705aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44011 } }
I20260926 23:30:51.133395 5966 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10fae078abb14af78040acba6b8705aa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:30:51.135846 5972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10fae078abb14af78040acba6b8705aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10fae078abb14af78040acba6b8705aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10fae078abb14af78040acba6b8705aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44011 } } }
I20260926 23:30:51.136389 5972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10fae078abb14af78040acba6b8705aa [sys.catalog]: This master's current role is: LEADER
I20260926 23:30:51.135841 5973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10fae078abb14af78040acba6b8705aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10fae078abb14af78040acba6b8705aa. Latest consensus state: current_term: 1 leader_uuid: "10fae078abb14af78040acba6b8705aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10fae078abb14af78040acba6b8705aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44011 } } }
I20260926 23:30:51.137102 5973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10fae078abb14af78040acba6b8705aa [sys.catalog]: This master's current role is: LEADER
I20260926 23:30:51.142122 5977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:30:51.147882 5977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:30:51.159912 5977 catalog_manager.cc:1347] Generated new cluster ID: e6b9b038b70e4711b807f5c845d6a2da
I20260926 23:30:51.160089 5977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:30:51.185103 5977 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:30:51.186791 5977 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:30:51.200102 5977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10fae078abb14af78040acba6b8705aa: Generated new TSK 0
I20260926 23:30:51.201359 5977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:30:51.308086 5969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:51.228638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45326 (local address 127.0.97.62:44011)
0926 23:30:51.228884 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:51.228891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:51.229370 (+ 479us) server_negotiation.cc:407] Connection header received
0926 23:30:51.229622 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:51.229627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:51.229739 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:51.229966 (+ 227us) server_negotiation.cc:226] Negotiated authn=JWT
0926 23:30:51.230590 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:51.231869 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:51.233570 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:51.234372 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:51.234554 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:51.235239 (+ 685us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 23:30:51.307154 (+ 71915us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 23:30:51.307693 (+ 539us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45326: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 23:30:51.308292 5995 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:30:51.228388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44011 (local address 127.0.0.1:45326)
0926 23:30:51.229239 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:51.229258 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:51.229434 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:51.230026 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:51.230030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:51.230043 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0926 23:30:51.230398 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:51.230409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:51.232071 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:51.232076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:51.233403 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:51.233418 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:51.234548 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:51.234621 (+ 73us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:51.234842 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:51.234906 (+ 64us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 23:30:51.307538 (+ 72632us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 23:30:51.307885 (+ 347us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:44011: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":513,"threads_started":1}
W20260926 23:30:51.308388 5969 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45326: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260926 23:30:51.310470 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 5900
I20260926 23:30:51.311889 5961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:30:51.420233 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 5900
2026-09-26T23:30:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (900 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (499 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (516 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:30:52 dist-test-slave-gwfx krb5kdc[6010](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:30:52 dist-test-slave-gwfx krb5kdc[6010](info): set up 2 sockets
Sep 26 23:30:52 dist-test-slave-gwfx krb5kdc[6010](info): commencing operation
krb5kdc: starting...
W20260926 23:30:53.471808 388 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:30:53 dist-test-slave-gwfx krb5kdc[6010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:30:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:30:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:30:53.720304 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35083
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:30:54.140633 6023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:54.141054 6023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:54.141187 6023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:54.149783 6023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:30:54.149951 6023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:54.150002 6023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:54.150084 6023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:30:54.150148 6023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:30:54.161854 6023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:54.163712 6023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:54.165416 6023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:54.180841 6028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:54.180820 6029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:54.181097 6031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:54.181859 6023 server_base.cc:1034] running on GCE node
I20260926 23:30:54.183030 6023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:54.184996 6023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:54.186251 6023 hybrid_clock.cc:648] HybridClock initialized: now 1790465454186154 us; error 98 us; skew 500 ppm
I20260926 23:30:54.186853 6023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:54 dist-test-slave-gwfx krb5kdc[6010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:54.191120 6023 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:30:54.193784 6023 webserver.cc:460] Webserver started at http://127.0.97.62:37757/ using document root <none> and password file <none>
I20260926 23:30:54.194964 6023 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:54.195143 6023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:54.195591 6023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:54.199579 6023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "f411b8abbc5c47e1a8cdb429b1fbd7d1"
format_stamp: "Formatted at 2026-09-26 23:30:54 on dist-test-slave-gwfx"
server_key: "ebf40837caf3915914db951157741199"
server_key_iv: "1b6e14d94589a7b91fb42cb359cfb800"
server_key_version: "encryptionkey@0"
I20260926 23:30:54.200546 6023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "f411b8abbc5c47e1a8cdb429b1fbd7d1"
format_stamp: "Formatted at 2026-09-26 23:30:54 on dist-test-slave-gwfx"
server_key: "ebf40837caf3915914db951157741199"
server_key_iv: "1b6e14d94589a7b91fb42cb359cfb800"
server_key_version: "encryptionkey@0"
I20260926 23:30:54.207278 6023 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 23:30:54.212028 6038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:54.214097 6023 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 23:30:54.214340 6023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "f411b8abbc5c47e1a8cdb429b1fbd7d1"
format_stamp: "Formatted at 2026-09-26 23:30:54 on dist-test-slave-gwfx"
server_key: "ebf40837caf3915914db951157741199"
server_key_iv: "1b6e14d94589a7b91fb42cb359cfb800"
server_key_version: "encryptionkey@0"
I20260926 23:30:54.214658 6023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:54.237236 6023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:54.243266 6023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:54.243614 6023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:54.265151 6023 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35083
I20260926 23:30:54.265173 6089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35083 every 8 connection(s)
I20260926 23:30:54.267066 6023 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:30:54.269402 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 6023
I20260926 23:30:54.269627 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:30:54.270295 388 external_mini_cluster.cc:1428] Setting key c1de221de0d9bb733ef1bf3b7d5e3bb3
I20260926 23:30:54.272675 6090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 23:30:54 dist-test-slave-gwfx krb5kdc[6010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:54.290290 6090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f411b8abbc5c47e1a8cdb429b1fbd7d1: Bootstrap starting.
I20260926 23:30:54.294778 6090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f411b8abbc5c47e1a8cdb429b1fbd7d1: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:54.296531 6090 log.cc:824] T 00000000000000000000000000000000 P f411b8abbc5c47e1a8cdb429b1fbd7d1: Log is configured to *not* fsync() on all Append() calls
I20260926 23:30:54.299827 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:54.272748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56156 (local address 127.0.97.62:35083)
0926 23:30:54.273843 (+ 1095us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:54.273862 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:54.273927 (+ 65us) server_negotiation.cc:407] Connection header received
0926 23:30:54.276411 (+ 2484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:54.276448 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:54.277099 (+ 651us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:54.277625 (+ 526us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:54.280117 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:54.281838 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:54.283805 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:54.284719 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:54.284906 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:54.290657 (+ 5751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:54.290670 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:54.290731 (+ 61us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:54.290794 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:54.294339 (+ 3545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:54.295602 (+ 1263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:54.295613 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:54.295624 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:54.295854 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:54.296347 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:54.296351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:54.296353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:54.297141 (+ 788us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:54.297579 (+ 438us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:54.298010 (+ 431us) server_negotiation.cc:299] Negotiation successful
0926 23:30:54.298520 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":429,"threads_started":1}
I20260926 23:30:54.300474 6090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f411b8abbc5c47e1a8cdb429b1fbd7d1: No bootstrap required, opened a new log
I20260926 23:30:54.310405 6090 raft_consensus.cc:348] T 00000000000000000000000000000000 P f411b8abbc5c47e1a8cdb429b1fbd7d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f411b8abbc5c47e1a8cdb429b1fbd7d1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35083 } }
I20260926 23:30:54.310994 6090 raft_consensus.cc:374] T 00000000000000000000000000000000 P f411b8abbc5c47e1a8cdb429b1fbd7d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:54.311124 6090 raft_consensus.cc:729] T 00000000000000000000000000000000 P f411b8abbc5c47e1a8cdb429b1fbd7d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f411b8abbc5c47e1a8cdb429b1fbd7d1, State: Initialized, Role: FOLLOWER
I20260926 23:30:54.311913 6090 consensus_queue.cc:260] T 00000000000000000000000000000000 P f411b8abbc5c47e1a8cdb429b1fbd7d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f411b8abbc5c47e1a8cdb429b1fbd7d1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35083 } }
I20260926 23:30:54.312191 6090 raft_consensus.cc:388] T 00000000000000000000000000000000 P f411b8abbc5c47e1a8cdb429b1fbd7d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:30:54.312343 6090 raft_consensus.cc:482] T 00000000000000000000000000000000 P f411b8abbc5c47e1a8cdb429b1fbd7d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:30:54.312525 6090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f411b8abbc5c47e1a8cdb429b1fbd7d1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:54.315203 6090 raft_consensus.cc:504] T 00000000000000000000000000000000 P f411b8abbc5c47e1a8cdb429b1fbd7d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f411b8abbc5c47e1a8cdb429b1fbd7d1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35083 } }
I20260926 23:30:54.315943 6090 leader_election.cc:302] T 00000000000000000000000000000000 P f411b8abbc5c47e1a8cdb429b1fbd7d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f411b8abbc5c47e1a8cdb429b1fbd7d1; no voters:
I20260926 23:30:54.316911 6090 leader_election.cc:288] T 00000000000000000000000000000000 P f411b8abbc5c47e1a8cdb429b1fbd7d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:30:54.317094 6095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f411b8abbc5c47e1a8cdb429b1fbd7d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:30:54.319111 6095 raft_consensus.cc:686] T 00000000000000000000000000000000 P f411b8abbc5c47e1a8cdb429b1fbd7d1 [term 1 LEADER]: Becoming Leader. State: Replica: f411b8abbc5c47e1a8cdb429b1fbd7d1, State: Running, Role: LEADER
I20260926 23:30:54.319988 6090 sys_catalog.cc:564] T 00000000000000000000000000000000 P f411b8abbc5c47e1a8cdb429b1fbd7d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:30:54.319870 6095 consensus_queue.cc:237] T 00000000000000000000000000000000 P f411b8abbc5c47e1a8cdb429b1fbd7d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f411b8abbc5c47e1a8cdb429b1fbd7d1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35083 } }
I20260926 23:30:54.324116 6097 sys_catalog.cc:455] T 00000000000000000000000000000000 P f411b8abbc5c47e1a8cdb429b1fbd7d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f411b8abbc5c47e1a8cdb429b1fbd7d1. Latest consensus state: current_term: 1 leader_uuid: "f411b8abbc5c47e1a8cdb429b1fbd7d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f411b8abbc5c47e1a8cdb429b1fbd7d1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35083 } } }
I20260926 23:30:54.324132 6096 sys_catalog.cc:455] T 00000000000000000000000000000000 P f411b8abbc5c47e1a8cdb429b1fbd7d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f411b8abbc5c47e1a8cdb429b1fbd7d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f411b8abbc5c47e1a8cdb429b1fbd7d1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35083 } } }
I20260926 23:30:54.324810 6097 sys_catalog.cc:458] T 00000000000000000000000000000000 P f411b8abbc5c47e1a8cdb429b1fbd7d1 [sys.catalog]: This master's current role is: LEADER
I20260926 23:30:54.324909 6096 sys_catalog.cc:458] T 00000000000000000000000000000000 P f411b8abbc5c47e1a8cdb429b1fbd7d1 [sys.catalog]: This master's current role is: LEADER
I20260926 23:30:54.325909 6103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:30:54.332973 6103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:30:54.345232 6103 catalog_manager.cc:1347] Generated new cluster ID: 5df34cfccb624d7ebaeeed604a693621
I20260926 23:30:54.345398 6103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:30:54.372140 6103 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:30:54.373937 6103 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:30:54.382829 6103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f411b8abbc5c47e1a8cdb429b1fbd7d1: Generated new TSK 0
I20260926 23:30:54.384277 6103 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:30:54.452841 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35083
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:30:54.871070 6116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:54.871457 6116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:54.871562 6116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:54.879813 6116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:54.879971 6116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:54.880100 6116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:30:54.891506 6116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35083
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:54.893234 6116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:54.894963 6116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:54.911108 6124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:54.911029 6122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:54.911052 6121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:54.912034 6116 server_base.cc:1034] running on GCE node
I20260926 23:30:54.912753 6116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:54.913913 6116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:54.915131 6116 hybrid_clock.cc:648] HybridClock initialized: now 1790465454915074 us; error 44 us; skew 500 ppm
I20260926 23:30:54.915571 6116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:54 dist-test-slave-gwfx krb5kdc[6010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:54.926054 6116 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:30:54.928126 6116 webserver.cc:460] Webserver started at http://127.0.97.1:35195/ using document root <none> and password file <none>
I20260926 23:30:54.929054 6116 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:54.929211 6116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:54.929569 6116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:54.933152 6116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-0/data/instance:
uuid: "8836a9efc36c44d8a15a1e33d03bd5bc"
format_stamp: "Formatted at 2026-09-26 23:30:54 on dist-test-slave-gwfx"
server_key: "fe58736df40ad4566c961982217d80dc"
server_key_iv: "78b950f16ddbf9148d6ceb6827f0c012"
server_key_version: "encryptionkey@0"
I20260926 23:30:54.934052 6116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8836a9efc36c44d8a15a1e33d03bd5bc"
format_stamp: "Formatted at 2026-09-26 23:30:54 on dist-test-slave-gwfx"
server_key: "fe58736df40ad4566c961982217d80dc"
server_key_iv: "78b950f16ddbf9148d6ceb6827f0c012"
server_key_version: "encryptionkey@0"
I20260926 23:30:54.940543 6116 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.007s
I20260926 23:30:54.944973 6131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:54.946889 6116 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:30:54.947094 6116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "8836a9efc36c44d8a15a1e33d03bd5bc"
format_stamp: "Formatted at 2026-09-26 23:30:54 on dist-test-slave-gwfx"
server_key: "fe58736df40ad4566c961982217d80dc"
server_key_iv: "78b950f16ddbf9148d6ceb6827f0c012"
server_key_version: "encryptionkey@0"
I20260926 23:30:54.947346 6116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:54.966176 6116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:54.972450 6116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:54.972805 6116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:54.974086 6116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:30:54.976023 6116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:30:54.976112 6116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:54.976233 6116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:30:54.976312 6116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:55.014667 6116 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44573
I20260926 23:30:55.014739 6243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44573 every 8 connection(s)
I20260926 23:30:55.016350 6116 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:30:55.021874 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 6116
I20260926 23:30:55.022099 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:30:55.022825 388 external_mini_cluster.cc:1428] Setting key d4725947de20fe7c46bc33a80b57aaf6
Sep 26 23:30:55 dist-test-slave-gwfx krb5kdc[6010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:55.042114 6246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:55.019742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58611 (local address 127.0.97.62:35083)
0926 23:30:55.020276 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:55.020283 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:55.022434 (+ 2151us) server_negotiation.cc:407] Connection header received
0926 23:30:55.024769 (+ 2335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:55.024775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:55.024879 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:55.024979 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:55.027057 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:55.027848 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:55.029643 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:55.030080 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:55.030201 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:55.034823 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:55.034830 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:55.034841 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:55.034894 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:55.037951 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:55.039033 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:55.039039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:55.039041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:55.039194 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:55.039747 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:55.039751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:55.039753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:55.040112 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:55.040296 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:55.041274 (+ 978us) server_negotiation.cc:299] Negotiation successful
0926 23:30:55.041579 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":232,"threads_started":1}
I20260926 23:30:55.043815 6247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:55.020483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35083 (local address 127.0.97.1:58611)
0926 23:30:55.022218 (+ 1735us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:55.022280 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:55.024448 (+ 2168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:55.025300 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:55.025311 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:55.025998 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:55.026846 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:55.026873 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:55.028028 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:55.028033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:55.029469 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:55.029483 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:55.030358 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:55.030367 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:55.030631 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:55.031469 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:55.031497 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:55.034392 (+ 2895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:55.038188 (+ 3796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:55.038199 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:55.038228 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:55.038864 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:55.039331 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:55.039337 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:55.039341 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:55.039592 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:55.040282 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:55.040293 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:55.040776 (+ 483us) client_negotiation.cc:769] Sending connection context
0926 23:30:55.041102 (+ 326us) client_negotiation.cc:240] Negotiation successful
0926 23:30:55.041851 (+ 749us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":348,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 23:30:55.046419 6244 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35083
I20260926 23:30:55.046882 6244 heartbeater.cc:464] Registering TS with master...
I20260926 23:30:55.048032 6244 heartbeater.cc:511] Master 127.0.97.62:35083 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 23:30:55.051374 6055 ts_manager.cc:194] Registered new tserver with Master: 8836a9efc36c44d8a15a1e33d03bd5bc (127.0.97.1:44573)
I20260926 23:30:55.053902 6055 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58611
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:30:55.079120 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35083
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:30:55.496620 6250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:55.497023 6250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:55.497129 6250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:55.505306 6250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:55.505481 6250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:55.505671 6250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:30:55.517249 6250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35083
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:55.519026 6250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:55.520749 6250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:55.537488 6255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:55.537582 6256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:55.537707 6258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:55.538442 6250 server_base.cc:1034] running on GCE node
I20260926 23:30:55.539341 6250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:55.540580 6250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:55.541824 6250 hybrid_clock.cc:648] HybridClock initialized: now 1790465455541754 us; error 65 us; skew 500 ppm
I20260926 23:30:55.542234 6250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:55 dist-test-slave-gwfx krb5kdc[6010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:55.546397 6250 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:30:55.548398 6250 webserver.cc:460] Webserver started at http://127.0.97.2:41331/ using document root <none> and password file <none>
I20260926 23:30:55.549329 6250 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:55.549468 6250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:55.549803 6250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:55.553601 6250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-1/data/instance:
uuid: "acdf607a587845699256a33f80180389"
format_stamp: "Formatted at 2026-09-26 23:30:55 on dist-test-slave-gwfx"
server_key: "51ad15a4ddcd131fbd64a38a3bb1bb3a"
server_key_iv: "60af323f649f78242606ed14ab318aa0"
server_key_version: "encryptionkey@0"
I20260926 23:30:55.554525 6250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-1/wal/instance:
uuid: "acdf607a587845699256a33f80180389"
format_stamp: "Formatted at 2026-09-26 23:30:55 on dist-test-slave-gwfx"
server_key: "51ad15a4ddcd131fbd64a38a3bb1bb3a"
server_key_iv: "60af323f649f78242606ed14ab318aa0"
server_key_version: "encryptionkey@0"
I20260926 23:30:55.561205 6250 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260926 23:30:55.565910 6265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:55.567950 6250 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260926 23:30:55.568171 6250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "acdf607a587845699256a33f80180389"
format_stamp: "Formatted at 2026-09-26 23:30:55 on dist-test-slave-gwfx"
server_key: "51ad15a4ddcd131fbd64a38a3bb1bb3a"
server_key_iv: "60af323f649f78242606ed14ab318aa0"
server_key_version: "encryptionkey@0"
I20260926 23:30:55.568496 6250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:55.582708 6250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:55.587893 6250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:55.588200 6250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:55.589389 6250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:30:55.591173 6250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:30:55.591264 6250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:55.591367 6250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:30:55.591410 6250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:55.627130 6250 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46681
I20260926 23:30:55.627213 6377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46681 every 8 connection(s)
I20260926 23:30:55.628800 6250 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:30:55.638229 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 6250
I20260926 23:30:55.638461 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:30:55.639055 388 external_mini_cluster.cc:1428] Setting key 7b873f8ef7e73935974e89a0119b9110
Sep 26 23:30:55 dist-test-slave-gwfx krb5kdc[6010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:55.656805 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:55.632178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48887 (local address 127.0.97.62:35083)
0926 23:30:55.632713 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:55.632721 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:55.634032 (+ 1311us) server_negotiation.cc:407] Connection header received
0926 23:30:55.636285 (+ 2253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:55.636291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:55.636383 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:55.636501 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:55.638901 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:55.639845 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:55.642330 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:55.642945 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:55.643072 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:55.648661 (+ 5589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:55.648669 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:55.648673 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:55.648732 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:55.652163 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:55.653553 (+ 1390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:55.653559 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:55.653561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:55.653762 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:55.654351 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:55.654355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:55.654357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:55.654762 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:55.654937 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:55.655914 (+ 977us) server_negotiation.cc:299] Negotiation successful
0926 23:30:55.656221 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":257,"threads_started":1}
I20260926 23:30:55.658320 6381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:55.632823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35083 (local address 127.0.97.2:48887)
0926 23:30:55.633807 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:55.633873 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:55.635875 (+ 2002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:55.636805 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:55.636815 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:55.637526 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:55.638574 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:55.638611 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:55.640091 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:55.640097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:55.642102 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:55.642136 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:55.643266 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:55.643275 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:55.643677 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:55.644679 (+ 1002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:55.644714 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:55.648184 (+ 3470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:55.652541 (+ 4357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:55.652554 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:55.652576 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:55.653312 (+ 736us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:55.653935 (+ 623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:55.653940 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:55.653942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:55.654201 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:55.654921 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:55.654932 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:55.655446 (+ 514us) client_negotiation.cc:769] Sending connection context
0926 23:30:55.655760 (+ 314us) client_negotiation.cc:240] Negotiation successful
0926 23:30:55.656655 (+ 895us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":460,"threads_started":1}
I20260926 23:30:55.660555 6378 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35083
I20260926 23:30:55.661037 6378 heartbeater.cc:464] Registering TS with master...
I20260926 23:30:55.662160 6378 heartbeater.cc:511] Master 127.0.97.62:35083 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 23:30:55.665095 6055 ts_manager.cc:194] Registered new tserver with Master: acdf607a587845699256a33f80180389 (127.0.97.2:46681)
I20260926 23:30:55.666205 6055 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48887
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:30:55.699816 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35083
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:30:56.058478 6244 heartbeater.cc:503] Master 127.0.97.62:35083 was elected leader, sending a full tablet report...
W20260926 23:30:56.124104 6384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:56.124701 6384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:56.124892 6384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:56.134510 6384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:56.134771 6384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:56.134968 6384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:30:56.146771 6384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35083
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:56.148828 6384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:56.150897 6384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:56.169804 6392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:56.169829 6389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:56.169868 6390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:56.171969 6384 server_base.cc:1034] running on GCE node
I20260926 23:30:56.172977 6384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:56.174363 6384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:56.175679 6384 hybrid_clock.cc:648] HybridClock initialized: now 1790465456175568 us; error 114 us; skew 500 ppm
I20260926 23:30:56.176090 6384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:56 dist-test-slave-gwfx krb5kdc[6010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:56.180040 6384 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:30:56.182184 6384 webserver.cc:460] Webserver started at http://127.0.97.3:39517/ using document root <none> and password file <none>
I20260926 23:30:56.183151 6384 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:56.183281 6384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:56.183612 6384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:56.187191 6384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-2/data/instance:
uuid: "59cdc2b4aba84b3e949feeb46eb7b40f"
format_stamp: "Formatted at 2026-09-26 23:30:56 on dist-test-slave-gwfx"
server_key: "54dafac673c547975b11995d827a7394"
server_key_iv: "7ace8be1b3de7a00fd7fe9c3f872fc70"
server_key_version: "encryptionkey@0"
I20260926 23:30:56.188074 6384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-2/wal/instance:
uuid: "59cdc2b4aba84b3e949feeb46eb7b40f"
format_stamp: "Formatted at 2026-09-26 23:30:56 on dist-test-slave-gwfx"
server_key: "54dafac673c547975b11995d827a7394"
server_key_iv: "7ace8be1b3de7a00fd7fe9c3f872fc70"
server_key_version: "encryptionkey@0"
I20260926 23:30:56.195046 6384 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260926 23:30:56.200167 6399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:56.202828 6384 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260926 23:30:56.203081 6384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "59cdc2b4aba84b3e949feeb46eb7b40f"
format_stamp: "Formatted at 2026-09-26 23:30:56 on dist-test-slave-gwfx"
server_key: "54dafac673c547975b11995d827a7394"
server_key_iv: "7ace8be1b3de7a00fd7fe9c3f872fc70"
server_key_version: "encryptionkey@0"
I20260926 23:30:56.203318 6384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:56.218858 6384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:56.225230 6384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:56.225571 6384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:56.226923 6384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:30:56.228675 6384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:30:56.228766 6384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:56.228883 6384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:30:56.228935 6384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:56.268163 6384 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37371
I20260926 23:30:56.268165 6511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37371 every 8 connection(s)
I20260926 23:30:56.270478 6384 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:30:56.279297 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 6384
I20260926 23:30:56.279510 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:30:56.279992 388 external_mini_cluster.cc:1428] Setting key 7ef0d0ec59ef6dbd713bb377a85059be
Sep 26 23:30:56 dist-test-slave-gwfx krb5kdc[6010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:30:56.297777 6514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:56.273920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33085 (local address 127.0.97.62:35083)
0926 23:30:56.274503 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:56.274508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:56.276138 (+ 1630us) server_negotiation.cc:407] Connection header received
0926 23:30:56.278276 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:56.278284 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:56.278384 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:56.278487 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:56.280401 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:56.281533 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:56.284242 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:56.284893 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:56.285036 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:56.289510 (+ 4474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:56.289516 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:56.289520 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:56.289568 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:56.293137 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:56.294592 (+ 1455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:56.294597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:56.294600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:56.294974 (+ 374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:56.295521 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:56.295525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:56.295527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:56.295942 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:56.296123 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:56.297071 (+ 948us) server_negotiation.cc:299] Negotiation successful
0926 23:30:56.297357 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":289,"threads_started":1}
I20260926 23:30:56.299045 6515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:56.274645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35083 (local address 127.0.97.3:33085)
0926 23:30:56.275741 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:56.275910 (+ 169us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:56.277909 (+ 1999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:56.278853 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:56.278864 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:56.279450 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:56.280197 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:56.280219 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:56.281811 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:56.281819 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:56.283979 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:56.284005 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:56.285222 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:56.285229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:56.285535 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:56.286392 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:56.286421 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:56.289100 (+ 2679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:56.293551 (+ 4451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:56.293562 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:56.293591 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:56.294410 (+ 819us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:56.295130 (+ 720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:56.295136 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:56.295141 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:56.295375 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:56.296093 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:56.296104 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:56.296597 (+ 493us) client_negotiation.cc:769] Sending connection context
0926 23:30:56.296939 (+ 342us) client_negotiation.cc:240] Negotiation successful
0926 23:30:56.297540 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"thread_start_us":438,"threads_started":1}
I20260926 23:30:56.301456 6512 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35083
I20260926 23:30:56.301846 6512 heartbeater.cc:464] Registering TS with master...
I20260926 23:30:56.302798 6512 heartbeater.cc:511] Master 127.0.97.62:35083 requested a full tablet report, sending...
I20260926 23:30:56.304804 6055 ts_manager.cc:194] Registered new tserver with Master: 59cdc2b4aba84b3e949feeb46eb7b40f (127.0.97.3:37371)
I20260926 23:30:56.305874 6055 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33085
I20260926 23:30:56.313211 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/test-admin.keytab.
Sep 26 23:30:56 dist-test-slave-gwfx krb5kdc[6010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:56.351089 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 23:30:56.358060 6520 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:30:56.354894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46446 (local address 127.0.97.1:44573)
0926 23:30:56.355798 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:56.355806 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:56.355841 (+ 35us) server_negotiation.cc:407] Connection header received
0926 23:30:56.355918 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:56.355925 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:56.356189 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:56.356353 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:56.357686 (+ 1333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46446: BlockingRecv error: recv got EOF from 127.0.0.1:46446 (error 108)
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":442,"threads_started":1}
W20260926 23:30:56.358731 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/krb5cc' of type FILE
Sep 26 23:30:56 dist-test-slave-gwfx krb5kdc[6010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:56.361219 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 23:30:56 dist-test-slave-gwfx krb5kdc[6010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:56.363039 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 23:30:56 dist-test-slave-gwfx krb5kdc[6010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:30:56.382578 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:56.365184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46460 (local address 127.0.97.1:44573)
0926 23:30:56.365457 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:56.365464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:56.365802 (+ 338us) server_negotiation.cc:407] Connection header received
0926 23:30:56.366013 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:56.366019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:56.366101 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:56.366206 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:56.367445 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:56.368560 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:56.370106 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:56.370953 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:56.371082 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:56.375536 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:56.375554 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:56.375576 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:56.375626 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:56.378947 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:56.379912 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:56.379922 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:56.379924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:56.380096 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:56.380543 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:56.380548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:56.380552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:56.381087 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:56.381438 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:56.381822 (+ 384us) server_negotiation.cc:299] Negotiation successful
0926 23:30:56.382170 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260926 23:30:56.389796 6520 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:30:56.387663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46472 (local address 127.0.97.1:44573)
0926 23:30:56.387898 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:56.387905 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:56.388156 (+ 251us) server_negotiation.cc:407] Connection header received
0926 23:30:56.388287 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:56.388292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:56.388436 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:56.388532 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:56.389519 (+ 987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46472: BlockingRecv error: recv got EOF from 127.0.0.1:46472 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260926 23:30:56.390107 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/krb5cc' of type FILE
Sep 26 23:30:56 dist-test-slave-gwfx krb5kdc[6010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:56.391959 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 23:30:56 dist-test-slave-gwfx krb5kdc[6010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:56.393775 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 23:30:56 dist-test-slave-gwfx krb5kdc[6010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:30:56.413218 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:56.395876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46476 (local address 127.0.97.1:44573)
0926 23:30:56.396173 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:56.396180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:56.396353 (+ 173us) server_negotiation.cc:407] Connection header received
0926 23:30:56.396561 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:56.396565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:56.396670 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:56.396766 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:56.398469 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:56.399581 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:56.401101 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:56.401565 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:56.401693 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:56.406983 (+ 5290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:56.406992 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:56.407020 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:56.407077 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:56.410043 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:56.411103 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:56.411109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:56.411111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:56.411291 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:56.411705 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:56.411710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:56.411712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:56.412040 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:56.412334 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:56.412612 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 23:30:56.412872 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
W20260926 23:30:56.419296 6520 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:30:56.416943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46480 (local address 127.0.97.1:44573)
0926 23:30:56.417237 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:56.417244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:56.417535 (+ 291us) server_negotiation.cc:407] Connection header received
0926 23:30:56.417720 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:56.417724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:56.417814 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:56.417911 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:56.418964 (+ 1053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46480: BlockingRecv error: recv got EOF from 127.0.0.1:46480 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
W20260926 23:30:56.419556 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790465400694755-388-0/krb5kdc/krb5cc' of type FILE
Sep 26 23:30:56 dist-test-slave-gwfx krb5kdc[6010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:56.422006 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 23:30:56 dist-test-slave-gwfx krb5kdc[6010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:56.423985 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 23:30:56 dist-test-slave-gwfx krb5kdc[6010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:30:56.442395 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:56.426142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46496 (local address 127.0.97.1:44573)
0926 23:30:56.426448 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:56.426455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:56.426972 (+ 517us) server_negotiation.cc:407] Connection header received
0926 23:30:56.427270 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:56.427274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:56.427357 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:56.427463 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:56.428561 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:56.429307 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:56.430607 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:56.431089 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:56.431210 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:56.435627 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:56.435635 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:56.435640 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:56.435695 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:56.439015 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:56.440010 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:56.440015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:56.440018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:56.440205 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:56.440704 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:56.440709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:56.440712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:56.441097 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:56.441404 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:56.441760 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 23:30:56.442019 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 23:30:56.444634 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 6116
I20260926 23:30:56.459630 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:56.445488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46508 (local address 127.0.97.1:44573)
0926 23:30:56.445679 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:56.445684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:56.445948 (+ 264us) server_negotiation.cc:407] Connection header received
0926 23:30:56.446072 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:56.446075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:56.446146 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:56.446246 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:56.447329 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:56.448144 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:56.449539 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:56.450056 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:56.450199 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:56.452550 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:56.452556 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:56.452560 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:56.452609 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:56.455991 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:56.457010 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:56.457016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:56.457019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:56.457227 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:56.457867 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:56.457873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:56.457875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:56.458247 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:56.458591 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:56.459046 (+ 455us) server_negotiation.cc:299] Negotiation successful
0926 23:30:56.459302 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 23:30:56.460668 6239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:30:56.552758 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 6116
I20260926 23:30:56.568318 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 6250
Sep 26 23:30:56 dist-test-slave-gwfx krb5kdc[6010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:30:56.587312 6533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:56.569306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57506 (local address 127.0.97.2:46681)
0926 23:30:56.569981 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:56.569990 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:56.570020 (+ 30us) server_negotiation.cc:407] Connection header received
0926 23:30:56.570086 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:56.570093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:56.570324 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:56.570461 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:56.571908 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:56.573057 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:56.574878 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:56.575505 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:56.575654 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:56.580219 (+ 4565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:56.580236 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:56.580254 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:56.580298 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:56.583624 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:56.584677 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:56.584687 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:56.584689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:56.584847 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:56.585357 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:56.585362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:56.585364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:56.585785 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:56.586106 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:56.586546 (+ 440us) server_negotiation.cc:299] Negotiation successful
0926 23:30:56.586885 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":317,"threads_started":1}
I20260926 23:30:56.588879 6373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:30:56.680819 6378 heartbeater.cc:503] Master 127.0.97.62:35083 was elected leader, sending a full tablet report...
I20260926 23:30:56.682157 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 6250
W20260926 23:30:56.696985 6040 connection.cc:535] server connection from 127.0.97.2:48887 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 23:30:56.697345 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 6384
Sep 26 23:30:56 dist-test-slave-gwfx krb5kdc[6010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:30:56.716800 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:56.698545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51182 (local address 127.0.97.3:37371)
0926 23:30:56.699282 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:56.699290 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:56.699323 (+ 33us) server_negotiation.cc:407] Connection header received
0926 23:30:56.699398 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:56.699408 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:56.699702 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:56.699855 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:56.701140 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:56.702310 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:56.704033 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:56.704822 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:56.704950 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:56.709840 (+ 4890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:56.709856 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:56.709878 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:56.709931 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:56.713303 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:56.714217 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:56.714227 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:56.714229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:56.714437 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:56.715029 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:56.715033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:56.715036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:56.715483 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:56.715763 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:56.716133 (+ 370us) server_negotiation.cc:299] Negotiation successful
0926 23:30:56.716411 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":463,"threads_started":1}
I20260926 23:30:56.718554 6507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:30:56.808326 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 6384
I20260926 23:30:56.823735 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 6023
I20260926 23:30:56.825165 6085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:30:56.942798 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 6023
2026-09-26T23:30:56Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4512 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:30:57 dist-test-slave-gwfx krb5kdc[6540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:30:57 dist-test-slave-gwfx krb5kdc[6540](info): set up 2 sockets
Sep 26 23:30:57 dist-test-slave-gwfx krb5kdc[6540](info): commencing operation
krb5kdc: starting...
W20260926 23:30:57.987872 388 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.007s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:30:58 dist-test-slave-gwfx krb5kdc[6540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:30:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:30:58Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260926 23:30:58.231045 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35009
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35009
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:30:58.624888 6553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:58.625275 6553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:58.625422 6553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:58.633327 6553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:30:58.633499 6553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:58.633584 6553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:58.633666 6553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:30:58.633731 6553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:30:58.644879 6553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:58.646713 6553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:58.648341 6553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:58.662890 6558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:58.663595 6559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:58.664588 6561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:58.665235 6553 server_base.cc:1034] running on GCE node
I20260926 23:30:58.666453 6553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:58.668653 6553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:58.669891 6553 hybrid_clock.cc:648] HybridClock initialized: now 1790465458669849 us; error 48 us; skew 500 ppm
I20260926 23:30:58.670389 6553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:58 dist-test-slave-gwfx krb5kdc[6540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465458, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:58.674081 6553 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260926 23:30:58.676090 6553 webserver.cc:460] Webserver started at http://127.0.97.62:43739/ using document root <none> and password file <none>
I20260926 23:30:58.676941 6553 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:58.677070 6553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:58.677407 6553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:58.681015 6553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "4e49775de4d9427ebcdcf5e2303fc63c"
format_stamp: "Formatted at 2026-09-26 23:30:58 on dist-test-slave-gwfx"
server_key: "b9de11d0d4f6b4b09eab95b213f91860"
server_key_iv: "f218e797021de25bcec57d3f1510db21"
server_key_version: "encryptionkey@0"
I20260926 23:30:58.681905 6553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "4e49775de4d9427ebcdcf5e2303fc63c"
format_stamp: "Formatted at 2026-09-26 23:30:58 on dist-test-slave-gwfx"
server_key: "b9de11d0d4f6b4b09eab95b213f91860"
server_key_iv: "f218e797021de25bcec57d3f1510db21"
server_key_version: "encryptionkey@0"
I20260926 23:30:58.688318 6553 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 23:30:58.692430 6568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:58.694146 6553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260926 23:30:58.694406 6553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "4e49775de4d9427ebcdcf5e2303fc63c"
format_stamp: "Formatted at 2026-09-26 23:30:58 on dist-test-slave-gwfx"
server_key: "b9de11d0d4f6b4b09eab95b213f91860"
server_key_iv: "f218e797021de25bcec57d3f1510db21"
server_key_version: "encryptionkey@0"
I20260926 23:30:58.694639 6553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:58.743958 6553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:58.749301 6553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:58.749634 6553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:58.771252 6553 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35009
I20260926 23:30:58.771261 6619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35009 every 8 connection(s)
I20260926 23:30:58.773082 6553 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:30:58.777782 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 6553
I20260926 23:30:58.777992 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:30:58.778378 6620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:30:58.778743 388 external_mini_cluster.cc:1428] Setting key 93f43bfafedc9e9ab481bf9839d3324a
I20260926 23:30:58.795665 6620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c: Bootstrap starting.
Sep 26 23:30:58 dist-test-slave-gwfx krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 23:30:58.799942 6620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c: Neither blocks nor log segments found. Creating new log.
I20260926 23:30:58.801623 6620 log.cc:824] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c: Log is configured to *not* fsync() on all Append() calls
I20260926 23:30:58.805771 6620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c: No bootstrap required, opened a new log
I20260926 23:30:58.808080 6623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:58.781141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40290 (local address 127.0.97.62:35009)
0926 23:30:58.782214 (+ 1073us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:58.782235 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:58.782782 (+ 547us) server_negotiation.cc:407] Connection header received
0926 23:30:58.785818 (+ 3036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:58.785865 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:58.786760 (+ 895us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:58.787454 (+ 694us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:58.789171 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:58.791333 (+ 2162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:58.792857 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:58.794136 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:58.794367 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:58.798782 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:58.798797 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:58.798825 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:58.798892 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:58.803139 (+ 4247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:58.804016 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:58.804026 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:58.804037 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:58.804280 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:58.804691 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:58.804695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:58.804698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:58.805495 (+ 797us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:58.805814 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:58.806271 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 23:30:58.806805 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":434,"threads_started":1}
I20260926 23:30:58.815959 6620 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } }
I20260926 23:30:58.816461 6620 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:30:58.816569 6620 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e49775de4d9427ebcdcf5e2303fc63c, State: Initialized, Role: FOLLOWER
I20260926 23:30:58.817263 6620 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } }
I20260926 23:30:58.817524 6620 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:30:58.817670 6620 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:30:58.817842 6620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:30:58.820623 6620 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } }
I20260926 23:30:58.821527 6620 leader_election.cc:302] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e49775de4d9427ebcdcf5e2303fc63c; no voters:
I20260926 23:30:58.822610 6620 leader_election.cc:288] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:30:58.822806 6625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:30:58.823536 6625 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [term 1 LEADER]: Becoming Leader. State: Replica: 4e49775de4d9427ebcdcf5e2303fc63c, State: Running, Role: LEADER
I20260926 23:30:58.824301 6625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } }
I20260926 23:30:58.825253 6620 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:30:58.828809 6627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e49775de4d9427ebcdcf5e2303fc63c. Latest consensus state: current_term: 1 leader_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } }
I20260926 23:30:58.828964 6626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } }
I20260926 23:30:58.829393 6626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [sys.catalog]: This master's current role is: LEADER
I20260926 23:30:58.829394 6627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [sys.catalog]: This master's current role is: LEADER
I20260926 23:30:58.833114 6633 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:30:58.839350 6633 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:30:58.850513 6633 catalog_manager.cc:1347] Generated new cluster ID: b180128e00d44860b0f3bc07ab669d75
I20260926 23:30:58.850659 6633 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:30:58.861891 6633 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:30:58.863590 6633 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:30:58.872413 6633 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c: Generated new TSK 0
I20260926 23:30:58.873598 6633 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260926 23:30:58.938210 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35009
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:30:59.372615 6646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:30:59.373021 6646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:30:59.373169 6646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:30:59.381474 6646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:30:59.381655 6646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:30:59.381822 6646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:30:59.393332 6646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:30:59.395246 6646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:30:59.397043 6646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:30:59.413720 6652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:59.413805 6654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:30:59.413750 6651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:30:59.414733 6646 server_base.cc:1034] running on GCE node
I20260926 23:30:59.415587 6646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:30:59.416850 6646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:30:59.418112 6646 hybrid_clock.cc:648] HybridClock initialized: now 1790465459418037 us; error 74 us; skew 500 ppm
I20260926 23:30:59.418527 6646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:30:59 dist-test-slave-gwfx krb5kdc[6540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465459, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:30:59.422894 6646 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260926 23:30:59.425083 6646 webserver.cc:460] Webserver started at http://127.0.97.1:41823/ using document root <none> and password file <none>
I20260926 23:30:59.426154 6646 fs_manager.cc:359] Metadata directory not provided
I20260926 23:30:59.426326 6646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:30:59.426719 6646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:30:59.431246 6646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/data/instance:
uuid: "152cd7b5a33b478695dbc5d5aba24d1e"
format_stamp: "Formatted at 2026-09-26 23:30:59 on dist-test-slave-gwfx"
server_key: "50de5c7bb786dd899b54d0907830406b"
server_key_iv: "8b398fe7ac756e8b354d2fb6a4ae01c7"
server_key_version: "encryptionkey@0"
I20260926 23:30:59.432415 6646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/wal/instance:
uuid: "152cd7b5a33b478695dbc5d5aba24d1e"
format_stamp: "Formatted at 2026-09-26 23:30:59 on dist-test-slave-gwfx"
server_key: "50de5c7bb786dd899b54d0907830406b"
server_key_iv: "8b398fe7ac756e8b354d2fb6a4ae01c7"
server_key_version: "encryptionkey@0"
I20260926 23:30:59.439976 6646 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260926 23:30:59.444802 6661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:59.446795 6646 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 23:30:59.447010 6646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "152cd7b5a33b478695dbc5d5aba24d1e"
format_stamp: "Formatted at 2026-09-26 23:30:59 on dist-test-slave-gwfx"
server_key: "50de5c7bb786dd899b54d0907830406b"
server_key_iv: "8b398fe7ac756e8b354d2fb6a4ae01c7"
server_key_version: "encryptionkey@0"
I20260926 23:30:59.447265 6646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:30:59.487979 6646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:30:59.493281 6646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:30:59.493609 6646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:30:59.494772 6646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:30:59.496336 6646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:30:59.496438 6646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:59.496564 6646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:30:59.496618 6646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:30:59.532382 6646 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39193
I20260926 23:30:59.532434 6773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39193 every 8 connection(s)
I20260926 23:30:59.534000 6646 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:30:59.540831 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 6646
I20260926 23:30:59.541071 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:30:59.541951 388 external_mini_cluster.cc:1428] Setting key 7af476519dacf7a3b17efaba521a6a41
Sep 26 23:30:59 dist-test-slave-gwfx krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465459, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 23:30:59.559993 6776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:59.537362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57503 (local address 127.0.97.62:35009)
0926 23:30:59.537971 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0926 23:30:59.537976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:30:59.539212 (+ 1236us) server_negotiation.cc:407] Connection header received
0926 23:30:59.541935 (+ 2723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:30:59.541940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:30:59.542043 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:30:59.542186 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:30:59.544257 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:59.545062 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:59.546912 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:30:59.547338 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:30:59.547460 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:59.552173 (+ 4713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:30:59.552181 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:30:59.552186 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:30:59.552236 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:30:59.555928 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:59.556939 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:59.556946 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:59.556949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:59.557111 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:30:59.557711 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:30:59.557715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:30:59.557718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:30:59.558118 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:30:59.558273 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 23:30:59.559303 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0926 23:30:59.559589 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":325,"threads_started":1}
I20260926 23:30:59.561340 6777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:30:59.538125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.97.1:57503)
0926 23:30:59.539018 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0926 23:30:59.539071 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 23:30:59.541469 (+ 2398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:30:59.542562 (+ 1093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:30:59.542572 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:30:59.543278 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:30:59.544053 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:59.544077 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:59.545240 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:59.545253 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:59.546740 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:30:59.546757 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:30:59.547600 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:30:59.547607 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:30:59.547875 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:30:59.548723 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:30:59.548758 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:30:59.551840 (+ 3082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:30:59.556173 (+ 4333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:59.556182 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:59.556203 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:59.556784 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:59.557282 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:30:59.557287 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:30:59.557289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:30:59.557578 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:30:59.558289 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:30:59.558297 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:30:59.558834 (+ 537us) client_negotiation.cc:769] Sending connection context
0926 23:30:59.559155 (+ 321us) client_negotiation.cc:240] Negotiation successful
0926 23:30:59.559728 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":331,"threads_started":1}
I20260926 23:30:59.563476 6774 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35009
I20260926 23:30:59.563959 6774 heartbeater.cc:464] Registering TS with master...
I20260926 23:30:59.565207 6774 heartbeater.cc:511] Master 127.0.97.62:35009 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 23:30:59.568403 6585 ts_manager.cc:194] Registered new tserver with Master: 152cd7b5a33b478695dbc5d5aba24d1e (127.0.97.1:39193)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
I20260926 23:30:59.570402 6585 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57503
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260926 23:30:59.603415 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35009
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:31:00.002933 6780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:31:00.003335 6780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:31:00.003473 6780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:31:00.012219 6780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:31:00.012403 6780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:31:00.012564 6780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:31:00.024063 6780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:31:00.025933 6780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:31:00.027669 6780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:00.044412 6788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:00.044468 6786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:00.044514 6785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:00.045400 6780 server_base.cc:1034] running on GCE node
I20260926 23:31:00.046205 6780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:00.047621 6780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:00.048897 6780 hybrid_clock.cc:648] HybridClock initialized: now 1790465460048820 us; error 83 us; skew 500 ppm
I20260926 23:31:00.049302 6780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:31:00 dist-test-slave-gwfx krb5kdc[6540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465460, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:31:00.061091 6780 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260926 23:31:00.064463 6780 webserver.cc:460] Webserver started at http://127.0.97.2:39517/ using document root <none> and password file <none>
I20260926 23:31:00.065691 6780 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:00.065852 6780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:00.066217 6780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:31:00.070034 6780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/data/instance:
uuid: "b228fb1eb3574b03868c488ff3df2172"
format_stamp: "Formatted at 2026-09-26 23:31:00 on dist-test-slave-gwfx"
server_key: "bfc81e771d892b52ad7f5c8598fa0865"
server_key_iv: "fe1b006213d93a74c23500bae477cb68"
server_key_version: "encryptionkey@0"
I20260926 23:31:00.071056 6780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b228fb1eb3574b03868c488ff3df2172"
format_stamp: "Formatted at 2026-09-26 23:31:00 on dist-test-slave-gwfx"
server_key: "bfc81e771d892b52ad7f5c8598fa0865"
server_key_iv: "fe1b006213d93a74c23500bae477cb68"
server_key_version: "encryptionkey@0"
I20260926 23:31:00.077877 6780 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260926 23:31:00.082790 6795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:00.084594 6780 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:31:00.084800 6780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "b228fb1eb3574b03868c488ff3df2172"
format_stamp: "Formatted at 2026-09-26 23:31:00 on dist-test-slave-gwfx"
server_key: "bfc81e771d892b52ad7f5c8598fa0865"
server_key_iv: "fe1b006213d93a74c23500bae477cb68"
server_key_version: "encryptionkey@0"
I20260926 23:31:00.085057 6780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:00.103976 6780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:00.109159 6780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:31:00.109534 6780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:00.110854 6780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:31:00.112703 6780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:31:00.112797 6780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:00.112919 6780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:31:00.112980 6780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:00.155352 6780 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37943
I20260926 23:31:00.155412 6907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37943 every 8 connection(s)
I20260926 23:31:00.156994 6780 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:31:00.160971 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 6780
I20260926 23:31:00.161180 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:31:00.161792 388 external_mini_cluster.cc:1428] Setting key 95e2345d37a30178875576afb2d0224f
Sep 26 23:31:00 dist-test-slave-gwfx krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465460, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 23:31:00.182109 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:00.160266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39981 (local address 127.0.97.62:35009)
0926 23:31:00.160778 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:00.160788 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:00.162116 (+ 1328us) server_negotiation.cc:407] Connection header received
0926 23:31:00.164459 (+ 2343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:00.164465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:00.164566 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:00.164707 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:00.166825 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:00.167882 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:00.169744 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:00.170439 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:00.170586 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:00.175239 (+ 4653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:00.175246 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:00.175250 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:00.175299 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:00.178167 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:00.179338 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:00.179343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:00.179346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:00.179523 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:00.180059 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:00.180062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:00.180064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:00.180446 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:00.180577 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:00.181542 (+ 965us) server_negotiation.cc:299] Negotiation successful
0926 23:31:00.181782 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":297,"threads_started":1}
I20260926 23:31:00.183758 6911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:00.160913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.97.2:39981)
0926 23:31:00.161914 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:00.161979 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:00.164023 (+ 2044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:00.165136 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:00.165179 (+ 43us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:00.165826 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:00.166538 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:00.166559 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:00.168130 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:00.168135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:00.169542 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:00.169561 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:00.170786 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:00.170795 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:00.171062 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:00.171860 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:00.171884 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:00.174910 (+ 3026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:00.178406 (+ 3496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:00.178417 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:00.178440 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:00.179165 (+ 725us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:00.179642 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:00.179647 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:00.179651 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:00.179924 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:00.180586 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:00.180594 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:00.181081 (+ 487us) client_negotiation.cc:769] Sending connection context
0926 23:31:00.181444 (+ 363us) client_negotiation.cc:240] Negotiation successful
0926 23:31:00.182079 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":362,"threads_started":1}
I20260926 23:31:00.185663 6908 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35009
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 23:31:00.186045 6908 heartbeater.cc:464] Registering TS with master...
I20260926 23:31:00.187047 6908 heartbeater.cc:511] Master 127.0.97.62:35009 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
I20260926 23:31:00.188932 6585 ts_manager.cc:194] Registered new tserver with Master: b228fb1eb3574b03868c488ff3df2172 (127.0.97.2:37943)
I20260926 23:31:00.190173 6585 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39981
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260926 23:31:00.219108 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35009
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:31:00.575049 6774 heartbeater.cc:503] Master 127.0.97.62:35009 was elected leader, sending a full tablet report...
W20260926 23:31:00.618816 6914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:31:00.619213 6914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:31:00.619345 6914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:31:00.627696 6914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:31:00.627847 6914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:31:00.628006 6914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:31:00.639343 6914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:31:00.641114 6914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:31:00.642822 6914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:00.658933 6922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:00.659148 6920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:00.658993 6919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:00.660898 6914 server_base.cc:1034] running on GCE node
I20260926 23:31:00.661731 6914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:00.663112 6914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:00.664409 6914 hybrid_clock.cc:648] HybridClock initialized: now 1790465460664309 us; error 82 us; skew 500 ppm
I20260926 23:31:00.664878 6914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:31:00 dist-test-slave-gwfx krb5kdc[6540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465460, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:31:00.669322 6914 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260926 23:31:00.671599 6914 webserver.cc:460] Webserver started at http://127.0.97.3:37179/ using document root <none> and password file <none>
I20260926 23:31:00.672562 6914 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:00.672714 6914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:00.673069 6914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:31:00.676908 6914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/data/instance:
uuid: "29d55f61ce314ce2924917d0d54dbffb"
format_stamp: "Formatted at 2026-09-26 23:31:00 on dist-test-slave-gwfx"
server_key: "82b5b5c3caccc7be8f9002000de2dad2"
server_key_iv: "d785e3c3c79598066bfb96679f3e9312"
server_key_version: "encryptionkey@0"
I20260926 23:31:00.678013 6914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/wal/instance:
uuid: "29d55f61ce314ce2924917d0d54dbffb"
format_stamp: "Formatted at 2026-09-26 23:31:00 on dist-test-slave-gwfx"
server_key: "82b5b5c3caccc7be8f9002000de2dad2"
server_key_iv: "d785e3c3c79598066bfb96679f3e9312"
server_key_version: "encryptionkey@0"
I20260926 23:31:00.685127 6914 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260926 23:31:00.690147 6929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:00.692162 6914 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 23:31:00.692421 6914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "29d55f61ce314ce2924917d0d54dbffb"
format_stamp: "Formatted at 2026-09-26 23:31:00 on dist-test-slave-gwfx"
server_key: "82b5b5c3caccc7be8f9002000de2dad2"
server_key_iv: "d785e3c3c79598066bfb96679f3e9312"
server_key_version: "encryptionkey@0"
I20260926 23:31:00.692703 6914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:00.706986 6914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:00.712113 6914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:31:00.712441 6914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:00.713649 6914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:31:00.715456 6914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:31:00.715553 6914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:00.715690 6914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:31:00.715742 6914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:00.752964 6914 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35483
I20260926 23:31:00.753060 7041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35483 every 8 connection(s)
I20260926 23:31:00.754527 6914 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:31:00.755509 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 6914
I20260926 23:31:00.755712 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:31:00.756302 388 external_mini_cluster.cc:1428] Setting key a89f9fe9e0e6ed94a5ba282a27c8f0f8
Sep 26 23:31:00 dist-test-slave-gwfx krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465460, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 23:31:00.779258 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:00.759004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52663 (local address 127.0.97.62:35009)
0926 23:31:00.759636 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:00.759642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:00.761062 (+ 1420us) server_negotiation.cc:407] Connection header received
0926 23:31:00.763212 (+ 2150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:00.763217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:00.763307 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:00.763451 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:00.765203 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:00.766088 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:00.767933 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:00.768387 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:00.768510 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:00.772871 (+ 4361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:00.772878 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:00.772881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:00.772937 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:00.775894 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:00.776748 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:00.776752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:00.776755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:00.776888 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:00.777333 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:00.777336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:00.777338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:00.777653 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:00.777789 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:00.778567 (+ 778us) server_negotiation.cc:299] Negotiation successful
0926 23:31:00.778855 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":300,"threads_started":1}
I20260926 23:31:00.780230 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:00.759842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.97.3:52663)
0926 23:31:00.760861 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:00.760919 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:00.762870 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:00.763698 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:00.763708 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:00.764299 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:00.764995 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:00.765015 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:00.766285 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:00.766289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:00.767745 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:00.767761 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:00.768614 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:00.768619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:00.768854 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:00.769583 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:00.769623 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:00.772633 (+ 3010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:00.776107 (+ 3474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:00.776117 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:00.776135 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:00.776607 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:00.777011 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:00.777017 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:00.777021 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:00.777215 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:00.777784 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:00.777794 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:00.778174 (+ 380us) client_negotiation.cc:769] Sending connection context
0926 23:31:00.778425 (+ 251us) client_negotiation.cc:240] Negotiation successful
0926 23:31:00.779024 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":412,"threads_started":1}
I20260926 23:31:00.781855 7042 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35009
I20260926 23:31:00.782331 7042 heartbeater.cc:464] Registering TS with master...
I20260926 23:31:00.783296 7042 heartbeater.cc:511] Master 127.0.97.62:35009 requested a full tablet report, sending...
I20260926 23:31:00.785274 6585 ts_manager.cc:194] Registered new tserver with Master: 29d55f61ce314ce2924917d0d54dbffb (127.0.97.3:35483)
I20260926 23:31:00.786454 6585 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52663
I20260926 23:31:00.786914 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260926 23:31:00.843988 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38317
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35009,127.0.97.61:38317
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:31:01.193629 6908 heartbeater.cc:503] Master 127.0.97.62:35009 was elected leader, sending a full tablet report...
W20260926 23:31:01.238734 7048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:31:01.239178 7048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:31:01.239322 7048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:31:01.247819 7048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:31:01.247995 7048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:31:01.248080 7048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:31:01.248162 7048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:31:01.248225 7048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:31:01.259744 7048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35009,127.0.97.61:38317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
W20260926 23:31:01.260251 7048 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:35009,127.0.97.61:38317'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 23:31:01.261694 7048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:31:01.263367 7048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:01.277634 7054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:01.277587 7056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:01.278384 7053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:01.279232 7048 server_base.cc:1034] running on GCE node
I20260926 23:31:01.280313 7048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:01.282426 7048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:01.283703 7048 hybrid_clock.cc:648] HybridClock initialized: now 1790465461283583 us; error 92 us; skew 500 ppm
I20260926 23:31:01.284118 7048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:31:01 dist-test-slave-gwfx krb5kdc[6540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465461, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:31:01.288136 7048 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260926 23:31:01.290203 7048 webserver.cc:460] Webserver started at http://127.0.97.61:33649/ using document root <none> and password file <none>
I20260926 23:31:01.291141 7048 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:01.291260 7048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:01.291627 7048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:31:01.295369 7048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-1/data/instance:
uuid: "002fa788f2fe4d8e83c0e20cec0b7b80"
format_stamp: "Formatted at 2026-09-26 23:31:01 on dist-test-slave-gwfx"
server_key: "c69dc51edcb0d36fc76c2c88db37f35b"
server_key_iv: "e458f4c94b218f6e4a1d45e0e454ccb2"
server_key_version: "encryptionkey@0"
I20260926 23:31:01.296561 7048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-1/wal/instance:
uuid: "002fa788f2fe4d8e83c0e20cec0b7b80"
format_stamp: "Formatted at 2026-09-26 23:31:01 on dist-test-slave-gwfx"
server_key: "c69dc51edcb0d36fc76c2c88db37f35b"
server_key_iv: "e458f4c94b218f6e4a1d45e0e454ccb2"
server_key_version: "encryptionkey@0"
I20260926 23:31:01.303954 7048 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260926 23:31:01.308717 7063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:01.310942 7048 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260926 23:31:01.311187 7048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-1/wal
uuid: "002fa788f2fe4d8e83c0e20cec0b7b80"
format_stamp: "Formatted at 2026-09-26 23:31:01 on dist-test-slave-gwfx"
server_key: "c69dc51edcb0d36fc76c2c88db37f35b"
server_key_iv: "e458f4c94b218f6e4a1d45e0e454ccb2"
server_key_version: "encryptionkey@0"
I20260926 23:31:01.311532 7048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:01.331660 7048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:01.337522 7048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:31:01.337918 7048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:01.359969 7048 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:38317
I20260926 23:31:01.360016 7114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:38317 every 8 connection(s)
I20260926 23:31:01.361560 7048 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-1/data/info.pb
I20260926 23:31:01.366286 7115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:31:01.370811 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 7048
I20260926 23:31:01.375648 7115 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:31:01 dist-test-slave-gwfx krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 26 23:31:01 dist-test-slave-gwfx krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465461, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 23:31:01.396486 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:01.379739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40300 (local address 127.0.97.62:35009)
0926 23:31:01.380285 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:01.380292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:01.380786 (+ 494us) server_negotiation.cc:407] Connection header received
0926 23:31:01.380993 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:01.380998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:01.381068 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:01.381164 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:01.382649 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:01.383695 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:01.385384 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:01.385836 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:01.385931 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:01.390387 (+ 4456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:01.390393 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:01.390397 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:01.390439 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:01.393307 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:01.394086 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:01.394091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:01.394094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:01.394234 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:01.394635 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:01.394639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:01.394643 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:01.395069 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:01.395216 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:01.395845 (+ 629us) server_negotiation.cc:299] Negotiation successful
0926 23:31:01.396118 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":254,"threads_started":1}
I20260926 23:31:01.397837 7118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:01.373738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34278 (local address 127.0.97.61:38317)
0926 23:31:01.374756 (+ 1018us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:01.374771 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:01.374824 (+ 53us) server_negotiation.cc:407] Connection header received
0926 23:31:01.376719 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:01.376752 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:01.377360 (+ 608us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:01.377877 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:01.379644 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:01.381582 (+ 1938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:01.383312 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:01.384313 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:01.384455 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:01.389917 (+ 5462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:01.389930 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:01.389952 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:01.390000 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:01.393031 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:01.393832 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:01.393840 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:01.393842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:01.394021 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:01.394411 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:01.394415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:01.394417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:01.395127 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:01.395471 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:01.395812 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 23:31:01.396373 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":721,"thread_start_us":425,"threads_started":1}
I20260926 23:31:01.397855 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:01.379784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.0.1:40300)
0926 23:31:01.380624 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:01.380665 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:01.380864 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:01.381257 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:01.381265 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:01.382136 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:01.382484 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:01.382497 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:01.383830 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:01.383834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:01.385246 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:01.385257 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:01.386117 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:01.386125 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:01.386316 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:01.386985 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:01.387026 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:01.390129 (+ 3103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:01.393460 (+ 3331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:01.393475 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:01.393492 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:01.393962 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:01.394343 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:01.394349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:01.394353 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:01.394531 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:01.395241 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:01.395251 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:01.395474 (+ 223us) client_negotiation.cc:769] Sending connection context
0926 23:31:01.395745 (+ 271us) client_negotiation.cc:240] Negotiation successful
0926 23:31:01.396512 (+ 767us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":163,"thread_start_us":453,"threads_started":1}
I20260926 23:31:01.400102 7115 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:31:01 dist-test-slave-gwfx krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465461, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260926 23:31:01.414436 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:01.401203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38317 (local address 127.0.0.1:34282)
0926 23:31:01.401491 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:01.401522 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:01.401649 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:01.401950 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:01.401953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:01.402410 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:01.402581 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:01.402590 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:01.403658 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:01.403662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:01.404693 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:01.404705 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:01.405419 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:01.405424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:01.405605 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:01.406202 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:01.406220 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:01.408574 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:01.411832 (+ 3258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:01.411838 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:01.411843 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:01.412245 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:01.412666 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:01.412671 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:01.412675 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:01.412816 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:01.413398 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:01.413403 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:01.413550 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 23:31:01.413676 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 23:31:01.414018 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260926 23:31:01.414474 7118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:01.401465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34282 (local address 127.0.97.61:38317)
0926 23:31:01.401612 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:01.401617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:01.401634 (+ 17us) server_negotiation.cc:407] Connection header received
0926 23:31:01.401759 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:01.401763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:01.401823 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:01.401913 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:01.402778 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:01.403532 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:01.404818 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:01.405264 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:01.405385 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:01.408825 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:01.408832 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:01.408836 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:01.408873 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:01.411684 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:01.412371 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:01.412375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:01.412377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:01.412556 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:01.412923 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:01.412928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:01.412930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:01.413286 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:01.413430 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:01.413775 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 23:31:01.414180 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":179,"server-negotiator.queue_time_us":46}
I20260926 23:31:01.420601 7115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80: Bootstrap starting.
I20260926 23:31:01.424672 7115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80: Neither blocks nor log segments found. Creating new log.
I20260926 23:31:01.426043 7115 log.cc:824] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80: Log is configured to *not* fsync() on all Append() calls
I20260926 23:31:01.429916 7115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80: No bootstrap required, opened a new log
I20260926 23:31:01.438195 7115 raft_consensus.cc:348] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } }
I20260926 23:31:01.438645 7115 raft_consensus.cc:374] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:31:01.438794 7115 raft_consensus.cc:729] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002fa788f2fe4d8e83c0e20cec0b7b80, State: Initialized, Role: FOLLOWER
I20260926 23:31:01.439549 7115 consensus_queue.cc:260] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } }
I20260926 23:31:01.440624 7122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } } }
I20260926 23:31:01.440999 7122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:31:01.442104 7115 sys_catalog.cc:564] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 23:31:01.452885 7133 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 23:31:01.453101 7133 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 23:31:01.456751 7048 master_runner.cc:428] Detected that this master 002fa788f2fe4d8e83c0e20cec0b7b80 is joining an existing cluster
I20260926 23:31:01.456899 7048 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:38317
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
I20260926 23:31:01.491518 7118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:01.467023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34284 (local address 127.0.97.61:38317)
0926 23:31:01.467266 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:01.467275 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:01.474168 (+ 6893us) server_negotiation.cc:407] Connection header received
0926 23:31:01.474393 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:01.474399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:01.474475 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:01.474555 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:01.476231 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:01.477368 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:01.479594 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:01.480324 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:01.480468 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:01.483246 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:01.483253 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:01.483259 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:01.483295 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:01.487474 (+ 4179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:01.488623 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:01.488630 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:01.488635 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:01.488824 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:01.489372 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:01.489377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:01.489381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:01.489876 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:01.490014 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:01.490777 (+ 763us) server_negotiation.cc:299] Negotiation successful
0926 23:31:01.491071 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":79}
I20260926 23:31:01.493054 7141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:01.466281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38317 (local address 127.0.0.1:34284)
0926 23:31:01.474033 (+ 7752us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:01.474065 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:01.474271 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:01.474801 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:01.474807 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:01.475672 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:01.476049 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:01.476062 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:01.477619 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:01.477624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:01.479433 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:01.479449 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:01.480840 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:01.480856 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:01.481054 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:01.481675 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:01.481705 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:01.482998 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:01.487682 (+ 4684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:01.487687 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:01.487691 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:01.488321 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:01.489058 (+ 737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:01.489063 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:01.489067 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:01.489256 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:01.490084 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:01.490088 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:01.490385 (+ 297us) client_negotiation.cc:769] Sending connection context
0926 23:31:01.492159 (+ 1774us) client_negotiation.cc:240] Negotiation successful
0926 23:31:01.492519 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7600,"mutex_wait_us":49,"thread_start_us":6436,"threads_started":1}
I20260926 23:31:01.493508 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:01.467397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40304 (local address 127.0.97.62:35009)
0926 23:31:01.467613 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:01.467620 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:01.473324 (+ 5704us) server_negotiation.cc:407] Connection header received
0926 23:31:01.473692 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:01.473699 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:01.473809 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:01.473883 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:01.475679 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:01.476804 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:01.479156 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:01.479856 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:01.479981 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:01.485147 (+ 5166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:01.485157 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:01.485163 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:01.485218 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:01.488708 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:01.489651 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:01.489657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:01.489660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:01.489843 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:01.490577 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:01.490582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:01.490585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:01.491030 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:01.492298 (+ 1268us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:01.492928 (+ 630us) server_negotiation.cc:299] Negotiation successful
0926 23:31:01.493193 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 23:31:01.494701 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:01.466229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.0.1:40304)
0926 23:31:01.473098 (+ 6869us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:01.473129 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:01.473509 (+ 380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:01.474127 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:01.474131 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:01.474958 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:01.475472 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:01.475487 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:01.477024 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:01.477029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:01.478945 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:01.478964 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:01.480414 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:01.480422 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:01.480694 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:01.482938 (+ 2244us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:01.482961 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:01.484776 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:01.488913 (+ 4137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:01.488918 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:01.488922 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:01.489490 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:01.489993 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:01.489998 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:01.490002 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:01.490437 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:01.492378 (+ 1941us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:01.492382 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:01.492597 (+ 215us) client_negotiation.cc:769] Sending connection context
0926 23:31:01.493781 (+ 1184us) client_negotiation.cc:240] Negotiation successful
0926 23:31:01.494173 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6665,"mutex_wait_us":58,"spinlock_wait_cycles":107776,"thread_start_us":7197,"threads_started":1}
Sep 26 23:31:01 dist-test-slave-gwfx krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465458, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260926 23:31:01.527815 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:43465
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35009,127.0.97.61:38317,127.0.97.60:43465
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:31:01.529182 7143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:01.502743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38317 (local address 127.0.0.1:34298)
0926 23:31:01.508685 (+ 5942us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:01.508744 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:01.509923 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:01.510501 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:01.510510 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:01.511519 (+ 1009us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:01.511774 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:01.511786 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:01.512946 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:01.512952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:01.514080 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:01.514094 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:01.514991 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:01.514997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:01.515217 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:01.515989 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:01.516017 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:01.521037 (+ 5020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:01.525041 (+ 4004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:01.525066 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:01.525077 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:01.525688 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:01.526427 (+ 739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:01.526435 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:01.526439 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:01.526727 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:01.527586 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:01.527597 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:01.527890 (+ 293us) client_negotiation.cc:769] Sending connection context
0926 23:31:01.528155 (+ 265us) client_negotiation.cc:240] Negotiation successful
0926 23:31:01.528677 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5612,"thread_start_us":5426,"threads_started":1}
I20260926 23:31:01.530468 7118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:01.502750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34298 (local address 127.0.97.61:38317)
0926 23:31:01.502993 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:01.503002 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:01.509698 (+ 6696us) server_negotiation.cc:407] Connection header received
0926 23:31:01.510195 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:01.510201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:01.510323 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:01.510426 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:01.511972 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:01.512805 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:01.514240 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:01.514813 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:01.514935 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:01.521378 (+ 6443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:01.521386 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:01.521390 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:01.521440 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:01.524778 (+ 3338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:01.525958 (+ 1180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:01.525965 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:01.525969 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:01.526248 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:01.526896 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:01.526903 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:01.526906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:01.527389 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:01.527600 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:01.529679 (+ 2079us) server_negotiation.cc:299] Negotiation successful
0926 23:31:01.530053 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 23:31:01.533087 6585 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:38317: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 38317
}
attrs {
promote: true
}
}
dest_uuid: "4e49775de4d9427ebcdcf5e2303fc63c"
cas_config_opid_index: -1
I20260926 23:31:01.534641 6585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: true } }
I20260926 23:31:01.542096 7145 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 002fa788f2fe4d8e83c0e20cec0b7b80 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: true } } }
I20260926 23:31:01.544107 7090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:31:01.547585 7145 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:38317
I20260926 23:31:01.548170 7146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: true } } }
I20260926 23:31:01.548522 7048 master.cc:537] Master@127.0.97.61:38317 shutting down...
I20260926 23:31:01.548533 7146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [sys.catalog]: This master's current role is: LEADER
I20260926 23:31:01.550238 7090 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [term 1 FOLLOWER]: Refusing update from remote peer 4e49775de4d9427ebcdcf5e2303fc63c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 23:31:01.552584 7146 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [LEADER]: Connected to new peer: Peer: permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 23:31:01.556154 6569 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260926 23:31:01.563750 7048 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:31:01.564497 7048 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:31:01.564767 7048 tablet_replica.cc:330] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80: stopping tablet replica
W20260926 23:31:01.567598 6569 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c -> Peer 002fa788f2fe4d8e83c0e20cec0b7b80 (127.0.97.61:38317): Couldn't send request to peer 002fa788f2fe4d8e83c0e20cec0b7b80. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 23:31:01.573175 7048 master.cc:552] Master@127.0.97.61:38317 shutdown complete.
I20260926 23:31:01.573313 7048 master_runner.cc:305] Clearing existing system tablet
I20260926 23:31:01.573848 7048 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:01.573954 7048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:01.577672 7048 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 23:31:01.580832 7151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:01.581226 7048 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:01.581607 7048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 23:31:01.581737 7048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-1/wal
uuid: "002fa788f2fe4d8e83c0e20cec0b7b80"
format_stamp: "Formatted at 2026-09-26 23:31:01 on dist-test-slave-gwfx"
server_key: "c69dc51edcb0d36fc76c2c88db37f35b"
server_key_iv: "e458f4c94b218f6e4a1d45e0e454ccb2"
server_key_version: "encryptionkey@0"
I20260926 23:31:01.587751 7048 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:31:01.596122 7048 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 23:31:01.596391 7048 log.cc:1196] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 23:31:01.596925 7048 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80: Deleting consensus metadata
I20260926 23:31:01.597210 7048 master_runner.cc:315] Copying system tablet from 127.0.97.62:35009
I20260926 23:31:01.604838 7048 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:35009
I20260926 23:31:01.628526 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:01.606619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40310 (local address 127.0.97.62:35009)
0926 23:31:01.606904 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:01.606912 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:01.611552 (+ 4640us) server_negotiation.cc:407] Connection header received
0926 23:31:01.611814 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:01.611820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:01.611947 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:01.612075 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:01.613504 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:01.614732 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:01.616705 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:01.617440 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:01.617585 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:01.620344 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:01.620352 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:01.620358 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:01.620407 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:01.624507 (+ 4100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:01.625727 (+ 1220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:01.625735 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:01.625740 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:01.625960 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:01.626550 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:01.626556 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:01.626560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:01.627044 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:01.627219 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:01.627822 (+ 603us) server_negotiation.cc:299] Negotiation successful
0926 23:31:01.628119 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260926 23:31:01.628870 7157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:01.606028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.0.1:40310)
0926 23:31:01.611323 (+ 5295us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:01.611361 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:01.611650 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:01.612340 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:01.612346 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:01.612967 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:01.613287 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:01.613302 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:01.614919 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:01.614926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:01.616508 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:01.616528 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:01.617630 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:01.617637 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:01.617834 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:01.618717 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:01.618750 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:01.620070 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:01.624824 (+ 4754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:01.624832 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:01.624837 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:01.625527 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:01.626124 (+ 597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:01.626130 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:01.626133 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:01.626365 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:01.627200 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:01.627207 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:01.627471 (+ 264us) client_negotiation.cc:769] Sending connection context
0926 23:31:01.627694 (+ 223us) client_negotiation.cc:240] Negotiation successful
0926 23:31:01.628183 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5082,"thread_start_us":4903,"threads_started":1}
I20260926 23:31:01.630175 6605 tablet_copy_service.cc:140] P 4e49775de4d9427ebcdcf5e2303fc63c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 002fa788f2fe4d8e83c0e20cec0b7b80 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:40310)
I20260926 23:31:01.630538 6605 tablet_copy_service.cc:161] P 4e49775de4d9427ebcdcf5e2303fc63c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 002fa788f2fe4d8e83c0e20cec0b7b80 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:40310: session id = 002fa788f2fe4d8e83c0e20cec0b7b80-00000000000000000000000000000000
I20260926 23:31:01.636004 6605 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 23:31:01.639921 7048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 23:31:01.649531 7048 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80: tablet copy: Starting download of 0 data blocks...
I20260926 23:31:01.650071 7048 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80: tablet copy: Starting download of 1 WAL segments...
I20260926 23:31:01.654287 7048 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 23:31:01.661233 6605 tablet_copy_service.cc:342] P 4e49775de4d9427ebcdcf5e2303fc63c: Request end of tablet copy session 002fa788f2fe4d8e83c0e20cec0b7b80-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:40310
I20260926 23:31:01.661580 6605 tablet_copy_service.cc:433] P 4e49775de4d9427ebcdcf5e2303fc63c: ending tablet copy session 002fa788f2fe4d8e83c0e20cec0b7b80-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 002fa788f2fe4d8e83c0e20cec0b7b80
I20260926 23:31:01.676648 7048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:01.680136 7158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:01.680686 7159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:01.688485 7048 server_base.cc:1034] running on GCE node
W20260926 23:31:01.689078 7162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:01.689677 7048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:01.690562 7048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:01.691891 7048 hybrid_clock.cc:648] HybridClock initialized: now 1790465461691831 us; error 71 us; skew 500 ppm
I20260926 23:31:01.692199 7048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:31:01 dist-test-slave-gwfx krb5kdc[6540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465461, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:31:01.694941 7048 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260926 23:31:01.703006 7048 webserver.cc:460] Webserver started at http://127.0.97.61:37535/ using document root <none> and password file <none>
I20260926 23:31:01.703637 7048 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:01.703753 7048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:01.707629 7048 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260926 23:31:01.710210 7168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:01.710817 7048 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260926 23:31:01.711006 7048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-1/wal
uuid: "002fa788f2fe4d8e83c0e20cec0b7b80"
format_stamp: "Formatted at 2026-09-26 23:31:01 on dist-test-slave-gwfx"
server_key: "c69dc51edcb0d36fc76c2c88db37f35b"
server_key_iv: "e458f4c94b218f6e4a1d45e0e454ccb2"
server_key_version: "encryptionkey@0"
I20260926 23:31:01.711216 7048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:01.761346 7048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:01.766021 7048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:01.790520 7042 heartbeater.cc:503] Master 127.0.97.62:35009 was elected leader, sending a full tablet report...
I20260926 23:31:01.840305 7048 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:38317
I20260926 23:31:01.840305 7219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:38317 every 8 connection(s)
I20260926 23:31:01.841917 7048 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-1/data/info.pb
I20260926 23:31:01.848662 7220 sys_catalog.cc:263] Verifying existing consensus state
I20260926 23:31:01.852357 7220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80: Bootstrap starting.
I20260926 23:31:01.870371 7220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 23:31:01.870931 7220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80: Bootstrap complete.
I20260926 23:31:01.871939 7220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: true } }
I20260926 23:31:01.872256 7220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 002fa788f2fe4d8e83c0e20cec0b7b80, State: Initialized, Role: LEARNER
I20260926 23:31:01.872543 7220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: true } }
I20260926 23:31:01.873489 7223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: true } } }
I20260926 23:31:01.873785 7220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:31:01.873912 7223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [sys.catalog]: This master's current role is: LEARNER
I20260926 23:31:01.887683 7234 catalog_manager.cc:1259] Loaded cluster ID: b180128e00d44860b0f3bc07ab669d75
I20260926 23:31:01.887815 7234 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80: loading cluster ID for follower catalog manager: success
I20260926 23:31:01.890815 7234 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80: acquiring CA information for follower catalog manager: success
I20260926 23:31:01.893093 7234 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 23:31:02.051718 7147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:31:02.052124 7147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:31:02.052265 7147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:31:02.061179 7147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:31:02.061367 7147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:31:02.061439 7147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:31:02.061498 7147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:31:02.061543 7147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:31:02.073388 7147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35009,127.0.97.61:38317,127.0.97.60:43465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:43465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:31:02.075302 7147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:31:02.076943 7147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:02.091599 7239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:02.091641 7240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:02.091673 7242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:02.092432 7147 server_base.cc:1034] running on GCE node
I20260926 23:31:02.093402 7147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:02.095264 7147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:02.096498 7147 hybrid_clock.cc:648] HybridClock initialized: now 1790465462096450 us; error 42 us; skew 500 ppm
I20260926 23:31:02.096966 7147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:31:02 dist-test-slave-gwfx krb5kdc[6540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465462, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:31:02.101358 7147 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260926 23:31:02.105599 7147 webserver.cc:460] Webserver started at http://127.0.97.60:35747/ using document root <none> and password file <none>
I20260926 23:31:02.106854 7147 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:02.107061 7147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:02.107487 7147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:31:02.107766 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:02.100000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38317 (local address 127.0.0.1:34314)
0926 23:31:02.100491 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:02.100525 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:02.100725 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:02.101289 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:02.101294 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:02.101330 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:31:02.101546 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:02.101560 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.103208 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.103216 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:02.104911 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:02.104940 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.106917 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.106924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:02.107127 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:02.107220 (+ 93us) client_negotiation.cc:769] Sending connection context
0926 23:31:02.107321 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 23:31:02.107465 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":228,"threads_started":1}
I20260926 23:31:02.107887 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:02.100249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34314 (local address 127.0.97.61:38317)
0926 23:31:02.100897 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:02.100904 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:02.100923 (+ 19us) server_negotiation.cc:407] Connection header received
0926 23:31:02.100998 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:02.101001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:02.101091 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:02.101222 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:31:02.101703 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.103025 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.105199 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.106656 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.106856 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:02.107119 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:02.107382 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 23:31:02.107543 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":335,"threads_started":1}
I20260926 23:31:02.109030 7195 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260926 23:31:02.109660 7223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e49775de4d9427ebcdcf5e2303fc63c. Latest consensus state: current_term: 1 leader_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: true } } }
I20260926 23:31:02.109941 7223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [sys.catalog]: This master's current role is: LEARNER
I20260926 23:31:02.111903 7147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-2/data/instance:
uuid: "8154d80de9b846d7a1fc101eb64b28b8"
format_stamp: "Formatted at 2026-09-26 23:31:02 on dist-test-slave-gwfx"
server_key: "529d823e42a37a59258ecea69d93348f"
server_key_iv: "d8c7eb2b4af5258ec2c39d0dc4c00637"
server_key_version: "encryptionkey@0"
I20260926 23:31:02.112936 7147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-2/wal/instance:
uuid: "8154d80de9b846d7a1fc101eb64b28b8"
format_stamp: "Formatted at 2026-09-26 23:31:02 on dist-test-slave-gwfx"
server_key: "529d823e42a37a59258ecea69d93348f"
server_key_iv: "d8c7eb2b4af5258ec2c39d0dc4c00637"
server_key_version: "encryptionkey@0"
I20260926 23:31:02.119954 7147 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260926 23:31:02.124599 7252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:02.126856 7147 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260926 23:31:02.127112 7147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-2/wal
uuid: "8154d80de9b846d7a1fc101eb64b28b8"
format_stamp: "Formatted at 2026-09-26 23:31:02 on dist-test-slave-gwfx"
server_key: "529d823e42a37a59258ecea69d93348f"
server_key_iv: "d8c7eb2b4af5258ec2c39d0dc4c00637"
server_key_version: "encryptionkey@0"
I20260926 23:31:02.127364 7147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:02.147467 7147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:02.153246 7147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:31:02.153589 7147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:02.176844 7147 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:43465
I20260926 23:31:02.176855 7303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:43465 every 8 connection(s)
I20260926 23:31:02.178889 7147 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-2/data/info.pb
I20260926 23:31:02.184305 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 7147
I20260926 23:31:02.184597 7304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:31:02.195808 7304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:31:02 dist-test-slave-gwfx krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 26 23:31:02 dist-test-slave-gwfx krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465462, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 23:31:02.218333 7308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:02.199710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40316 (local address 127.0.97.62:35009)
0926 23:31:02.200350 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:02.200356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:02.200938 (+ 582us) server_negotiation.cc:407] Connection header received
0926 23:31:02.201172 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:02.201177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:02.201255 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:02.201357 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:02.202624 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.203674 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.205957 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.206494 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.206659 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:02.211702 (+ 5043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:02.211711 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:02.211715 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:02.211759 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:02.215027 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:02.215908 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:02.215914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:02.215918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:02.216095 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:02.216596 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:02.216602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:02.216605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:02.216968 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:02.217119 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:02.217632 (+ 513us) server_negotiation.cc:299] Negotiation successful
0926 23:31:02.217886 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":318,"threads_started":1}
I20260926 23:31:02.218585 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:02.187107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38720 (local address 127.0.97.60:43465)
0926 23:31:02.188230 (+ 1123us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:02.188244 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:02.188302 (+ 58us) server_negotiation.cc:407] Connection header received
0926 23:31:02.190786 (+ 2484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:02.190821 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:02.191409 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:02.192094 (+ 685us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:02.193769 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.195327 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.197950 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.199170 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.199380 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:02.208381 (+ 9001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:02.208397 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:02.208422 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:02.208470 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:02.213221 (+ 4751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:02.214190 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:02.214212 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:02.214224 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:02.214415 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:02.214875 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:02.214881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:02.214885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:02.215781 (+ 896us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:02.216063 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:02.216666 (+ 603us) server_negotiation.cc:299] Negotiation successful
0926 23:31:02.217171 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":868,"thread_start_us":448,"threads_started":1}
I20260926 23:31:02.218914 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:02.199757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.0.1:40316)
0926 23:31:02.200747 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:02.200798 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:02.201015 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:02.201449 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:02.201455 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:02.202178 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:02.202465 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:02.202474 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.203835 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.203841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:02.205741 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:02.205763 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.206904 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.206913 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:02.207136 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:02.207984 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:02.208026 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:02.211438 (+ 3412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:02.215158 (+ 3720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:02.215168 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:02.215177 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:02.215755 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:02.216289 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:02.216292 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:02.216295 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:02.216459 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:02.217108 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:02.217117 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:02.217304 (+ 187us) client_negotiation.cc:769] Sending connection context
0926 23:31:02.217502 (+ 198us) client_negotiation.cc:240] Negotiation successful
0926 23:31:02.217894 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":541,"threads_started":1}
I20260926 23:31:02.221627 7304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:31:02 dist-test-slave-gwfx krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465462, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260926 23:31:02.236920 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:02.222789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38317 (local address 127.0.0.1:34324)
0926 23:31:02.222991 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:02.223012 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:02.223130 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:02.223500 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:02.223504 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:02.223914 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:02.224080 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:02.224089 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.225327 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.225332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:02.226416 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:02.226430 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.227383 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.227389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:02.227527 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:02.228177 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:02.228197 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:02.230600 (+ 2403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:02.234162 (+ 3562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:02.234167 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:02.234170 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:02.234747 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:02.235170 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:02.235176 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:02.235180 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:02.235357 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:02.235997 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:02.236001 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:02.236171 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 23:31:02.236302 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 23:31:02.236600 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 23:31:02.236944 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:02.222605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34324 (local address 127.0.97.61:38317)
0926 23:31:02.222825 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:02.222831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:02.223073 (+ 242us) server_negotiation.cc:407] Connection header received
0926 23:31:02.223243 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:02.223247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:02.223321 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:02.223412 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:02.224244 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.225086 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.226592 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.227114 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.227251 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:02.230892 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:02.230898 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:02.230901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:02.230944 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:02.233982 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:02.234867 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:02.234872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:02.234874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:02.235035 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:02.235503 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:02.235507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:02.235509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:02.235855 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:02.235993 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:02.236450 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 23:31:02.236685 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 23:31:02.238947 7304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43465 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:31:02 dist-test-slave-gwfx krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465462, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260926 23:31:02.253019 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:02.239756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43465 (local address 127.0.0.1:38722)
0926 23:31:02.239994 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:02.240014 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:02.240124 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:02.240441 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:02.240445 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:02.240863 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:02.241026 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:02.241034 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.242127 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.242131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:02.243231 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:02.243244 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.243941 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.243946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:02.244119 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:02.244769 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:02.244788 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:02.247180 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:02.250332 (+ 3152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:02.250336 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:02.250339 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:02.250846 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:02.251289 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:02.251292 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:02.251295 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:02.251458 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:02.252027 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:02.252031 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:02.252292 (+ 261us) client_negotiation.cc:769] Sending connection context
0926 23:31:02.252426 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 23:31:02.252685 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 23:31:02.253090 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:02.240008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38722 (local address 127.0.97.60:43465)
0926 23:31:02.240166 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:02.240170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:02.240185 (+ 15us) server_negotiation.cc:407] Connection header received
0926 23:31:02.240226 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:02.240229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:02.240287 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:02.240416 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:02.241163 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.241999 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.243382 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.243798 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.243902 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:02.247427 (+ 3525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:02.247432 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:02.247436 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:02.247477 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:02.250154 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:02.250999 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:02.251004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:02.251031 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:02.251174 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:02.251557 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:02.251561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:02.251563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:02.251896 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:02.252017 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:02.252537 (+ 520us) server_negotiation.cc:299] Negotiation successful
0926 23:31:02.252839 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":48}
I20260926 23:31:02.258893 7304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8: Bootstrap starting.
I20260926 23:31:02.264078 7304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8: Neither blocks nor log segments found. Creating new log.
I20260926 23:31:02.265479 7304 log.cc:824] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8: Log is configured to *not* fsync() on all Append() calls
I20260926 23:31:02.269410 7304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8: No bootstrap required, opened a new log
I20260926 23:31:02.279407 7304 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43465 } }
I20260926 23:31:02.279938 7304 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:31:02.280082 7304 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8154d80de9b846d7a1fc101eb64b28b8, State: Initialized, Role: FOLLOWER
I20260926 23:31:02.280879 7304 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43465 } }
I20260926 23:31:02.281908 7311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43465 } } }
I20260926 23:31:02.282272 7311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:31:02.283252 7304 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 23:31:02.295601 7322 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 23:31:02.295819 7322 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 23:31:02.300416 7147 master_runner.cc:428] Detected that this master 8154d80de9b846d7a1fc101eb64b28b8 is joining an existing cluster
I20260926 23:31:02.300549 7147 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:43465
I20260926 23:31:02.328401 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 6646
I20260926 23:31:02.332912 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:02.305412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34338 (local address 127.0.97.61:38317)
0926 23:31:02.305859 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:02.305866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:02.310958 (+ 5092us) server_negotiation.cc:407] Connection header received
0926 23:31:02.311506 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:02.311513 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:02.311607 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:02.311678 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:02.313455 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.314528 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.317238 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.317936 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.318032 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:02.323354 (+ 5322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:02.323363 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:02.323368 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:02.323422 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:02.327637 (+ 4215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:02.330021 (+ 2384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:02.330065 (+ 44us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:02.330069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:02.330283 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:02.330837 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:02.330843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:02.330846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:02.331275 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:02.331428 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:02.332114 (+ 686us) server_negotiation.cc:299] Negotiation successful
0926 23:31:02.332498 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291}
I20260926 23:31:02.333518 7308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:02.305170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40328 (local address 127.0.97.62:35009)
0926 23:31:02.305665 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:02.305674 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:02.311158 (+ 5484us) server_negotiation.cc:407] Connection header received
0926 23:31:02.311451 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:02.311459 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:02.311614 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:02.311701 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:02.313691 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.314897 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.317824 (+ 2927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.318582 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.318717 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:02.322803 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:02.322812 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:02.322818 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:02.322877 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:02.329296 (+ 6419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:02.330328 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:02.330335 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:02.330338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:02.330518 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:02.331046 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:02.331052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:02.331056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:02.331445 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:02.331572 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:02.332405 (+ 833us) server_negotiation.cc:299] Negotiation successful
0926 23:31:02.332762 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330}
I20260926 23:31:02.337500 7328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:02.305410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38317 (local address 127.0.0.1:34338)
0926 23:31:02.310828 (+ 5418us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:02.310848 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:02.311314 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:02.311917 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:02.311922 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:02.312850 (+ 928us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:02.313255 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:02.313267 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.314769 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.314774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:02.317065 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:02.317080 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.318321 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.318327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:02.318520 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:02.319257 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:02.319299 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:02.323007 (+ 3708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:02.329080 (+ 6073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:02.329086 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:02.329091 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:02.329633 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:02.330447 (+ 814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:02.330451 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:02.330454 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:02.330649 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:02.331512 (+ 863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:02.331518 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:02.331769 (+ 251us) client_negotiation.cc:769] Sending connection context
0926 23:31:02.333403 (+ 1634us) client_negotiation.cc:240] Negotiation successful
0926 23:31:02.336820 (+ 3417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5210,"spinlock_wait_cycles":41600,"thread_start_us":4006,"threads_started":1}
I20260926 23:31:02.340587 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:02.307581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.0.1:40328)
0926 23:31:02.311031 (+ 3450us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:02.311053 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:02.311272 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:02.311990 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:02.311999 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:02.313021 (+ 1022us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:02.313482 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:02.313494 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.315098 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.315104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:02.317624 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:02.317640 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.319033 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.319039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:02.319226 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:02.320959 (+ 1733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:02.320989 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:02.322491 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:02.329505 (+ 7014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:02.329510 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:02.329514 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:02.330182 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:02.330715 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:02.330721 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:02.330725 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:02.330905 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:02.331661 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:02.331666 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:02.332066 (+ 400us) client_negotiation.cc:769] Sending connection context
0926 23:31:02.334023 (+ 1957us) client_negotiation.cc:240] Negotiation successful
0926 23:31:02.339867 (+ 5844us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3313,"mutex_wait_us":4938,"spinlock_wait_cycles":7424,"thread_start_us":3390,"threads_started":1}
I20260926 23:31:02.347332 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:02.305181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43465 (local address 127.0.0.1:38738)
0926 23:31:02.309843 (+ 4662us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:02.309877 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:02.310213 (+ 336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:02.310905 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:02.310910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:02.311870 (+ 960us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:02.312460 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:02.312476 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.314242 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.314248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:02.316777 (+ 2529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:02.316794 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.318221 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.318228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:02.318432 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:02.319345 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:02.319371 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:02.321951 (+ 2580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:02.334603 (+ 12652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:02.334608 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:02.334611 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:02.335352 (+ 741us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:02.340114 (+ 4762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:02.340120 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:02.340125 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:02.340401 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:02.341212 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:02.341217 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:02.341560 (+ 343us) client_negotiation.cc:769] Sending connection context
0926 23:31:02.341750 (+ 190us) client_negotiation.cc:240] Negotiation successful
0926 23:31:02.342148 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4435,"mutex_wait_us":1,"thread_start_us":4877,"threads_started":1}
I20260926 23:31:02.348639 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:02.306019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38738 (local address 127.0.97.60:43465)
0926 23:31:02.306236 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:02.306242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:02.310043 (+ 3801us) server_negotiation.cc:407] Connection header received
0926 23:31:02.310410 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:02.310418 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:02.310525 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:02.310656 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:02.312638 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.313978 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.316952 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.317766 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.317887 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:02.326731 (+ 8844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:02.326739 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:02.326744 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:02.326798 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:02.334430 (+ 7632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:02.335580 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:02.335587 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:02.335590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:02.339608 (+ 4018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:02.340528 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:02.340532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:02.340535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:02.341048 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:02.341381 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:02.347926 (+ 6545us) server_negotiation.cc:299] Negotiation successful
0926 23:31:02.348241 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":68}
Sep 26 23:31:02 dist-test-slave-gwfx krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260926 23:31:02.361837 7311 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
Sep 26 23:31:02 dist-test-slave-gwfx krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465458, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260926 23:31:02.362275 7311 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43465 } }
I20260926 23:31:02.366377 7331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:02.330396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43888 (local address 127.0.97.1:39193)
0926 23:31:02.334406 (+ 4010us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:02.334417 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:02.334456 (+ 39us) server_negotiation.cc:407] Connection header received
0926 23:31:02.334540 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:02.334550 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:02.334975 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:02.338565 (+ 3590us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:02.338910 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.340524 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.344558 (+ 4034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.345740 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.346674 (+ 934us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:02.354766 (+ 8092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:02.354793 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:02.354820 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:02.354883 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:02.359757 (+ 4874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:02.360900 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:02.360912 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:02.360916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:02.361112 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:02.363296 (+ 2184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:02.363305 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:02.363309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:02.363843 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:02.365355 (+ 1512us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:02.365555 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 23:31:02.365890 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3743,"thread_start_us":3563,"threads_started":1}
I20260926 23:31:02.368641 6769 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:31:02.373683 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:02.352264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43465 (local address 127.0.0.1:38754)
0926 23:31:02.352686 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:02.352709 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:02.352859 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:02.353382 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:02.353387 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:02.353920 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:02.354141 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:02.354157 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.355703 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.355710 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:02.357184 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:02.357203 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.358367 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.358374 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:02.358558 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:02.359209 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:02.359235 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:02.363145 (+ 3910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:02.370164 (+ 7019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:02.370171 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:02.370176 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:02.370879 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:02.371448 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:02.371454 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:02.371458 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:02.371664 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:02.372465 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:02.372471 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:02.372671 (+ 200us) client_negotiation.cc:769] Sending connection context
0926 23:31:02.372878 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 23:31:02.373278 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281}
I20260926 23:31:02.374938 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:02.352343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38754 (local address 127.0.97.60:43465)
0926 23:31:02.352586 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:02.352593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:02.352768 (+ 175us) server_negotiation.cc:407] Connection header received
0926 23:31:02.353071 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:02.353076 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:02.353201 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:02.353306 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:02.354334 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.355554 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.357495 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.358198 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.358350 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:02.363370 (+ 5020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:02.363377 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:02.363385 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:02.363430 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:02.369831 (+ 6401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:02.371110 (+ 1279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:02.371116 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:02.371120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:02.371306 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:02.371820 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:02.371825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:02.371828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:02.372230 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:02.374044 (+ 1814us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:02.374202 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 23:31:02.374499 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260926 23:31:02.376693 6585 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:43465: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 43465
}
attrs {
promote: true
}
}
dest_uuid: "4e49775de4d9427ebcdcf5e2303fc63c"
cas_config_opid_index: 5
I20260926 23:31:02.377995 6585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: true } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: true } }
I20260926 23:31:02.379266 7311 leader_election.cc:288] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e49775de4d9427ebcdcf5e2303fc63c (127.0.97.62:35009), 002fa788f2fe4d8e83c0e20cec0b7b80 (127.0.97.61:38317)
I20260926 23:31:02.380185 7245 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8154d80de9b846d7a1fc101eb64b28b8 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: true } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: true } } }
I20260926 23:31:02.380530 7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8154d80de9b846d7a1fc101eb64b28b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" is_pre_election: true
I20260926 23:31:02.380999 7279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:31:02.381722 6595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8154d80de9b846d7a1fc101eb64b28b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" is_pre_election: true
I20260926 23:31:02.383410 7253 leader_election.cc:302] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8154d80de9b846d7a1fc101eb64b28b8; no voters: 002fa788f2fe4d8e83c0e20cec0b7b80, 4e49775de4d9427ebcdcf5e2303fc63c
I20260926 23:31:02.385715 7279 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [term 1 FOLLOWER]: Refusing update from remote peer 4e49775de4d9427ebcdcf5e2303fc63c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 23:31:02.385929 7311 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 23:31:02.395028 7195 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [term 1 LEARNER]: Refusing update from remote peer 4e49775de4d9427ebcdcf5e2303fc63c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20260926 23:31:02.406543 7258 tablet.cc:2369] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:31:02.408893 7335 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:31:02.411576 7335 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [LEADER]: Connected to new peer: Peer: permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 23:31:02.420526 7279 raft_consensus.cc:858] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43465 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: true } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: true } }
W20260926 23:31:02.420868 7279 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43465 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: true } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: true } }
I20260926 23:31:02.431056 7245 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:43465
I20260926 23:31:02.432220 7147 master.cc:537] Master@127.0.97.60:43465 shutting down...
I20260926 23:31:02.432801 7333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: true } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: true } } }
I20260926 23:31:02.433391 7333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [sys.catalog]: This master's current role is: LEADER
I20260926 23:31:02.441020 7195 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8154d80de9b846d7a1fc101eb64b28b8 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: true } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: true } } }
I20260926 23:31:02.446064 7223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: true } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: true } } }
I20260926 23:31:02.446408 7223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [sys.catalog]: This master's current role is: LEARNER
I20260926 23:31:02.447662 7335 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c: attempting to promote NON_VOTER 002fa788f2fe4d8e83c0e20cec0b7b80 to VOTER
I20260926 23:31:02.448796 7335 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: false } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: true } }
I20260926 23:31:02.457690 7195 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [term 1 LEARNER]: Refusing update from remote peer 4e49775de4d9427ebcdcf5e2303fc63c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 23:31:02.458808 7335 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [LEADER]: Connected to new peer: Peer: permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 23:31:02.463275 7336 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 002fa788f2fe4d8e83c0e20cec0b7b80 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: false } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: true } } }
I20260926 23:31:02.468253 7194 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 002fa788f2fe4d8e83c0e20cec0b7b80 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: false } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: true } } }
I20260926 23:31:02.475250 7333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: false } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: true } } }
I20260926 23:31:02.475492 7333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [sys.catalog]: This master's current role is: LEADER
I20260926 23:31:02.474840 7336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: false } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: true } } }
I20260926 23:31:02.475942 7336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [sys.catalog]: This master's current role is: LEADER
I20260926 23:31:02.475348 7279 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 002fa788f2fe4d8e83c0e20cec0b7b80 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: true } } }
I20260926 23:31:02.476701 7223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: false } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: true } } }
I20260926 23:31:02.476967 7223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 23:31:02.481299 6571 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c -> Peer 8154d80de9b846d7a1fc101eb64b28b8 (127.0.97.60:43465): Couldn't send request to peer 8154d80de9b846d7a1fc101eb64b28b8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 23:31:02.483084 7311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e49775de4d9427ebcdcf5e2303fc63c. Latest consensus state: current_term: 1 leader_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: true } } }
I20260926 23:31:02.483119 7340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: true } } }
I20260926 23:31:02.483389 7311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 23:31:02.483439 7340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 23:31:02.482616 7345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: true } } }
I20260926 23:31:02.484493 7345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 23:31:02.486665 7147 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260926 23:31:02.487231 7147 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260926 23:31:02.487428 7147 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8: stopping tablet replica
I20260926 23:31:02.496150 7147 master.cc:552] Master@127.0.97.60:43465 shutdown complete.
I20260926 23:31:02.496349 7147 master_runner.cc:305] Clearing existing system tablet
I20260926 23:31:02.496899 7147 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:02.497087 7147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:02.507544 7147 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.010s sys 0.000s
I20260926 23:31:02.509333 7347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:02.509989 7147 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:02.510267 7147 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 23:31:02.510373 7147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-2/wal
uuid: "8154d80de9b846d7a1fc101eb64b28b8"
format_stamp: "Formatted at 2026-09-26 23:31:02 on dist-test-slave-gwfx"
server_key: "529d823e42a37a59258ecea69d93348f"
server_key_iv: "d8c7eb2b4af5258ec2c39d0dc4c00637"
server_key_version: "encryptionkey@0"
I20260926 23:31:02.514532 7147 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:31:02.520490 7147 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 23:31:02.520705 7147 log.cc:1196] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 23:31:02.521205 7147 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8: Deleting consensus metadata
I20260926 23:31:02.521458 7147 master_runner.cc:315] Copying system tablet from 127.0.97.62:35009
I20260926 23:31:02.524492 7147 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:35009
I20260926 23:31:02.539037 7308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:02.525612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40336 (local address 127.0.97.62:35009)
0926 23:31:02.526070 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:02.526082 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:02.527181 (+ 1099us) server_negotiation.cc:407] Connection header received
0926 23:31:02.527393 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:02.527399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:02.527494 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:02.527583 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:02.528535 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.529333 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.530773 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.531242 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.531384 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:02.533460 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:02.533466 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:02.533470 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:02.533507 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:02.536212 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:02.536867 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:02.536873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:02.536877 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:02.537026 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:02.537422 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:02.537427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:02.537429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:02.537760 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:02.537911 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:02.538355 (+ 444us) server_negotiation.cc:299] Negotiation successful
0926 23:31:02.538593 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221}
I20260926 23:31:02.539122 7353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:02.526413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.0.1:40336)
0926 23:31:02.527037 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:02.527057 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:02.527257 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:02.527641 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:02.527646 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:02.528137 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:02.528376 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:02.528387 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.529487 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.529491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:02.530586 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:02.530603 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.531438 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.531444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:02.531622 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:02.532320 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:02.532339 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:02.533264 (+ 925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:02.536371 (+ 3107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:02.536377 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:02.536382 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:02.536751 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:02.537138 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:02.537143 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:02.537147 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:02.537320 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:02.537880 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:02.537886 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:02.538087 (+ 201us) client_negotiation.cc:769] Sending connection context
0926 23:31:02.538213 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 23:31:02.538582 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":317,"threads_started":1}
I20260926 23:31:02.540093 6605 tablet_copy_service.cc:140] P 4e49775de4d9427ebcdcf5e2303fc63c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8154d80de9b846d7a1fc101eb64b28b8 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:40336)
I20260926 23:31:02.540271 6605 tablet_copy_service.cc:161] P 4e49775de4d9427ebcdcf5e2303fc63c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8154d80de9b846d7a1fc101eb64b28b8 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:40336: session id = 8154d80de9b846d7a1fc101eb64b28b8-00000000000000000000000000000000
I20260926 23:31:02.543644 6605 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 23:31:02.545655 7147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 23:31:02.551124 7147 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8: tablet copy: Starting download of 0 data blocks...
I20260926 23:31:02.551510 7147 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8: tablet copy: Starting download of 1 WAL segments...
I20260926 23:31:02.554188 7147 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 23:31:02.558111 6605 tablet_copy_service.cc:342] P 4e49775de4d9427ebcdcf5e2303fc63c: Request end of tablet copy session 8154d80de9b846d7a1fc101eb64b28b8-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:40336
I20260926 23:31:02.558274 6605 tablet_copy_service.cc:433] P 4e49775de4d9427ebcdcf5e2303fc63c: ending tablet copy session 8154d80de9b846d7a1fc101eb64b28b8-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8154d80de9b846d7a1fc101eb64b28b8
I20260926 23:31:02.570568 7147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:02.573510 7354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:02.580350 7355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:02.592242 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 6646
I20260926 23:31:02.593616 7147 server_base.cc:1034] running on GCE node
W20260926 23:31:02.594976 7357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:02.595494 7147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:02.596222 7147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:02.597499 7147 hybrid_clock.cc:648] HybridClock initialized: now 1790465462597459 us; error 34 us; skew 500 ppm
I20260926 23:31:02.597797 7147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:31:02 dist-test-slave-gwfx krb5kdc[6540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465462, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:31:02.600039 7147 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260926 23:31:02.607896 7147 webserver.cc:460] Webserver started at http://127.0.97.60:37599/ using document root <none> and password file <none>
I20260926 23:31:02.608282 7147 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:02.608392 7147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:02.611268 7147 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260926 23:31:02.615175 7364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:02.615770 7147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 23:31:02.615943 7147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-2/wal
uuid: "8154d80de9b846d7a1fc101eb64b28b8"
format_stamp: "Formatted at 2026-09-26 23:31:02 on dist-test-slave-gwfx"
server_key: "529d823e42a37a59258ecea69d93348f"
server_key_iv: "d8c7eb2b4af5258ec2c39d0dc4c00637"
server_key_version: "encryptionkey@0"
I20260926 23:31:02.616153 7147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:02.616709 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39193
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:35009
--webserver_port=41823
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:35009,127.0.97.61:38317
--tserver_master_addrs=127.0.97.62:35009,127.0.97.61:38317,127.0.97.60:43465 with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:31:02.648610 7147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:02.652444 7147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:02.690275 7147 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:43465
I20260926 23:31:02.690287 7416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:43465 every 8 connection(s)
I20260926 23:31:02.691553 7147 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/master-2/data/info.pb
I20260926 23:31:02.701308 7417 sys_catalog.cc:263] Verifying existing consensus state
I20260926 23:31:02.706219 7417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8: Bootstrap starting.
I20260926 23:31:02.720399 7417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 23:31:02.721091 7417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8: Bootstrap complete.
I20260926 23:31:02.722138 7417 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: false } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: true } }
I20260926 23:31:02.722399 7417 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8154d80de9b846d7a1fc101eb64b28b8, State: Initialized, Role: LEARNER
I20260926 23:31:02.722647 7417 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: false } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: true } }
I20260926 23:31:02.723590 7420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: false } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: true } } }
I20260926 23:31:02.723960 7417 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:31:02.724143 7420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [sys.catalog]: This master's current role is: LEARNER
I20260926 23:31:02.733387 7431 catalog_manager.cc:1259] Loaded cluster ID: b180128e00d44860b0f3bc07ab669d75
I20260926 23:31:02.733510 7431 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8: loading cluster ID for follower catalog manager: success
I20260926 23:31:02.736302 7431 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8: acquiring CA information for follower catalog manager: success
I20260926 23:31:02.738425 7431 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 23:31:02.876508 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:02.867107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43465 (local address 127.0.0.1:38768)
0926 23:31:02.867467 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:02.867492 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:02.867671 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:02.868909 (+ 1238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:02.868917 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:02.868970 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:31:02.869265 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:02.869280 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.871431 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.871437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:02.873447 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:02.873474 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.875531 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.875538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:02.875781 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:02.875823 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 23:31:02.875946 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 23:31:02.876133 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205}
I20260926 23:31:02.876679 7432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:02.867378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38768 (local address 127.0.97.60:43465)
0926 23:31:02.868365 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:02.868372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:02.868397 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:31:02.868500 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:02.868505 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:02.868605 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:02.868770 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:31:02.869528 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.871220 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.873729 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:02.875255 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:02.875420 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:02.875776 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:02.876009 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 23:31:02.876206 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":811,"thread_start_us":553,"threads_started":1}
I20260926 23:31:02.878331 7392 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 23:31:02.879055 7420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e49775de4d9427ebcdcf5e2303fc63c. Latest consensus state: current_term: 1 leader_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: false } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: true } } }
I20260926 23:31:02.879454 7420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [sys.catalog]: This master's current role is: LEARNER
W20260926 23:31:03.086380 7369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:31:03.086767 7369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:31:03.086908 7369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:31:03.095434 7369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:31:03.095589 7369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:31:03.095738 7369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:31:03.106824 7369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39193
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=41823
--tserver_master_addrs=127.0.97.62:35009,127.0.97.61:38317,127.0.97.60:43465
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:31:03.108520 7369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:31:03.110196 7369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:03.127346 7440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:03.127351 7438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:03.127951 7437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:03.128446 7369 server_base.cc:1034] running on GCE node
I20260926 23:31:03.129227 7369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:03.130474 7369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:03.133884 7369 hybrid_clock.cc:648] HybridClock initialized: now 1790465463133764 us; error 134 us; skew 500 ppm
I20260926 23:31:03.134393 7369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:31:03 dist-test-slave-gwfx krb5kdc[6540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465463, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:31:03.139271 7369 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260926 23:31:03.141121 7369 webserver.cc:460] Webserver started at http://127.0.97.1:41823/ using document root <none> and password file <none>
I20260926 23:31:03.141853 7369 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:03.141995 7369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:03.148061 7369 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 23:31:03.151510 7447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:03.153307 7369 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260926 23:31:03.153523 7369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "152cd7b5a33b478695dbc5d5aba24d1e"
format_stamp: "Formatted at 2026-09-26 23:30:59 on dist-test-slave-gwfx"
server_key: "50de5c7bb786dd899b54d0907830406b"
server_key_iv: "8b398fe7ac756e8b354d2fb6a4ae01c7"
server_key_version: "encryptionkey@0"
I20260926 23:31:03.154597 7369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:03.175972 7369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:03.180963 7369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:31:03.181248 7369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:03.182331 7369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:31:03.183904 7369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:31:03.184003 7369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:03.184118 7369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:31:03.184186 7369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:03.225173 7369 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39193
I20260926 23:31:03.225267 7559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39193 every 8 connection(s)
I20260926 23:31:03.226853 7369 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:31:03.233256 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 7369
I20260926 23:31:03.233680 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 6780
Sep 26 23:31:03 dist-test-slave-gwfx krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260926 23:31:03.273023 7570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:03.235341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41068 (local address 127.0.97.2:37943)
0926 23:31:03.235982 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:03.235993 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:03.239394 (+ 3401us) server_negotiation.cc:407] Connection header received
0926 23:31:03.239695 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:03.239721 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:03.240059 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:03.240298 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:03.245535 (+ 5237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:03.247286 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:03.251347 (+ 4061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:03.252429 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:03.252559 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:03.264232 (+ 11673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:03.264251 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:03.264269 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:03.264318 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:03.268897 (+ 4579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:03.269952 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:03.269964 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:03.269967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:03.270176 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:03.270680 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:03.270711 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:03.270715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:03.271260 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:03.271593 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:03.272333 (+ 740us) server_negotiation.cc:299] Negotiation successful
0926 23:31:03.272583 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":323,"threads_started":1}
Sep 26 23:31:03 dist-test-slave-gwfx krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465463, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260926 23:31:03.276532 6903 generic_service.cc:196] Checking for leaks (request via RPC)
Sep 26 23:31:03 dist-test-slave-gwfx krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465463, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 26 23:31:03 dist-test-slave-gwfx krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465463, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 23:31:03.286353 7432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:03.232402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33723 (local address 127.0.97.60:43465)
0926 23:31:03.232625 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:03.232632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:03.260958 (+ 28326us) server_negotiation.cc:407] Connection header received
0926 23:31:03.262350 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:03.262356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:03.262479 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:03.262560 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:03.267563 (+ 5003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:03.268294 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:03.271419 (+ 3125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:03.272073 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:03.272184 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:03.278608 (+ 6424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:03.278616 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:03.278623 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:03.278674 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:03.282320 (+ 3646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:03.283230 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:03.283235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:03.283238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:03.283847 (+ 609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:03.284288 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:03.284293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:03.284297 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:03.284665 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:03.284793 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:03.285715 (+ 922us) server_negotiation.cc:299] Negotiation successful
0926 23:31:03.285967 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 23:31:03.287528 7565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:03.231341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53427 (local address 127.0.97.61:38317)
0926 23:31:03.263126 (+ 31785us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:03.263135 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:03.264926 (+ 1791us) server_negotiation.cc:407] Connection header received
0926 23:31:03.265088 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:03.265094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:03.265198 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:03.265309 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:03.267177 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:03.268380 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:03.271207 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:03.271887 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:03.271995 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:03.277872 (+ 5877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:03.277880 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:03.277885 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:03.277935 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:03.281414 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:03.282630 (+ 1216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:03.282636 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:03.282640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:03.283609 (+ 969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:03.284373 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:03.284379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:03.284382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:03.284807 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:03.284959 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:03.286673 (+ 1714us) server_negotiation.cc:299] Negotiation successful
0926 23:31:03.286974 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31588,"thread_start_us":31841,"threads_started":1}
I20260926 23:31:03.288508 7567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:03.231909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43465 (local address 127.0.97.1:33723)
0926 23:31:03.260801 (+ 28892us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:03.260829 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:03.262172 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:03.263190 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:03.263206 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:03.264262 (+ 1056us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:03.265327 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:03.265366 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:03.268459 (+ 3093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:03.268465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:03.270892 (+ 2427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:03.270911 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:03.272849 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:03.272858 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:03.273355 (+ 497us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:03.274121 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:03.274169 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:03.278317 (+ 4148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:03.282567 (+ 4250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:03.282573 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:03.282578 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:03.283090 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:03.283966 (+ 876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:03.283984 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:03.283988 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:03.284165 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:03.284883 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:03.284890 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:03.285325 (+ 435us) client_negotiation.cc:769] Sending connection context
0926 23:31:03.285583 (+ 258us) client_negotiation.cc:240] Negotiation successful
0926 23:31:03.286913 (+ 1330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28689,"mutex_wait_us":133,"thread_start_us":29176,"threads_started":1}
I20260926 23:31:03.288713 7566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:03.233026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38317 (local address 127.0.97.1:53427)
0926 23:31:03.264805 (+ 31779us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:03.264826 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:03.264964 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:03.265631 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:03.265636 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:03.266436 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:03.266966 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:03.266984 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:03.268613 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:03.268619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:03.271025 (+ 2406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:03.271039 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:03.272845 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:03.272855 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:03.273276 (+ 421us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:03.274203 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:03.274228 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:03.277519 (+ 3291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:03.281636 (+ 4117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:03.281649 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:03.281676 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:03.282460 (+ 784us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:03.283775 (+ 1315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:03.283781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:03.283785 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:03.284128 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:03.286101 (+ 1973us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:03.286106 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:03.286356 (+ 250us) client_negotiation.cc:769] Sending connection context
0926 23:31:03.287857 (+ 1501us) client_negotiation.cc:240] Negotiation successful
0926 23:31:03.288247 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31621,"mutex_wait_us":46,"thread_start_us":25452,"threads_started":1}
I20260926 23:31:03.290527 7562 heartbeater.cc:347] Connected to a master server at 127.0.97.61:38317
I20260926 23:31:03.290614 7560 heartbeater.cc:347] Connected to a master server at 127.0.97.60:43465
I20260926 23:31:03.290963 7562 heartbeater.cc:464] Registering TS with master...
I20260926 23:31:03.290968 7560 heartbeater.cc:464] Registering TS with master...
I20260926 23:31:03.292184 7560 heartbeater.cc:511] Master 127.0.97.60:43465 requested a full tablet report, sending...
I20260926 23:31:03.292209 7562 heartbeater.cc:511] Master 127.0.97.61:38317 requested a full tablet report, sending...
I20260926 23:31:03.295197 7382 ts_manager.cc:194] Registered new tserver with Master: 152cd7b5a33b478695dbc5d5aba24d1e (127.0.97.1:39193)
I20260926 23:31:03.295650 7185 ts_manager.cc:194] Registered new tserver with Master: 152cd7b5a33b478695dbc5d5aba24d1e (127.0.97.1:39193)
I20260926 23:31:03.297595 7568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:03.231790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.97.1:60017)
0926 23:31:03.255523 (+ 23733us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:03.255590 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:03.261738 (+ 6148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:03.265602 (+ 3864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:03.265608 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:03.266253 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:03.266614 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:03.266628 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:03.270812 (+ 4184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:03.270818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:03.272862 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:03.272874 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:03.274390 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:03.274396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:03.274620 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:03.275296 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:03.275316 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:03.284640 (+ 9324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:03.292782 (+ 8142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:03.292786 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:03.292789 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:03.293182 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:03.293557 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:03.293561 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:03.293563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:03.293714 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:03.296510 (+ 2796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:03.296516 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:03.296687 (+ 171us) client_negotiation.cc:769] Sending connection context
0926 23:31:03.296815 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 23:31:03.297155 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23257,"thread_start_us":32501,"threads_started":1}
I20260926 23:31:03.297665 7564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:03.230995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60017 (local address 127.0.97.62:35009)
0926 23:31:03.263660 (+ 32665us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:03.263667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:03.263695 (+ 28us) server_negotiation.cc:407] Connection header received
0926 23:31:03.263794 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:03.263800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:03.263912 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:03.264087 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:03.269130 (+ 5043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:03.270277 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:03.273327 (+ 3050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:03.273983 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:03.274100 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:03.285200 (+ 11100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:03.285209 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:03.285215 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:03.285267 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:03.292571 (+ 7304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:03.293320 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:03.293326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:03.293330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:03.293454 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:03.295933 (+ 2479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:03.295939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:03.295942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:03.296385 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:03.296555 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:03.296949 (+ 394us) server_negotiation.cc:299] Negotiation successful
0926 23:31:03.297255 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32524,"thread_start_us":34013,"threads_started":1}
I20260926 23:31:03.298530 7561 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35009
I20260926 23:31:03.298766 7561 heartbeater.cc:464] Registering TS with master...
I20260926 23:31:03.299333 7561 heartbeater.cc:511] Master 127.0.97.62:35009 requested a full tablet report, sending...
I20260926 23:31:03.300647 6585 ts_manager.cc:194] Re-registered known tserver with Master: 152cd7b5a33b478695dbc5d5aba24d1e (127.0.97.1:39193)
I20260926 23:31:03.301494 6585 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60017
I20260926 23:31:03.376272 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 6780
I20260926 23:31:03.391067 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:37943
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:35009
--webserver_port=39517
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:35009,127.0.97.61:38317
--tserver_master_addrs=127.0.97.62:35009,127.0.97.61:38317,127.0.97.60:43465 with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:31:03.484395 7336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: false } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: true } } }
I20260926 23:31:03.484792 7336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [sys.catalog]: This master's current role is: LEADER
I20260926 23:31:03.486536 7574 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c: attempting to promote NON_VOTER 8154d80de9b846d7a1fc101eb64b28b8 to VOTER
I20260926 23:31:03.487354 7574 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: false } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: false } }
I20260926 23:31:03.489660 7195 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [term 1 FOLLOWER]: Refusing update from remote peer 4e49775de4d9427ebcdcf5e2303fc63c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 23:31:03.489835 7392 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [term 1 LEARNER]: Refusing update from remote peer 4e49775de4d9427ebcdcf5e2303fc63c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 23:31:03.490409 7573 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [LEADER]: Connected to new peer: Peer: permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 23:31:03.490950 7574 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 23:31:03.494418 7573 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8154d80de9b846d7a1fc101eb64b28b8 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: false } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: false } } }
I20260926 23:31:03.495627 7392 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8154d80de9b846d7a1fc101eb64b28b8 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: false } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: false } } }
I20260926 23:31:03.498020 7573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: false } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: false } } }
I20260926 23:31:03.498497 7336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: false } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: false } } }
I20260926 23:31:03.498945 7336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [sys.catalog]: This master's current role is: LEADER
I20260926 23:31:03.498536 7195 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8154d80de9b846d7a1fc101eb64b28b8 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: false } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: false } } }
I20260926 23:31:03.499176 7336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: false } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: false } } }
I20260926 23:31:03.499444 7336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [sys.catalog]: This master's current role is: LEADER
I20260926 23:31:03.499775 7573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c [sys.catalog]: This master's current role is: LEADER
I20260926 23:31:03.506057 7581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: false } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: false } } }
I20260926 23:31:03.507390 7580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: false } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: false } } }
I20260926 23:31:03.507958 7580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:31:03.508229 7581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 23:31:03.790802 7572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:31:03.791214 7572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:31:03.791358 7572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:31:03.800305 7572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:31:03.800493 7572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:31:03.800664 7572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:31:03.812680 7572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:37943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=39517
--tserver_master_addrs=127.0.97.62:35009,127.0.97.61:38317,127.0.97.60:43465
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:31:03.814677 7572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:31:03.816596 7572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:03.834385 7587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:03.834384 7589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:03.834434 7586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:03.835357 7572 server_base.cc:1034] running on GCE node
I20260926 23:31:03.836176 7572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:03.837419 7572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:03.838747 7572 hybrid_clock.cc:648] HybridClock initialized: now 1790465463838643 us; error 59 us; skew 500 ppm
I20260926 23:31:03.839164 7572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:31:03 dist-test-slave-gwfx krb5kdc[6540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465463, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:31:03.843091 7572 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260926 23:31:03.845065 7572 webserver.cc:460] Webserver started at http://127.0.97.2:39517/ using document root <none> and password file <none>
I20260926 23:31:03.845942 7572 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:03.846077 7572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:03.852735 7572 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 23:31:03.856451 7596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:03.858258 7572 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 23:31:03.858490 7572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "b228fb1eb3574b03868c488ff3df2172"
format_stamp: "Formatted at 2026-09-26 23:31:00 on dist-test-slave-gwfx"
server_key: "bfc81e771d892b52ad7f5c8598fa0865"
server_key_iv: "fe1b006213d93a74c23500bae477cb68"
server_key_version: "encryptionkey@0"
I20260926 23:31:03.859758 7572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:03.882489 7572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:03.887398 7572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:31:03.887684 7572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:03.888856 7572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:31:03.890468 7572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:31:03.890565 7572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:03.890666 7572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:31:03.890764 7572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260926 23:31:03.928857 7572 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37943
I20260926 23:31:03.929060 7708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37943 every 8 connection(s)
I20260926 23:31:03.930576 7572 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:31:03.937330 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 7572
I20260926 23:31:03.937628 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 6914
Sep 26 23:31:03 dist-test-slave-gwfx krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465463, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 26 23:31:03 dist-test-slave-gwfx krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
Sep 26 23:31:03 dist-test-slave-gwfx krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465463, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260926 23:31:03.974293 7713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:03.934487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58713 (local address 127.0.97.62:35009)
0926 23:31:03.949848 (+ 15361us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:03.949854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:03.949882 (+ 28us) server_negotiation.cc:407] Connection header received
0926 23:31:03.950793 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:03.950799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:03.950914 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:03.951022 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:03.953109 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:03.953925 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:03.956295 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:03.956734 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:03.956853 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:03.965094 (+ 8241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:03.965102 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:03.965107 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:03.965173 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:03.969203 (+ 4030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:03.970901 (+ 1698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:03.970908 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:03.970913 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:03.971092 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:03.971808 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:03.971814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:03.971817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:03.972231 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:03.972418 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:03.973580 (+ 1162us) server_negotiation.cc:299] Negotiation successful
0926 23:31:03.973897 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15216,"thread_start_us":15392,"threads_started":1}
I20260926 23:31:03.976061 7715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:03.935547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.97.2:58713)
0926 23:31:03.945098 (+ 9551us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:03.945178 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:03.950318 (+ 5140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:03.951364 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:03.951378 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:03.952180 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:03.952911 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:03.952931 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:03.954084 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:03.954090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:03.956148 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:03.956165 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:03.956994 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:03.957002 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:03.957351 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:03.958555 (+ 1204us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:03.958592 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:03.964789 (+ 6197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:03.969449 (+ 4660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:03.969464 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:03.969494 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:03.970672 (+ 1178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:03.971244 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:03.971250 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:03.971254 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:03.971650 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:03.972396 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:03.972407 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:03.973044 (+ 637us) client_negotiation.cc:769] Sending connection context
0926 23:31:03.973445 (+ 401us) client_negotiation.cc:240] Negotiation successful
0926 23:31:03.974162 (+ 717us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9046,"thread_start_us":11154,"threads_started":1}
Sep 26 23:31:03 dist-test-slave-gwfx krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465463, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260926 23:31:03.980363 7710 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35009
I20260926 23:31:03.980849 7710 heartbeater.cc:464] Registering TS with master...
I20260926 23:31:03.980980 7714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:03.934500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55093 (local address 127.0.97.61:38317)
0926 23:31:03.948579 (+ 14079us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:03.948587 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:03.962531 (+ 13944us) server_negotiation.cc:407] Connection header received
0926 23:31:03.962739 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:03.962745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:03.962865 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:03.962991 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:03.964846 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:03.966129 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:03.968246 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:03.968852 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:03.968960 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:03.974371 (+ 5411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:03.974379 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:03.974385 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:03.974455 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:03.977876 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:03.978816 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:03.978822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:03.978826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:03.979012 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:03.979510 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:03.979515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:03.979519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:03.979875 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:03.979981 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:03.980423 (+ 442us) server_negotiation.cc:299] Negotiation successful
0926 23:31:03.980685 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13912,"thread_start_us":13726,"threads_started":1}
I20260926 23:31:03.981869 7718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:03.936384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38317 (local address 127.0.97.2:55093)
0926 23:31:03.962378 (+ 25994us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:03.962403 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:03.962565 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:03.963269 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:03.963276 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:03.964205 (+ 929us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:03.964627 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:03.964649 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:03.966346 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:03.966351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:03.968045 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:03.968064 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:03.969306 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:03.969312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:03.969479 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:03.970161 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:03.970179 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:03.974133 (+ 3954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:03.978045 (+ 3912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:03.978050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:03.978054 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:03.978623 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:03.979175 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:03.979183 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:03.979187 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:03.979335 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:03.980053 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:03.980057 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:03.980179 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 23:31:03.981260 (+ 1081us) client_negotiation.cc:240] Negotiation successful
0926 23:31:03.981522 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25820,"thread_start_us":26446,"threads_started":1}
I20260926 23:31:03.982064 7710 heartbeater.cc:511] Master 127.0.97.62:35009 requested a full tablet report, sending...
I20260926 23:31:03.983145 7711 heartbeater.cc:347] Connected to a master server at 127.0.97.61:38317
I20260926 23:31:03.983294 7711 heartbeater.cc:464] Registering TS with master...
I20260926 23:31:03.983826 7711 heartbeater.cc:511] Master 127.0.97.61:38317 requested a full tablet report, sending...
I20260926 23:31:03.984309 6585 ts_manager.cc:194] Re-registered known tserver with Master: b228fb1eb3574b03868c488ff3df2172 (127.0.97.2:37943)
I20260926 23:31:03.985189 7185 ts_manager.cc:194] Registered new tserver with Master: b228fb1eb3574b03868c488ff3df2172 (127.0.97.2:37943)
I20260926 23:31:03.985431 6585 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58713
I20260926 23:31:03.987841 7720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:03.939181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51364 (local address 127.0.97.3:35483)
0926 23:31:03.955375 (+ 16194us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:03.955386 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:03.956109 (+ 723us) server_negotiation.cc:407] Connection header received
0926 23:31:03.956346 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:03.956357 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:03.956688 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:03.956892 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:03.959394 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:03.961699 (+ 2305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:03.963535 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:03.964495 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:03.964640 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:03.974482 (+ 9842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:03.974506 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:03.974528 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:03.974588 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:03.979522 (+ 4934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:03.984434 (+ 4912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:03.984446 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:03.984449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:03.984670 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:03.985226 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:03.985231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:03.985234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:03.985759 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:03.986436 (+ 677us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:03.987074 (+ 638us) server_negotiation.cc:299] Negotiation successful
0926 23:31:03.987410 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15965,"thread_start_us":15772,"threads_started":1}
I20260926 23:31:03.988629 7716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:03.935890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33637 (local address 127.0.97.60:43465)
0926 23:31:03.966112 (+ 30222us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:03.966118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:03.966140 (+ 22us) server_negotiation.cc:407] Connection header received
0926 23:31:03.966201 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:03.966204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:03.966284 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:03.966395 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:03.967835 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:03.969217 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:03.974930 (+ 5713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:03.976079 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:03.977609 (+ 1530us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:03.981395 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:03.981404 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:03.981408 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:03.981458 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:03.985129 (+ 3671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:03.986090 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:03.986096 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:03.986100 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:03.986259 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:03.986758 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:03.986763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:03.986766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:03.987195 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:03.987343 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:03.987961 (+ 618us) server_negotiation.cc:299] Negotiation successful
0926 23:31:03.988252 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30063,"thread_start_us":29877,"threads_started":1}
I20260926 23:31:03.988656 7717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:03.938102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43465 (local address 127.0.97.2:33637)
0926 23:31:03.963170 (+ 25068us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:03.963189 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:03.963324 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:03.966468 (+ 3144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:03.966474 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:03.967226 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:03.967592 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:03.967606 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:03.971275 (+ 3669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:03.971281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:03.973205 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:03.973220 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:03.977117 (+ 3897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:03.977124 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:03.977317 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:03.977936 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:03.977960 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:03.981084 (+ 3124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:03.985329 (+ 4245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:03.985335 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:03.985339 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:03.985866 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:03.986385 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:03.986390 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:03.986394 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:03.986579 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:03.987400 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:03.987405 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:03.987611 (+ 206us) client_negotiation.cc:769] Sending connection context
0926 23:31:03.987754 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 23:31:03.988201 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24939,"mutex_wait_us":24,"thread_start_us":23895,"threads_started":1}
I20260926 23:31:03.989702 7709 heartbeater.cc:347] Connected to a master server at 127.0.97.60:43465
I20260926 23:31:03.989763 7037 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:31:03.989877 7709 heartbeater.cc:464] Registering TS with master...
I20260926 23:31:03.990377 7709 heartbeater.cc:511] Master 127.0.97.60:43465 requested a full tablet report, sending...
I20260926 23:31:03.991721 7382 ts_manager.cc:194] Registered new tserver with Master: b228fb1eb3574b03868c488ff3df2172 (127.0.97.2:37943)
I20260926 23:31:04.083106 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 6914
I20260926 23:31:04.099678 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35483
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:35009
--webserver_port=37179
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:35009,127.0.97.61:38317
--tserver_master_addrs=127.0.97.62:35009,127.0.97.61:38317,127.0.97.60:43465 with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:31:04.305015 7561 heartbeater.cc:503] Master 127.0.97.62:35009 was elected leader, sending a full tablet report...
W20260926 23:31:04.498535 7722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:31:04.499003 7722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:31:04.499176 7722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:31:04.507541 7722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:31:04.507689 7722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:31:04.507858 7722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:31:04.519265 7722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=37179
--tserver_master_addrs=127.0.97.62:35009,127.0.97.61:38317,127.0.97.60:43465
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:31:04.521103 7722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:31:04.522894 7722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:04.539744 7730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:04.539798 7728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:04.539809 7727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:04.540846 7722 server_base.cc:1034] running on GCE node
I20260926 23:31:04.541627 7722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:04.542925 7722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:04.544169 7722 hybrid_clock.cc:648] HybridClock initialized: now 1790465464544119 us; error 44 us; skew 500 ppm
I20260926 23:31:04.544556 7722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:31:04 dist-test-slave-gwfx krb5kdc[6540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465464, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:31:04.548462 7722 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260926 23:31:04.550357 7722 webserver.cc:460] Webserver started at http://127.0.97.3:37179/ using document root <none> and password file <none>
I20260926 23:31:04.551316 7722 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:04.551466 7722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:04.558461 7722 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 23:31:04.562269 7737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:04.564380 7722 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:31:04.564596 7722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "29d55f61ce314ce2924917d0d54dbffb"
format_stamp: "Formatted at 2026-09-26 23:31:00 on dist-test-slave-gwfx"
server_key: "82b5b5c3caccc7be8f9002000de2dad2"
server_key_iv: "d785e3c3c79598066bfb96679f3e9312"
server_key_version: "encryptionkey@0"
I20260926 23:31:04.565877 7722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:04.588740 7722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:04.594000 7722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:31:04.594316 7722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:04.595603 7722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:31:04.597277 7722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:31:04.597384 7722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:04.597501 7722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:31:04.597571 7722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:04.634812 7722 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35483
I20260926 23:31:04.634881 7849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35483 every 8 connection(s)
I20260926 23:31:04.636682 7722 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:31:04.646370 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 7722
Sep 26 23:31:04 dist-test-slave-gwfx krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465464, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 26 23:31:04 dist-test-slave-gwfx krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465464, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260926 23:31:04.686491 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:04.641055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55931 (local address 127.0.97.62:35009)
0926 23:31:04.653198 (+ 12143us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:04.653204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:04.653233 (+ 29us) server_negotiation.cc:407] Connection header received
0926 23:31:04.657376 (+ 4143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:04.657383 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:04.657486 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:04.657582 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:04.661383 (+ 3801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:04.662403 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:04.665243 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:04.665761 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:04.665893 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:04.671691 (+ 5798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:04.671700 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:04.671705 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:04.671759 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:04.677207 (+ 5448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:04.679776 (+ 2569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:04.679783 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:04.679787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:04.679970 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:04.682506 (+ 2536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:04.682512 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:04.682516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:04.682995 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:04.683167 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:04.685773 (+ 2606us) server_negotiation.cc:299] Negotiation successful
0926 23:31:04.686068 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11998,"thread_start_us":12173,"threads_started":1}
I20260926 23:31:04.688127 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:04.641423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.97.3:55931)
0926 23:31:04.652506 (+ 11083us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:04.652611 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:04.656885 (+ 4274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:04.658285 (+ 1400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:04.658302 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:04.659607 (+ 1305us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:04.660800 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:04.660836 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:04.662586 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:04.662594 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:04.665072 (+ 2478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:04.665096 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:04.666029 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:04.666036 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:04.666417 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:04.667534 (+ 1117us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:04.667572 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:04.671325 (+ 3753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:04.677421 (+ 6096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:04.677436 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:04.677469 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:04.678233 (+ 764us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:04.681960 (+ 3727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:04.681967 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:04.681972 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:04.682364 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:04.684537 (+ 2173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:04.684551 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:04.685258 (+ 707us) client_negotiation.cc:769] Sending connection context
0926 23:31:04.685626 (+ 368us) client_negotiation.cc:240] Negotiation successful
0926 23:31:04.686324 (+ 698us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10563,"mutex_wait_us":50,"thread_start_us":544,"threads_started":1}
I20260926 23:31:04.691589 7851 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35009
I20260926 23:31:04.692054 7851 heartbeater.cc:464] Registering TS with master...
I20260926 23:31:04.693194 7851 heartbeater.cc:511] Master 127.0.97.62:35009 requested a full tablet report, sending...
I20260926 23:31:04.695317 6585 ts_manager.cc:194] Re-registered known tserver with Master: 29d55f61ce314ce2924917d0d54dbffb (127.0.97.3:35483)
I20260926 23:31:04.696555 6585 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55931
I20260926 23:31:04.703538 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:04.640503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55673 (local address 127.0.97.61:38317)
0926 23:31:04.653745 (+ 13242us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:04.653752 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:04.656500 (+ 2748us) server_negotiation.cc:407] Connection header received
0926 23:31:04.658812 (+ 2312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:04.658819 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:04.658935 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:04.659034 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:04.668662 (+ 9628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:04.669810 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:04.673232 (+ 3422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:04.673975 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:04.674098 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:04.679429 (+ 5331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:04.679438 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:04.679444 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:04.679498 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:04.684321 (+ 4823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:04.697026 (+ 12705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:04.697034 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:04.697038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:04.697308 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:04.701490 (+ 4182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:04.701500 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:04.701504 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:04.702007 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:04.702172 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:04.702759 (+ 587us) server_negotiation.cc:299] Negotiation successful
0926 23:31:04.703092 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13105,"thread_start_us":12997,"threads_started":1}
I20260926 23:31:04.710630 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:04.642296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38317 (local address 127.0.97.3:55673)
0926 23:31:04.656309 (+ 14013us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:04.656344 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:04.658471 (+ 2127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:04.659282 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:04.659287 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:04.666727 (+ 7440us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:04.668398 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:04.668432 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:04.669994 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:04.670002 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:04.672992 (+ 2990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:04.673010 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:04.674473 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:04.674480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:04.674668 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:04.675274 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:04.675301 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:04.679118 (+ 3817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:04.689068 (+ 9950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:04.689072 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:04.689076 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:04.696832 (+ 7756us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:04.700894 (+ 4062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:04.700901 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:04.700905 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:04.701232 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:04.702267 (+ 1035us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:04.702271 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:04.702437 (+ 166us) client_negotiation.cc:769] Sending connection context
0926 23:31:04.709728 (+ 7291us) client_negotiation.cc:240] Negotiation successful
0926 23:31:04.710180 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13216,"thread_start_us":11892,"threads_started":1}
I20260926 23:31:04.712159 7852 heartbeater.cc:347] Connected to a master server at 127.0.97.61:38317
I20260926 23:31:04.712320 7852 heartbeater.cc:464] Registering TS with master...
I20260926 23:31:04.713845 7852 heartbeater.cc:511] Master 127.0.97.61:38317 requested a full tablet report, sending...
Sep 26 23:31:04 dist-test-slave-gwfx krb5kdc[6540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465464, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260926 23:31:04.715763 7185 ts_manager.cc:194] Registered new tserver with Master: 29d55f61ce314ce2924917d0d54dbffb (127.0.97.3:35483)
I20260926 23:31:04.724529 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:04.641779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35661 (local address 127.0.97.60:43465)
0926 23:31:04.654599 (+ 12820us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:04.654608 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:04.654827 (+ 219us) server_negotiation.cc:407] Connection header received
0926 23:31:04.657372 (+ 2545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:04.657380 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:04.657486 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:04.657582 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:04.690025 (+ 32443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:04.691356 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:04.710178 (+ 18822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:04.710976 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:04.711146 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:04.716721 (+ 5575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:04.716730 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:04.716736 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:04.716788 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:04.721236 (+ 4448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:04.722185 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:04.722191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:04.722194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:04.722366 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:04.722754 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:04.722758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:04.722761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:04.723092 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:04.723215 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:04.723826 (+ 611us) server_negotiation.cc:299] Negotiation successful
0926 23:31:04.724094 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":12669,"thread_start_us":12722,"threads_started":1}
I20260926 23:31:04.725582 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:04.641425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43465 (local address 127.0.97.3:35661)
0926 23:31:04.654628 (+ 13203us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:04.654653 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:04.656906 (+ 2253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:04.658293 (+ 1387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:04.658302 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:04.659581 (+ 1279us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:04.689407 (+ 29826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:04.689423 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:04.694613 (+ 5190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:04.694620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:04.696141 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:04.696159 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:04.711444 (+ 15285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:04.711452 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:04.711635 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:04.713524 (+ 1889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:04.713548 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:04.716316 (+ 2768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:04.721474 (+ 5158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:04.721479 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:04.721484 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:04.722047 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:04.722493 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:04.722497 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:04.722500 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:04.722630 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:04.723310 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:04.723315 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:04.723484 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 23:31:04.724815 (+ 1331us) client_negotiation.cc:240] Negotiation successful
0926 23:31:04.725164 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13005,"mutex_wait_us":28871,"spinlock_wait_cycles":45184,"thread_start_us":13218,"threads_started":1}
I20260926 23:31:04.726509 7850 heartbeater.cc:347] Connected to a master server at 127.0.97.60:43465
I20260926 23:31:04.726675 7850 heartbeater.cc:464] Registering TS with master...
I20260926 23:31:04.727077 7850 heartbeater.cc:511] Master 127.0.97.60:43465 requested a full tablet report, sending...
I20260926 23:31:04.727501 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:04.660624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38770 (local address 127.0.97.60:43465)
0926 23:31:04.701998 (+ 41374us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:04.702005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:04.702028 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:31:04.702521 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:04.702527 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:04.702625 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:04.702823 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:04.705548 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:04.706848 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:04.710150 (+ 3302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:04.711255 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:04.711362 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:04.718169 (+ 6807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:04.718176 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:04.718181 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:04.718234 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:04.722894 (+ 4660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:04.724291 (+ 1397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:04.724297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:04.724301 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:04.724480 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:04.724964 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:04.724969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:04.724972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:04.725336 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:04.725613 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:04.726825 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0926 23:31:04.727084 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":337,"server-negotiator.queue_time_us":41194,"thread_start_us":46309,"threads_started":1}
I20260926 23:31:04.728225 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:04.660682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34344 (local address 127.0.97.61:38317)
0926 23:31:04.704250 (+ 43568us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:04.704256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:04.704274 (+ 18us) server_negotiation.cc:407] Connection header received
0926 23:31:04.704326 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:04.704332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:04.704407 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:04.706874 (+ 2467us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:04.707040 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:04.708202 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:04.710153 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:04.712414 (+ 2261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:04.712568 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:04.718231 (+ 5663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:04.718240 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:04.718245 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:04.718301 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:04.721912 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:04.724317 (+ 2405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:04.724324 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:04.724328 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:04.724513 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:04.725342 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:04.725348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:04.725352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:04.726047 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:04.726308 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:04.727340 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0926 23:31:04.727877 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43434,"thread_start_us":49196,"threads_started":1}
I20260926 23:31:04.728459 7382 ts_manager.cc:194] Registered new tserver with Master: 29d55f61ce314ce2924917d0d54dbffb (127.0.97.3:35483)
I20260926 23:31:04.733721 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:04.661131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40346 (local address 127.0.97.62:35009)
0926 23:31:04.687338 (+ 26207us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:04.687344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:04.709723 (+ 22379us) server_negotiation.cc:407] Connection header received
0926 23:31:04.710932 (+ 1209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:04.710938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:04.711042 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:04.711116 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:04.717938 (+ 6822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:04.719054 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:04.721030 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:04.721709 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:04.721825 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:04.726618 (+ 4793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:04.726627 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:04.726631 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:04.726723 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:04.730435 (+ 3712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:04.731230 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:04.731236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:04.731239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:04.731432 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:04.731877 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:04.731883 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:04.731885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:04.732341 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:04.732600 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:04.733114 (+ 514us) server_negotiation.cc:299] Negotiation successful
0926 23:31:04.733370 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26068,"thread_start_us":38295,"threads_started":1}
I20260926 23:31:04.738248 6585 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:31:04.745877 6585 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:31:04.810348 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:04.767915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39193 (local address 127.0.0.1:43900)
0926 23:31:04.800535 (+ 32620us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:04.800567 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:04.800761 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:04.801663 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:04.801668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:04.801700 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:31:04.802254 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:04.802268 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:04.804500 (+ 2232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:04.804506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:04.807723 (+ 3217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:04.807741 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:04.809221 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:04.809227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:04.809392 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:04.809425 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 23:31:04.809696 (+ 271us) client_negotiation.cc:240] Negotiation successful
0926 23:31:04.809867 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32416,"mutex_wait_us":107,"thread_start_us":32520,"threads_started":1}
I20260926 23:31:04.811523 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:04.768429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43900 (local address 127.0.97.1:39193)
0926 23:31:04.801033 (+ 32604us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:04.801042 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:04.801070 (+ 28us) server_negotiation.cc:407] Connection header received
0926 23:31:04.801167 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:04.801174 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:04.801424 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:04.801634 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:31:04.802641 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:04.804347 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:04.807957 (+ 3610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:04.809002 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:04.809133 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:04.809377 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:04.811007 (+ 1630us) server_negotiation.cc:299] Negotiation successful
0926 23:31:04.811217 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32416,"thread_start_us":31062,"threads_started":1}
I20260926 23:31:04.813272 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:04.769429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51372 (local address 127.0.97.3:35483)
0926 23:31:04.794047 (+ 24618us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:04.794059 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:04.800844 (+ 6785us) server_negotiation.cc:407] Connection header received
0926 23:31:04.801012 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:04.801022 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:04.801277 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:04.801483 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:31:04.802128 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:04.804114 (+ 1986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:04.809620 (+ 5506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:04.812016 (+ 2396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:04.812166 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:04.812477 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:04.812639 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 23:31:04.812961 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24389,"thread_start_us":23649,"threads_started":1}
I20260926 23:31:04.813448 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:04.768699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35483 (local address 127.0.0.1:51372)
0926 23:31:04.800691 (+ 31992us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:04.800705 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:04.800831 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:04.801490 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:04.801494 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:04.801520 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:31:04.801944 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:04.801958 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:04.804500 (+ 2542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:04.804506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:04.807856 (+ 3350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:04.807867 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:04.812259 (+ 4392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:04.812266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:04.812445 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:04.812487 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 23:31:04.812582 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 23:31:04.812735 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31872,"mutex_wait_us":144,"thread_start_us":31028,"threads_started":1}
I20260926 23:31:04.817646 7495 tablet_service.cc:1467] Processing CreateTablet for tablet 04acebac68694404a3d6f124dae7791e (DEFAULT_TABLE table=test-table [id=15fd0556f77c430997dee11557573142]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:31:04.819957 7495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04acebac68694404a3d6f124dae7791e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:31:04.819561 7785 tablet_service.cc:1467] Processing CreateTablet for tablet 04acebac68694404a3d6f124dae7791e (DEFAULT_TABLE table=test-table [id=15fd0556f77c430997dee11557573142]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:31:04.821588 7785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04acebac68694404a3d6f124dae7791e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:31:04.821666 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:04.767870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41070 (local address 127.0.97.2:37943)
0926 23:31:04.811384 (+ 43514us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:04.811396 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:04.811440 (+ 44us) server_negotiation.cc:407] Connection header received
0926 23:31:04.812178 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:04.812192 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:04.812593 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:04.813738 (+ 1145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:31:04.813984 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:04.815809 (+ 1825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:04.819443 (+ 3634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:04.820628 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:04.820769 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:04.821025 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:04.821244 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 23:31:04.821416 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43295,"thread_start_us":43159,"threads_started":1}
I20260926 23:31:04.821933 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:04.767572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37943 (local address 127.0.0.1:41070)
0926 23:31:04.800084 (+ 32512us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:04.800102 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:04.800228 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:04.812906 (+ 12678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:04.812912 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:04.812955 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:31:04.813485 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:04.813503 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:04.816049 (+ 2546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:04.816056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:04.818942 (+ 2886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:04.818960 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:04.820865 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:04.820872 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:04.821058 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:04.821094 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 23:31:04.821187 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 23:31:04.821333 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32334,"thread_start_us":33092,"threads_started":1}
I20260926 23:31:04.825505 7644 tablet_service.cc:1467] Processing CreateTablet for tablet 04acebac68694404a3d6f124dae7791e (DEFAULT_TABLE table=test-table [id=15fd0556f77c430997dee11557573142]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:31:04.827181 7644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04acebac68694404a3d6f124dae7791e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:31:04.842346 7887 tablet_bootstrap.cc:492] T 04acebac68694404a3d6f124dae7791e P 29d55f61ce314ce2924917d0d54dbffb: Bootstrap starting.
I20260926 23:31:04.844563 7888 tablet_bootstrap.cc:492] T 04acebac68694404a3d6f124dae7791e P b228fb1eb3574b03868c488ff3df2172: Bootstrap starting.
I20260926 23:31:04.844558 7886 tablet_bootstrap.cc:492] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e: Bootstrap starting.
I20260926 23:31:04.849474 7886 tablet_bootstrap.cc:654] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e: Neither blocks nor log segments found. Creating new log.
I20260926 23:31:04.849474 7888 tablet_bootstrap.cc:654] T 04acebac68694404a3d6f124dae7791e P b228fb1eb3574b03868c488ff3df2172: Neither blocks nor log segments found. Creating new log.
I20260926 23:31:04.850975 7887 tablet_bootstrap.cc:654] T 04acebac68694404a3d6f124dae7791e P 29d55f61ce314ce2924917d0d54dbffb: Neither blocks nor log segments found. Creating new log.
I20260926 23:31:04.852072 7888 log.cc:824] T 04acebac68694404a3d6f124dae7791e P b228fb1eb3574b03868c488ff3df2172: Log is configured to *not* fsync() on all Append() calls
I20260926 23:31:04.852072 7886 log.cc:824] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e: Log is configured to *not* fsync() on all Append() calls
I20260926 23:31:04.852667 7887 log.cc:824] T 04acebac68694404a3d6f124dae7791e P 29d55f61ce314ce2924917d0d54dbffb: Log is configured to *not* fsync() on all Append() calls
I20260926 23:31:04.863207 7886 tablet_bootstrap.cc:492] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e: No bootstrap required, opened a new log
I20260926 23:31:04.863484 7887 tablet_bootstrap.cc:492] T 04acebac68694404a3d6f124dae7791e P 29d55f61ce314ce2924917d0d54dbffb: No bootstrap required, opened a new log
I20260926 23:31:04.863520 7888 tablet_bootstrap.cc:492] T 04acebac68694404a3d6f124dae7791e P b228fb1eb3574b03868c488ff3df2172: No bootstrap required, opened a new log
I20260926 23:31:04.863653 7886 ts_tablet_manager.cc:1397] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.006s
I20260926 23:31:04.863801 7888 ts_tablet_manager.cc:1397] T 04acebac68694404a3d6f124dae7791e P b228fb1eb3574b03868c488ff3df2172: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.007s
I20260926 23:31:04.863837 7887 ts_tablet_manager.cc:1397] T 04acebac68694404a3d6f124dae7791e P 29d55f61ce314ce2924917d0d54dbffb: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.004s
I20260926 23:31:04.873677 7887 raft_consensus.cc:348] T 04acebac68694404a3d6f124dae7791e P 29d55f61ce314ce2924917d0d54dbffb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b228fb1eb3574b03868c488ff3df2172" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37943 } } peers { permanent_uuid: "152cd7b5a33b478695dbc5d5aba24d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "29d55f61ce314ce2924917d0d54dbffb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35483 } }
I20260926 23:31:04.874290 7887 raft_consensus.cc:374] T 04acebac68694404a3d6f124dae7791e P 29d55f61ce314ce2924917d0d54dbffb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:31:04.874398 7887 raft_consensus.cc:729] T 04acebac68694404a3d6f124dae7791e P 29d55f61ce314ce2924917d0d54dbffb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d55f61ce314ce2924917d0d54dbffb, State: Initialized, Role: FOLLOWER
I20260926 23:31:04.875116 7887 consensus_queue.cc:260] T 04acebac68694404a3d6f124dae7791e P 29d55f61ce314ce2924917d0d54dbffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b228fb1eb3574b03868c488ff3df2172" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37943 } } peers { permanent_uuid: "152cd7b5a33b478695dbc5d5aba24d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "29d55f61ce314ce2924917d0d54dbffb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35483 } }
I20260926 23:31:04.875664 7886 raft_consensus.cc:348] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b228fb1eb3574b03868c488ff3df2172" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37943 } } peers { permanent_uuid: "152cd7b5a33b478695dbc5d5aba24d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "29d55f61ce314ce2924917d0d54dbffb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35483 } }
I20260926 23:31:04.876273 7886 raft_consensus.cc:374] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:31:04.876365 7886 raft_consensus.cc:729] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 152cd7b5a33b478695dbc5d5aba24d1e, State: Initialized, Role: FOLLOWER
I20260926 23:31:04.876096 7888 raft_consensus.cc:348] T 04acebac68694404a3d6f124dae7791e P b228fb1eb3574b03868c488ff3df2172 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b228fb1eb3574b03868c488ff3df2172" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37943 } } peers { permanent_uuid: "152cd7b5a33b478695dbc5d5aba24d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "29d55f61ce314ce2924917d0d54dbffb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35483 } }
I20260926 23:31:04.876513 7888 raft_consensus.cc:374] T 04acebac68694404a3d6f124dae7791e P b228fb1eb3574b03868c488ff3df2172 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:31:04.876588 7888 raft_consensus.cc:729] T 04acebac68694404a3d6f124dae7791e P b228fb1eb3574b03868c488ff3df2172 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b228fb1eb3574b03868c488ff3df2172, State: Initialized, Role: FOLLOWER
I20260926 23:31:04.877070 7887 ts_tablet_manager.cc:1428] T 04acebac68694404a3d6f124dae7791e P 29d55f61ce314ce2924917d0d54dbffb: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260926 23:31:04.877224 7886 consensus_queue.cc:260] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b228fb1eb3574b03868c488ff3df2172" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37943 } } peers { permanent_uuid: "152cd7b5a33b478695dbc5d5aba24d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "29d55f61ce314ce2924917d0d54dbffb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35483 } }
I20260926 23:31:04.877523 7851 heartbeater.cc:503] Master 127.0.97.62:35009 was elected leader, sending a full tablet report...
I20260926 23:31:04.878413 7888 consensus_queue.cc:260] T 04acebac68694404a3d6f124dae7791e P b228fb1eb3574b03868c488ff3df2172 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b228fb1eb3574b03868c488ff3df2172" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37943 } } peers { permanent_uuid: "152cd7b5a33b478695dbc5d5aba24d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "29d55f61ce314ce2924917d0d54dbffb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35483 } }
I20260926 23:31:04.886916 7888 ts_tablet_manager.cc:1428] T 04acebac68694404a3d6f124dae7791e P b228fb1eb3574b03868c488ff3df2172: Time spent starting tablet: real 0.023s user 0.019s sys 0.002s
I20260926 23:31:04.886919 7886 ts_tablet_manager.cc:1428] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e: Time spent starting tablet: real 0.023s user 0.006s sys 0.018s
I20260926 23:31:04.887359 7710 heartbeater.cc:503] Master 127.0.97.62:35009 was elected leader, sending a full tablet report...
W20260926 23:31:04.908532 7853 tablet.cc:2369] T 04acebac68694404a3d6f124dae7791e P 29d55f61ce314ce2924917d0d54dbffb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:31:04.952862 7712 tablet.cc:2369] T 04acebac68694404a3d6f124dae7791e P b228fb1eb3574b03868c488ff3df2172: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:31:05.019263 7563 tablet.cc:2369] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:31:05.043495 7893 raft_consensus.cc:482] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:31:05.043860 7893 raft_consensus.cc:504] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b228fb1eb3574b03868c488ff3df2172" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37943 } } peers { permanent_uuid: "152cd7b5a33b478695dbc5d5aba24d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "29d55f61ce314ce2924917d0d54dbffb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35483 } }
I20260926 23:31:05.045861 7893 leader_election.cc:288] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b228fb1eb3574b03868c488ff3df2172 (127.0.97.2:37943), 29d55f61ce314ce2924917d0d54dbffb (127.0.97.3:35483)
I20260926 23:31:05.054180 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:05.046376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47263 (local address 127.0.97.3:35483)
0926 23:31:05.046636 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:05.046643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:05.047218 (+ 575us) server_negotiation.cc:407] Connection header received
0926 23:31:05.047436 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:05.047441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:05.047507 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:05.047586 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:31:05.048156 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:05.048930 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:05.052188 (+ 3258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:05.053068 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:05.053204 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:05.053335 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:05.053736 (+ 401us) server_negotiation.cc:299] Negotiation successful
0926 23:31:05.053896 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 23:31:05.054180 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:05.046460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51505 (local address 127.0.97.2:37943)
0926 23:31:05.046752 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:05.046759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:05.047396 (+ 637us) server_negotiation.cc:407] Connection header received
0926 23:31:05.047585 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:05.047589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:05.047654 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:05.047733 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:31:05.048361 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:05.049064 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:05.052199 (+ 3135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:05.053075 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:05.053204 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:05.053335 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:05.053736 (+ 401us) server_negotiation.cc:299] Negotiation successful
0926 23:31:05.053890 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 23:31:05.054342 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:05.046196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35483 (local address 127.0.97.1:47263)
0926 23:31:05.047045 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:05.047071 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:05.047285 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:05.047611 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:05.047615 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:05.047635 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:31:05.048026 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:05.048041 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:05.049067 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:05.049072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:05.051999 (+ 2927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:05.052013 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:05.053336 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:05.053343 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:05.053542 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:05.053594 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 23:31:05.053686 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 23:31:05.053874 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":120,"thread_start_us":500,"threads_started":1}
I20260926 23:31:05.054855 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:05.046482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37943 (local address 127.0.97.1:51505)
0926 23:31:05.047285 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:05.047299 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:05.047449 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:05.047767 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:05.047771 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:05.047793 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:31:05.048198 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:05.048207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:05.049199 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:05.049204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:05.051986 (+ 2782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:05.052007 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:05.053372 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:05.053379 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:05.053548 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:05.053599 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 23:31:05.053686 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 23:31:05.053882 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":154,"thread_start_us":523,"threads_started":1}
I20260926 23:31:05.055495 7805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04acebac68694404a3d6f124dae7791e" candidate_uuid: "152cd7b5a33b478695dbc5d5aba24d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d55f61ce314ce2924917d0d54dbffb" is_pre_election: true
I20260926 23:31:05.055928 7664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04acebac68694404a3d6f124dae7791e" candidate_uuid: "152cd7b5a33b478695dbc5d5aba24d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b228fb1eb3574b03868c488ff3df2172" is_pre_election: true
I20260926 23:31:05.056129 7805 raft_consensus.cc:2454] T 04acebac68694404a3d6f124dae7791e P 29d55f61ce314ce2924917d0d54dbffb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 152cd7b5a33b478695dbc5d5aba24d1e in term 0.
I20260926 23:31:05.056404 7664 raft_consensus.cc:2454] T 04acebac68694404a3d6f124dae7791e P b228fb1eb3574b03868c488ff3df2172 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 152cd7b5a33b478695dbc5d5aba24d1e in term 0.
I20260926 23:31:05.056928 7450 leader_election.cc:302] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 152cd7b5a33b478695dbc5d5aba24d1e, 29d55f61ce314ce2924917d0d54dbffb; no voters:
I20260926 23:31:05.057368 7893 raft_consensus.cc:2781] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:31:05.057520 7893 raft_consensus.cc:482] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:31:05.057631 7893 raft_consensus.cc:3037] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:31:05.060322 7893 raft_consensus.cc:504] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b228fb1eb3574b03868c488ff3df2172" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37943 } } peers { permanent_uuid: "152cd7b5a33b478695dbc5d5aba24d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "29d55f61ce314ce2924917d0d54dbffb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35483 } }
I20260926 23:31:05.061018 7893 leader_election.cc:288] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e [CANDIDATE]: Term 1 election: Requested vote from peers b228fb1eb3574b03868c488ff3df2172 (127.0.97.2:37943), 29d55f61ce314ce2924917d0d54dbffb (127.0.97.3:35483)
I20260926 23:31:05.061707 7805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04acebac68694404a3d6f124dae7791e" candidate_uuid: "152cd7b5a33b478695dbc5d5aba24d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d55f61ce314ce2924917d0d54dbffb"
I20260926 23:31:05.061719 7664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04acebac68694404a3d6f124dae7791e" candidate_uuid: "152cd7b5a33b478695dbc5d5aba24d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b228fb1eb3574b03868c488ff3df2172"
I20260926 23:31:05.061955 7805 raft_consensus.cc:3037] T 04acebac68694404a3d6f124dae7791e P 29d55f61ce314ce2924917d0d54dbffb [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:31:05.061972 7664 raft_consensus.cc:3037] T 04acebac68694404a3d6f124dae7791e P b228fb1eb3574b03868c488ff3df2172 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:31:05.065147 7664 raft_consensus.cc:2454] T 04acebac68694404a3d6f124dae7791e P b228fb1eb3574b03868c488ff3df2172 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 152cd7b5a33b478695dbc5d5aba24d1e in term 1.
I20260926 23:31:05.065692 7449 leader_election.cc:302] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 152cd7b5a33b478695dbc5d5aba24d1e, b228fb1eb3574b03868c488ff3df2172; no voters:
I20260926 23:31:05.065904 7805 raft_consensus.cc:2454] T 04acebac68694404a3d6f124dae7791e P 29d55f61ce314ce2924917d0d54dbffb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 152cd7b5a33b478695dbc5d5aba24d1e in term 1.
I20260926 23:31:05.066219 7893 raft_consensus.cc:2781] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:31:05.067081 7893 raft_consensus.cc:686] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e [term 1 LEADER]: Becoming Leader. State: Replica: 152cd7b5a33b478695dbc5d5aba24d1e, State: Running, Role: LEADER
I20260926 23:31:05.067945 7893 consensus_queue.cc:237] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b228fb1eb3574b03868c488ff3df2172" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37943 } } peers { permanent_uuid: "152cd7b5a33b478695dbc5d5aba24d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "29d55f61ce314ce2924917d0d54dbffb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35483 } }
I20260926 23:31:05.075073 6585 catalog_manager.cc:5495] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 152cd7b5a33b478695dbc5d5aba24d1e (127.0.97.1). New cstate: current_term: 1 leader_uuid: "152cd7b5a33b478695dbc5d5aba24d1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b228fb1eb3574b03868c488ff3df2172" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "152cd7b5a33b478695dbc5d5aba24d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29d55f61ce314ce2924917d0d54dbffb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35483 } health_report { overall_health: UNKNOWN } } }
I20260926 23:31:05.152946 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:05.146769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43908 (local address 127.0.97.1:39193)
0926 23:31:05.146986 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:05.146993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:05.147030 (+ 37us) server_negotiation.cc:407] Connection header received
0926 23:31:05.147154 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:05.147158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:05.147246 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:05.147419 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:31:05.148038 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:05.148878 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:05.150437 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:05.150905 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:05.151024 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:05.151593 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:31:05.152284 (+ 691us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:31:05.152450 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:05.152538 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 23:31:05.152639 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 23:31:05.164441 7805 raft_consensus.cc:1261] T 04acebac68694404a3d6f124dae7791e P 29d55f61ce314ce2924917d0d54dbffb [term 1 FOLLOWER]: Refusing update from remote peer 152cd7b5a33b478695dbc5d5aba24d1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:31:05.164444 7664 raft_consensus.cc:1261] T 04acebac68694404a3d6f124dae7791e P b228fb1eb3574b03868c488ff3df2172 [term 1 FOLLOWER]: Refusing update from remote peer 152cd7b5a33b478695dbc5d5aba24d1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:31:05.165696 7893 consensus_queue.cc:1035] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d55f61ce314ce2924917d0d54dbffb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:31:05.166177 7897 consensus_queue.cc:1035] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b228fb1eb3574b03868c488ff3df2172" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:31:05.212672 6584 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:31:05.213114 6584 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:31:05.221156 6584 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e49775de4d9427ebcdcf5e2303fc63c: Sending DeleteTablet for 3 replicas of tablet 04acebac68694404a3d6f124dae7791e
I20260926 23:31:05.222587 7644 tablet_service.cc:1514] Processing DeleteTablet for tablet 04acebac68694404a3d6f124dae7791e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:31:05 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41070
I20260926 23:31:05.222589 7495 tablet_service.cc:1514] Processing DeleteTablet for tablet 04acebac68694404a3d6f124dae7791e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:31:05 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43900
I20260926 23:31:05.224349 7785 tablet_service.cc:1514] Processing DeleteTablet for tablet 04acebac68694404a3d6f124dae7791e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:31:05 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51372
I20260926 23:31:05.236207 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 6553
I20260926 23:31:05.237840 6615 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:31:05.242601 7912 tablet_replica.cc:330] T 04acebac68694404a3d6f124dae7791e P b228fb1eb3574b03868c488ff3df2172: stopping tablet replica
I20260926 23:31:05.243170 7912 raft_consensus.cc:2229] T 04acebac68694404a3d6f124dae7791e P b228fb1eb3574b03868c488ff3df2172 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:31:05.243754 7912 raft_consensus.cc:2258] T 04acebac68694404a3d6f124dae7791e P b228fb1eb3574b03868c488ff3df2172 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:31:05.249377 7912 ts_tablet_manager.cc:1905] T 04acebac68694404a3d6f124dae7791e P b228fb1eb3574b03868c488ff3df2172: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:31:05.255390 7911 tablet_replica.cc:330] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e: stopping tablet replica
I20260926 23:31:05.256016 7911 raft_consensus.cc:2229] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:31:05.256846 7911 raft_consensus.cc:2258] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:31:05.259922 7911 ts_tablet_manager.cc:1905] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:31:05.260668 7912 ts_tablet_manager.cc:1918] T 04acebac68694404a3d6f124dae7791e P b228fb1eb3574b03868c488ff3df2172: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:31:05.260861 7912 log.cc:1196] T 04acebac68694404a3d6f124dae7791e P b228fb1eb3574b03868c488ff3df2172: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/wal/wals/04acebac68694404a3d6f124dae7791e
I20260926 23:31:05.261443 7912 ts_tablet_manager.cc:1939] T 04acebac68694404a3d6f124dae7791e P b228fb1eb3574b03868c488ff3df2172: Deleting consensus metadata
I20260926 23:31:05.267318 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:05.227485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34352 (local address 127.0.97.61:38317)
0926 23:31:05.227705 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:05.227711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:05.235254 (+ 7543us) server_negotiation.cc:407] Connection header received
0926 23:31:05.236472 (+ 1218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:05.236480 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:05.236597 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:05.236685 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:05.244985 (+ 8300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:05.246327 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:05.248543 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:05.249215 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:05.249333 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:05.257777 (+ 8444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:05.257786 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:05.257791 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:05.257850 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:05.261928 (+ 4078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:05.263264 (+ 1336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:05.263270 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:05.263273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:05.263517 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:05.263969 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:05.263975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:05.263979 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:05.264453 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:05.264809 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:05.266481 (+ 1672us) server_negotiation.cc:299] Negotiation successful
0926 23:31:05.266845 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 23:31:05.271464 7913 tablet_replica.cc:330] T 04acebac68694404a3d6f124dae7791e P 29d55f61ce314ce2924917d0d54dbffb: stopping tablet replica
I20260926 23:31:05.273689 7913 raft_consensus.cc:2229] T 04acebac68694404a3d6f124dae7791e P 29d55f61ce314ce2924917d0d54dbffb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:31:05.274410 7913 raft_consensus.cc:2258] T 04acebac68694404a3d6f124dae7791e P 29d55f61ce314ce2924917d0d54dbffb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:31:05.280462 7911 ts_tablet_manager.cc:1918] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:31:05.282464 7913 ts_tablet_manager.cc:1905] T 04acebac68694404a3d6f124dae7791e P 29d55f61ce314ce2924917d0d54dbffb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:31:05.287153 7911 log.cc:1196] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/wal/wals/04acebac68694404a3d6f124dae7791e
I20260926 23:31:05.287886 7911 ts_tablet_manager.cc:1939] T 04acebac68694404a3d6f124dae7791e P 152cd7b5a33b478695dbc5d5aba24d1e: Deleting consensus metadata
I20260926 23:31:05.289402 7913 ts_tablet_manager.cc:1918] T 04acebac68694404a3d6f124dae7791e P 29d55f61ce314ce2924917d0d54dbffb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:31:05.289672 7913 log.cc:1196] T 04acebac68694404a3d6f124dae7791e P 29d55f61ce314ce2924917d0d54dbffb: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/wal/wals/04acebac68694404a3d6f124dae7791e
I20260926 23:31:05.290328 7913 ts_tablet_manager.cc:1939] T 04acebac68694404a3d6f124dae7791e P 29d55f61ce314ce2924917d0d54dbffb: Deleting consensus metadata
I20260926 23:31:05.308234 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:05.235387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38774 (local address 127.0.97.60:43465)
0926 23:31:05.279464 (+ 44077us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:05.279473 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:05.279504 (+ 31us) server_negotiation.cc:407] Connection header received
0926 23:31:05.280754 (+ 1250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:05.280760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:05.280919 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:05.290123 (+ 9204us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:05.290409 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:05.291531 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:05.294302 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:05.295009 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:05.295157 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:05.298902 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:05.298912 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:05.298919 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:05.298981 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:05.303243 (+ 4262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:05.304596 (+ 1353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:05.304603 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:05.304608 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:05.304819 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:05.305397 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:05.305403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:05.305407 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:05.305877 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:05.307510 (+ 1633us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:05.307668 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 23:31:05.307916 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43901,"thread_start_us":43741,"threads_started":1}
I20260926 23:31:05.477523 6571 catalog_manager.cc:4841] TS 29d55f61ce314ce2924917d0d54dbffb (127.0.97.3:35483): tablet 04acebac68694404a3d6f124dae7791e (table test-table [id=15fd0556f77c430997dee11557573142]) successfully deleted
I20260926 23:31:05.479004 6569 catalog_manager.cc:4841] TS 152cd7b5a33b478695dbc5d5aba24d1e (127.0.97.1:39193): tablet 04acebac68694404a3d6f124dae7791e (table test-table [id=15fd0556f77c430997dee11557573142]) successfully deleted
I20260926 23:31:05.479553 6570 catalog_manager.cc:4841] TS b228fb1eb3574b03868c488ff3df2172 (127.0.97.2:37943): tablet 04acebac68694404a3d6f124dae7791e (table test-table [id=15fd0556f77c430997dee11557573142]) successfully deleted
I20260926 23:31:05.479851 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 6553
I20260926 23:31:05.531266 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:05.507927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38786 (local address 127.0.97.60:43465)
0926 23:31:05.508604 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:05.508612 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:05.512051 (+ 3439us) server_negotiation.cc:407] Connection header received
0926 23:31:05.512687 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:05.512696 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:05.512842 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:05.512940 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:05.516766 (+ 3826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:05.517949 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:05.519580 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:05.520042 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:05.520180 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:05.523687 (+ 3507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:05.523694 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:05.523699 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:05.523755 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:05.527442 (+ 3687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:05.529114 (+ 1672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:05.529122 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:05.529124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:05.529283 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:05.529644 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:05.529648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:05.529650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:05.530011 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:05.530275 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:05.530618 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 23:31:05.530894 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479}
I20260926 23:31:05.534679 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:05.508310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34368 (local address 127.0.97.61:38317)
0926 23:31:05.508786 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:05.508793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:05.517411 (+ 8618us) server_negotiation.cc:407] Connection header received
0926 23:31:05.517688 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:05.517695 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:05.517794 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:05.517865 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:05.520392 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:05.521476 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:05.523640 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:05.524203 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:05.524325 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:05.527767 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:05.527773 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:05.527777 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:05.527817 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:05.531435 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:05.532461 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:05.532467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:05.532471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:05.532653 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:05.533020 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:05.533024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:05.533028 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:05.533445 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:05.533729 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:05.534032 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 23:31:05.534300 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327}
I20260926 23:31:06.193361 7924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:06.192467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.97.2:43259)
0926 23:31:06.193053 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:06.193192 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":286,"threads_started":1}
W20260926 23:31:06.193934 7710 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35009 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
I20260926 23:31:06.203792 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:06.202822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.97.1:32889)
0926 23:31:06.203371 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:06.203522 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":273,"threads_started":1}
W20260926 23:31:06.204337 7561 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35009 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
I20260926 23:31:06.206231 7926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:06.205368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.97.3:52361)
0926 23:31:06.205916 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:06.206043 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":288,"threads_started":1}
W20260926 23:31:06.206717 7851 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35009 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
I20260926 23:31:06.722092 7927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e49775de4d9427ebcdcf5e2303fc63c)
I20260926 23:31:06.722205 7928 raft_consensus.cc:482] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e49775de4d9427ebcdcf5e2303fc63c)
I20260926 23:31:06.722385 7927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: false } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: false } }
I20260926 23:31:06.722573 7928 raft_consensus.cc:504] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: false } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: false } }
I20260926 23:31:06.723376 7927 leader_election.cc:288] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e49775de4d9427ebcdcf5e2303fc63c (127.0.97.62:35009), 002fa788f2fe4d8e83c0e20cec0b7b80 (127.0.97.61:38317)
I20260926 23:31:06.724730 7928 leader_election.cc:288] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e49775de4d9427ebcdcf5e2303fc63c (127.0.97.62:35009), 8154d80de9b846d7a1fc101eb64b28b8 (127.0.97.60:43465)
I20260926 23:31:06.726989 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:06.723670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.0.1:33394)
0926 23:31:06.726349 (+ 2679us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:06.726580 (+ 231us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2559,"thread_start_us":2416,"threads_started":1}
W20260926 23:31:06.727391 7368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
I20260926 23:31:06.728470 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:06.725600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.0.1:33400)
0926 23:31:06.727920 (+ 2320us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:06.728117 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2193,"thread_start_us":2276,"threads_started":1}
W20260926 23:31:06.728756 7172 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
I20260926 23:31:06.730597 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:06.729949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.0.1:33406)
0926 23:31:06.730174 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:06.730357 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 23:31:06.730894 7368 leader_election.cc:334] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e49775de4d9427ebcdcf5e2303fc63c (127.0.97.62:35009): Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
I20260926 23:31:06.731997 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:06.731488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.0.1:33412)
0926 23:31:06.731670 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:06.731787 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 23:31:06.732272 7172 leader_election.cc:334] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e49775de4d9427ebcdcf5e2303fc63c (127.0.97.62:35009): Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
I20260926 23:31:06.735755 7931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:06.723857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38442 (local address 127.0.97.61:38317)
0926 23:31:06.729053 (+ 5196us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:06.729060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:06.729080 (+ 20us) server_negotiation.cc:407] Connection header received
0926 23:31:06.729153 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:06.729158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:06.729238 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:06.729973 (+ 735us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:31:06.730178 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:06.731644 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:06.733585 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:06.734637 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:06.734814 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:06.734922 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:06.735225 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 23:31:06.735376 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5075,"thread_start_us":4869,"threads_started":1}
I20260926 23:31:06.735791 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:06.723749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38317 (local address 127.0.0.1:38442)
0926 23:31:06.727249 (+ 3500us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:06.727266 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:06.727411 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:06.729418 (+ 2007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:06.729425 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:06.729479 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:31:06.729830 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:06.729846 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:06.731794 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:06.731798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:06.733399 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:06.733414 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:06.734855 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:06.734861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:06.735046 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:06.735080 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 23:31:06.735188 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 23:31:06.735376 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3416,"thread_start_us":1395,"threads_started":1}
I20260926 23:31:06.736601 7194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8154d80de9b846d7a1fc101eb64b28b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" is_pre_election: true
I20260926 23:31:06.736966 7194 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8154d80de9b846d7a1fc101eb64b28b8 in term 1.
I20260926 23:31:06.737426 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:06.725924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47580 (local address 127.0.97.60:43465)
0926 23:31:06.730360 (+ 4436us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:06.730368 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:06.730393 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:31:06.730490 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:06.730495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:06.730587 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:06.731351 (+ 764us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:31:06.731708 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:06.732957 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:06.734920 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:06.736210 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:06.736335 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:06.736451 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:06.736815 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 23:31:06.736973 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":4272,"thread_start_us":4125,"threads_started":1}
I20260926 23:31:06.737432 7932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:06.724952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43465 (local address 127.0.0.1:47580)
0926 23:31:06.725600 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:06.725631 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:06.725817 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:06.730850 (+ 5033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:06.730856 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:06.730897 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:31:06.731467 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:06.731485 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:06.733120 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:06.733126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:06.734736 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:06.734756 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:06.736432 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:06.736439 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:06.736644 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:06.736673 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 23:31:06.736775 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 23:31:06.736953 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":3,"thread_start_us":441,"threads_started":1}
I20260926 23:31:06.737483 7365 leader_election.cc:302] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 002fa788f2fe4d8e83c0e20cec0b7b80, 8154d80de9b846d7a1fc101eb64b28b8; no voters: 4e49775de4d9427ebcdcf5e2303fc63c
I20260926 23:31:06.737917 7927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 23:31:06.738078 7927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4e49775de4d9427ebcdcf5e2303fc63c)
I20260926 23:31:06.738215 7927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [term 1 FOLLOWER]: Advancing to term 2
I20260926 23:31:06.738808 7391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8154d80de9b846d7a1fc101eb64b28b8" is_pre_election: true
I20260926 23:31:06.741258 7927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: false } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: false } }
I20260926 23:31:06.741763 7391 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 002fa788f2fe4d8e83c0e20cec0b7b80 in current term 2: Already voted for candidate 8154d80de9b846d7a1fc101eb64b28b8 in this term.
I20260926 23:31:06.741981 7927 leader_election.cc:288] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [CANDIDATE]: Term 2 election: Requested vote from peers 4e49775de4d9427ebcdcf5e2303fc63c (127.0.97.62:35009), 002fa788f2fe4d8e83c0e20cec0b7b80 (127.0.97.61:38317)
I20260926 23:31:06.742534 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:06.742091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.0.1:33436)
0926 23:31:06.742243 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:06.742328 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 23:31:06.743001 7171 leader_election.cc:302] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 002fa788f2fe4d8e83c0e20cec0b7b80; no voters: 4e49775de4d9427ebcdcf5e2303fc63c, 8154d80de9b846d7a1fc101eb64b28b8
I20260926 23:31:06.743481 7928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [term 1 FOLLOWER]: Advancing to term 2
I20260926 23:31:06.743714 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:06.743341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.0.1:33450)
0926 23:31:06.743461 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:06.743569 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 23:31:06.744037 7368 leader_election.cc:334] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4e49775de4d9427ebcdcf5e2303fc63c (127.0.97.62:35009): Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
I20260926 23:31:06.744124 7194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8154d80de9b846d7a1fc101eb64b28b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80"
I20260926 23:31:06.746170 7928 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 23:31:06.749306 7194 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8154d80de9b846d7a1fc101eb64b28b8 in term 2.
I20260926 23:31:06.750401 7365 leader_election.cc:302] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 002fa788f2fe4d8e83c0e20cec0b7b80, 8154d80de9b846d7a1fc101eb64b28b8; no voters: 4e49775de4d9427ebcdcf5e2303fc63c
I20260926 23:31:06.750849 7927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 23:31:06.751412 7927 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [term 2 LEADER]: Becoming Leader. State: Replica: 8154d80de9b846d7a1fc101eb64b28b8, State: Running, Role: LEADER
I20260926 23:31:06.751839 7927 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: false } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: false } }
I20260926 23:31:06.753794 7937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8154d80de9b846d7a1fc101eb64b28b8. Latest consensus state: current_term: 2 leader_uuid: "8154d80de9b846d7a1fc101eb64b28b8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: false } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: false } } }
I20260926 23:31:06.754333 7937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [sys.catalog]: This master's current role is: LEADER
I20260926 23:31:06.755268 7939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:31:06.759387 7939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:31:06.760300 7939 catalog_manager.cc:1259] Loaded cluster ID: b180128e00d44860b0f3bc07ab669d75
I20260926 23:31:06.760381 7939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:31:06.761492 7939 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:31:06.762291 7939 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8: Loaded TSK: 0
I20260926 23:31:06.763286 7939 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:31:06.799616 7381 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:31:06.807571 7381 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:31:06.813040 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:06.812476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.0.1:33472)
0926 23:31:06.812684 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:06.812842 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 23:31:06.813232 7194 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [term 2 FOLLOWER]: Refusing update from remote peer 8154d80de9b846d7a1fc101eb64b28b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 23:31:06.814185 7927 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 23:31:06.814453 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:06.813921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.0.1:33480)
0926 23:31:06.814126 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:06.814288 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260926 23:31:06.814677 7368 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 -> Peer 4e49775de4d9427ebcdcf5e2303fc63c (127.0.97.62:35009): Couldn't send request to peer 4e49775de4d9427ebcdcf5e2303fc63c. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 23:31:06.817914 7928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8154d80de9b846d7a1fc101eb64b28b8. Latest consensus state: current_term: 2 leader_uuid: "8154d80de9b846d7a1fc101eb64b28b8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: false } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: false } } }
I20260926 23:31:06.818363 7928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:31:06.819981 7927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "8154d80de9b846d7a1fc101eb64b28b8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: false } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: false } } }
I20260926 23:31:06.820475 7927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8 [sys.catalog]: This master's current role is: LEADER
I20260926 23:31:06.820753 7941 mvcc.cc:205] Tried to move back new op lower bound from 7333746552057237504 to 7333746551819542528. Current Snapshot: MvccSnapshot[applied={T|T < 7333746552057237504}]
I20260926 23:31:06.825126 7928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "8154d80de9b846d7a1fc101eb64b28b8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4e49775de4d9427ebcdcf5e2303fc63c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35009 } } peers { permanent_uuid: "002fa788f2fe4d8e83c0e20cec0b7b80" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38317 } attrs { promote: false } } peers { permanent_uuid: "8154d80de9b846d7a1fc101eb64b28b8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43465 } attrs { promote: false } } }
I20260926 23:31:06.825495 7928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 002fa788f2fe4d8e83c0e20cec0b7b80 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:31:06.837514 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:06.831419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39193 (local address 127.0.0.1:49584)
0926 23:31:06.831617 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:06.831649 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:06.831779 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:06.832460 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:06.832465 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:06.832500 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:31:06.832684 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:06.832694 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:06.833863 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:06.833869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:06.835305 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:06.835319 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:06.836758 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:06.836764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:06.836964 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:06.836989 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 23:31:06.837070 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 23:31:06.837209 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 23:31:06.837793 7946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:06.831614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49584 (local address 127.0.97.1:39193)
0926 23:31:06.832046 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:06.832051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:06.832072 (+ 21us) server_negotiation.cc:407] Connection header received
0926 23:31:06.832146 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:06.832150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:06.832285 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:06.832417 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:31:06.832813 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:06.833734 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:06.835802 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:06.836604 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:06.836725 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:06.836857 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:06.837107 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 23:31:06.837243 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":377,"threads_started":1}
I20260926 23:31:06.838855 7495 tablet_service.cc:1467] Processing CreateTablet for tablet 6c7bef98784846bcb726071e840076ec (DEFAULT_TABLE table=test-table [id=a7354ab369804135acaee6cf0999a2e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:31:06.839540 7495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c7bef98784846bcb726071e840076ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:31:06.849900 7947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:06.833133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37943 (local address 127.0.0.1:38554)
0926 23:31:06.833743 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:06.833764 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:06.835511 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:06.843638 (+ 8127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:06.843642 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:06.843660 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:31:06.844033 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:06.844042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:06.845452 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:06.845457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:06.847489 (+ 2032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:06.847503 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:06.849098 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:06.849103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:06.849279 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:06.849315 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 23:31:06.849385 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 23:31:06.849525 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":503,"threads_started":1}
I20260926 23:31:06.850988 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:06.833334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38554 (local address 127.0.97.2:37943)
0926 23:31:06.843334 (+ 10000us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:06.843340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:06.843357 (+ 17us) server_negotiation.cc:407] Connection header received
0926 23:31:06.843424 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:06.843428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:06.843496 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:06.844182 (+ 686us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:31:06.844331 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:06.845311 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:06.847706 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:06.848875 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:06.850349 (+ 1474us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:06.850472 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:06.850526 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 23:31:06.850680 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9882,"thread_start_us":17381,"threads_started":1}
I20260926 23:31:06.852102 7644 tablet_service.cc:1467] Processing CreateTablet for tablet 6c7bef98784846bcb726071e840076ec (DEFAULT_TABLE table=test-table [id=a7354ab369804135acaee6cf0999a2e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:31:06.852303 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:06.832357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35483 (local address 127.0.0.1:38712)
0926 23:31:06.832764 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:06.832781 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:06.832877 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:06.843588 (+ 10711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:06.843593 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:06.843626 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:31:06.844019 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:06.844031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:06.846945 (+ 2914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:06.846951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:06.848900 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:06.848914 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:06.851585 (+ 2671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:06.851592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:06.851757 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:06.851790 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 23:31:06.851856 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 23:31:06.851994 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":1}
I20260926 23:31:06.852761 7644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c7bef98784846bcb726071e840076ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:31:06.853106 7949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:06.833502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38712 (local address 127.0.97.3:35483)
0926 23:31:06.843166 (+ 9664us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:06.843173 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:06.843198 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:31:06.843284 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:06.843289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:06.843379 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:06.844183 (+ 804us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:31:06.844347 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:06.846785 (+ 2438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:06.849113 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:06.851340 (+ 2227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:06.852543 (+ 1203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:06.852686 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:06.852735 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 23:31:06.852849 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9516,"thread_start_us":9404,"threads_started":1}
I20260926 23:31:06.853827 7785 tablet_service.cc:1467] Processing CreateTablet for tablet 6c7bef98784846bcb726071e840076ec (DEFAULT_TABLE table=test-table [id=a7354ab369804135acaee6cf0999a2e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:31:06.853922 7950 tablet_bootstrap.cc:492] T 6c7bef98784846bcb726071e840076ec P 152cd7b5a33b478695dbc5d5aba24d1e: Bootstrap starting.
I20260926 23:31:06.854406 7785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c7bef98784846bcb726071e840076ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:31:06.856060 7950 tablet_bootstrap.cc:654] T 6c7bef98784846bcb726071e840076ec P 152cd7b5a33b478695dbc5d5aba24d1e: Neither blocks nor log segments found. Creating new log.
I20260926 23:31:06.864279 7952 tablet_bootstrap.cc:492] T 6c7bef98784846bcb726071e840076ec P b228fb1eb3574b03868c488ff3df2172: Bootstrap starting.
I20260926 23:31:06.867769 7952 tablet_bootstrap.cc:654] T 6c7bef98784846bcb726071e840076ec P b228fb1eb3574b03868c488ff3df2172: Neither blocks nor log segments found. Creating new log.
I20260926 23:31:06.868237 7953 tablet_bootstrap.cc:492] T 6c7bef98784846bcb726071e840076ec P 29d55f61ce314ce2924917d0d54dbffb: Bootstrap starting.
I20260926 23:31:06.869415 7950 tablet_bootstrap.cc:492] T 6c7bef98784846bcb726071e840076ec P 152cd7b5a33b478695dbc5d5aba24d1e: No bootstrap required, opened a new log
I20260926 23:31:06.869673 7950 ts_tablet_manager.cc:1397] T 6c7bef98784846bcb726071e840076ec P 152cd7b5a33b478695dbc5d5aba24d1e: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20260926 23:31:06.870740 7950 raft_consensus.cc:348] T 6c7bef98784846bcb726071e840076ec P 152cd7b5a33b478695dbc5d5aba24d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152cd7b5a33b478695dbc5d5aba24d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "29d55f61ce314ce2924917d0d54dbffb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35483 } } peers { permanent_uuid: "b228fb1eb3574b03868c488ff3df2172" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37943 } }
I20260926 23:31:06.871039 7950 raft_consensus.cc:374] T 6c7bef98784846bcb726071e840076ec P 152cd7b5a33b478695dbc5d5aba24d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:31:06.871138 7950 raft_consensus.cc:729] T 6c7bef98784846bcb726071e840076ec P 152cd7b5a33b478695dbc5d5aba24d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 152cd7b5a33b478695dbc5d5aba24d1e, State: Initialized, Role: FOLLOWER
I20260926 23:31:06.871444 7950 consensus_queue.cc:260] T 6c7bef98784846bcb726071e840076ec P 152cd7b5a33b478695dbc5d5aba24d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152cd7b5a33b478695dbc5d5aba24d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "29d55f61ce314ce2924917d0d54dbffb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35483 } } peers { permanent_uuid: "b228fb1eb3574b03868c488ff3df2172" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37943 } }
I20260926 23:31:06.872459 7953 tablet_bootstrap.cc:654] T 6c7bef98784846bcb726071e840076ec P 29d55f61ce314ce2924917d0d54dbffb: Neither blocks nor log segments found. Creating new log.
I20260926 23:31:06.876730 7952 tablet_bootstrap.cc:492] T 6c7bef98784846bcb726071e840076ec P b228fb1eb3574b03868c488ff3df2172: No bootstrap required, opened a new log
I20260926 23:31:06.876946 7952 ts_tablet_manager.cc:1397] T 6c7bef98784846bcb726071e840076ec P b228fb1eb3574b03868c488ff3df2172: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20260926 23:31:06.877686 7950 ts_tablet_manager.cc:1428] T 6c7bef98784846bcb726071e840076ec P 152cd7b5a33b478695dbc5d5aba24d1e: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260926 23:31:06.877910 7952 raft_consensus.cc:348] T 6c7bef98784846bcb726071e840076ec P b228fb1eb3574b03868c488ff3df2172 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152cd7b5a33b478695dbc5d5aba24d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "29d55f61ce314ce2924917d0d54dbffb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35483 } } peers { permanent_uuid: "b228fb1eb3574b03868c488ff3df2172" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37943 } }
I20260926 23:31:06.878189 7952 raft_consensus.cc:374] T 6c7bef98784846bcb726071e840076ec P b228fb1eb3574b03868c488ff3df2172 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:31:06.878278 7952 raft_consensus.cc:729] T 6c7bef98784846bcb726071e840076ec P b228fb1eb3574b03868c488ff3df2172 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b228fb1eb3574b03868c488ff3df2172, State: Initialized, Role: FOLLOWER
I20260926 23:31:06.878605 7952 consensus_queue.cc:260] T 6c7bef98784846bcb726071e840076ec P b228fb1eb3574b03868c488ff3df2172 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152cd7b5a33b478695dbc5d5aba24d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "29d55f61ce314ce2924917d0d54dbffb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35483 } } peers { permanent_uuid: "b228fb1eb3574b03868c488ff3df2172" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37943 } }
I20260926 23:31:06.883687 7560 heartbeater.cc:503] Master 127.0.97.60:43465 was elected leader, sending a full tablet report...
I20260926 23:31:06.890233 7952 ts_tablet_manager.cc:1428] T 6c7bef98784846bcb726071e840076ec P b228fb1eb3574b03868c488ff3df2172: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260926 23:31:06.890297 7953 tablet_bootstrap.cc:492] T 6c7bef98784846bcb726071e840076ec P 29d55f61ce314ce2924917d0d54dbffb: No bootstrap required, opened a new log
I20260926 23:31:06.890475 7953 ts_tablet_manager.cc:1397] T 6c7bef98784846bcb726071e840076ec P 29d55f61ce314ce2924917d0d54dbffb: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.005s
I20260926 23:31:06.891007 7957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:06.878461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.97.1:44039)
0926 23:31:06.890318 (+ 11857us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:06.890740 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11712,"thread_start_us":11578,"threads_started":1}
I20260926 23:31:06.891384 7953 raft_consensus.cc:348] T 6c7bef98784846bcb726071e840076ec P 29d55f61ce314ce2924917d0d54dbffb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152cd7b5a33b478695dbc5d5aba24d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "29d55f61ce314ce2924917d0d54dbffb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35483 } } peers { permanent_uuid: "b228fb1eb3574b03868c488ff3df2172" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37943 } }
I20260926 23:31:06.891637 7953 raft_consensus.cc:374] T 6c7bef98784846bcb726071e840076ec P 29d55f61ce314ce2924917d0d54dbffb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:31:06.891718 7953 raft_consensus.cc:729] T 6c7bef98784846bcb726071e840076ec P 29d55f61ce314ce2924917d0d54dbffb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d55f61ce314ce2924917d0d54dbffb, State: Initialized, Role: FOLLOWER
I20260926 23:31:06.892028 7953 consensus_queue.cc:260] T 6c7bef98784846bcb726071e840076ec P 29d55f61ce314ce2924917d0d54dbffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152cd7b5a33b478695dbc5d5aba24d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "29d55f61ce314ce2924917d0d54dbffb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35483 } } peers { permanent_uuid: "b228fb1eb3574b03868c488ff3df2172" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37943 } }
I20260926 23:31:06.893229 7709 heartbeater.cc:503] Master 127.0.97.60:43465 was elected leader, sending a full tablet report...
I20260926 23:31:06.894712 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:06.891486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.97.2:56541)
0926 23:31:06.894169 (+ 2683us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:06.894342 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2509,"thread_start_us":2345,"threads_started":1}
I20260926 23:31:06.894757 7957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:06.894240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.97.1:50339)
0926 23:31:06.894412 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:06.894519 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 23:31:06.896595 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:06.895537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.97.2:48965)
0926 23:31:06.896237 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:06.896375 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":577}
I20260926 23:31:06.897313 7953 ts_tablet_manager.cc:1428] T 6c7bef98784846bcb726071e840076ec P 29d55f61ce314ce2924917d0d54dbffb: Time spent starting tablet: real 0.007s user 0.000s sys 0.007s
I20260926 23:31:06.899240 7961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:06.897632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.97.3:45641)
0926 23:31:06.898798 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:06.898921 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":976,"thread_start_us":811,"threads_started":1}
I20260926 23:31:06.899682 7850 heartbeater.cc:503] Master 127.0.97.60:43465 was elected leader, sending a full tablet report...
I20260926 23:31:06.900424 7961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:06.899972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.97.3:43279)
0926 23:31:06.900180 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:06.900264 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260926 23:31:07.016382 7960 raft_consensus.cc:482] T 6c7bef98784846bcb726071e840076ec P 29d55f61ce314ce2924917d0d54dbffb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:31:07.016716 7960 raft_consensus.cc:504] T 6c7bef98784846bcb726071e840076ec P 29d55f61ce314ce2924917d0d54dbffb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152cd7b5a33b478695dbc5d5aba24d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "29d55f61ce314ce2924917d0d54dbffb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35483 } } peers { permanent_uuid: "b228fb1eb3574b03868c488ff3df2172" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37943 } }
I20260926 23:31:07.018821 7960 leader_election.cc:288] T 6c7bef98784846bcb726071e840076ec P 29d55f61ce314ce2924917d0d54dbffb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 152cd7b5a33b478695dbc5d5aba24d1e (127.0.97.1:39193), b228fb1eb3574b03868c488ff3df2172 (127.0.97.2:37943)
I20260926 23:31:07.027134 7946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:07.019281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39691 (local address 127.0.97.1:39193)
0926 23:31:07.019471 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:07.019477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:07.019502 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:31:07.019762 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:07.019767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:07.019850 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:07.019959 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:31:07.021044 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:07.021874 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:07.025077 (+ 3203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:07.025993 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:07.026111 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:07.026204 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:07.026711 (+ 507us) server_negotiation.cc:299] Negotiation successful
0926 23:31:07.026861 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 23:31:07.027176 7961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:07.019153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39193 (local address 127.0.97.3:39691)
0926 23:31:07.019348 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:07.019393 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:07.019582 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:07.019997 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:07.020000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:07.020033 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:31:07.020461 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:07.020474 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:07.022021 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:07.022027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:07.024851 (+ 2824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:07.024875 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:07.026285 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:07.026291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:07.026445 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:07.026477 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 23:31:07.026595 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 23:31:07.026802 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":233,"spinlock_wait_cycles":61952}
I20260926 23:31:07.028113 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7bef98784846bcb726071e840076ec" candidate_uuid: "29d55f61ce314ce2924917d0d54dbffb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "152cd7b5a33b478695dbc5d5aba24d1e" is_pre_election: true
I20260926 23:31:07.028602 7515 raft_consensus.cc:2454] T 6c7bef98784846bcb726071e840076ec P 152cd7b5a33b478695dbc5d5aba24d1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d55f61ce314ce2924917d0d54dbffb in term 0.
I20260926 23:31:07.029258 7738 leader_election.cc:302] T 6c7bef98784846bcb726071e840076ec P 29d55f61ce314ce2924917d0d54dbffb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 152cd7b5a33b478695dbc5d5aba24d1e, 29d55f61ce314ce2924917d0d54dbffb; no voters:
I20260926 23:31:07.029698 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:07.019281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58043 (local address 127.0.97.2:37943)
0926 23:31:07.019488 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:07.019494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:07.020061 (+ 567us) server_negotiation.cc:407] Connection header received
0926 23:31:07.020243 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:07.020249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:07.020325 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:07.020427 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:31:07.021070 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:07.021923 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:07.026305 (+ 4382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:07.027272 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:07.027405 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:07.027507 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:07.029259 (+ 1752us) server_negotiation.cc:299] Negotiation successful
0926 23:31:07.029432 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 23:31:07.029697 7962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:07.019153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37943 (local address 127.0.97.3:58043)
0926 23:31:07.019942 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:07.019965 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:07.020109 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:07.020499 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:07.020504 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:07.020534 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:31:07.020896 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:07.020908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:07.022057 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:07.022063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:07.024857 (+ 2794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:07.024875 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:07.027626 (+ 2751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:07.027632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:07.029077 (+ 1445us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:07.029107 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 23:31:07.029190 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 23:31:07.029323 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":350,"thread_start_us":320,"threads_started":1}
I20260926 23:31:07.029838 7960 raft_consensus.cc:2781] T 6c7bef98784846bcb726071e840076ec P 29d55f61ce314ce2924917d0d54dbffb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:31:07.029958 7960 raft_consensus.cc:482] T 6c7bef98784846bcb726071e840076ec P 29d55f61ce314ce2924917d0d54dbffb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:31:07.030050 7960 raft_consensus.cc:3037] T 6c7bef98784846bcb726071e840076ec P 29d55f61ce314ce2924917d0d54dbffb [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:31:07.030503 7664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7bef98784846bcb726071e840076ec" candidate_uuid: "29d55f61ce314ce2924917d0d54dbffb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b228fb1eb3574b03868c488ff3df2172" is_pre_election: true
I20260926 23:31:07.030819 7664 raft_consensus.cc:2454] T 6c7bef98784846bcb726071e840076ec P b228fb1eb3574b03868c488ff3df2172 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d55f61ce314ce2924917d0d54dbffb in term 0.
I20260926 23:31:07.033069 7960 raft_consensus.cc:504] T 6c7bef98784846bcb726071e840076ec P 29d55f61ce314ce2924917d0d54dbffb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152cd7b5a33b478695dbc5d5aba24d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "29d55f61ce314ce2924917d0d54dbffb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35483 } } peers { permanent_uuid: "b228fb1eb3574b03868c488ff3df2172" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37943 } }
I20260926 23:31:07.033761 7960 leader_election.cc:288] T 6c7bef98784846bcb726071e840076ec P 29d55f61ce314ce2924917d0d54dbffb [CANDIDATE]: Term 1 election: Requested vote from peers 152cd7b5a33b478695dbc5d5aba24d1e (127.0.97.1:39193), b228fb1eb3574b03868c488ff3df2172 (127.0.97.2:37943)
I20260926 23:31:07.034387 7664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7bef98784846bcb726071e840076ec" candidate_uuid: "29d55f61ce314ce2924917d0d54dbffb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b228fb1eb3574b03868c488ff3df2172"
I20260926 23:31:07.034409 7515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7bef98784846bcb726071e840076ec" candidate_uuid: "29d55f61ce314ce2924917d0d54dbffb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "152cd7b5a33b478695dbc5d5aba24d1e"
I20260926 23:31:07.034659 7515 raft_consensus.cc:3037] T 6c7bef98784846bcb726071e840076ec P 152cd7b5a33b478695dbc5d5aba24d1e [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:31:07.034711 7664 raft_consensus.cc:3037] T 6c7bef98784846bcb726071e840076ec P b228fb1eb3574b03868c488ff3df2172 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:31:07.037297 7664 raft_consensus.cc:2454] T 6c7bef98784846bcb726071e840076ec P b228fb1eb3574b03868c488ff3df2172 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d55f61ce314ce2924917d0d54dbffb in term 1.
I20260926 23:31:07.037313 7515 raft_consensus.cc:2454] T 6c7bef98784846bcb726071e840076ec P 152cd7b5a33b478695dbc5d5aba24d1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d55f61ce314ce2924917d0d54dbffb in term 1.
I20260926 23:31:07.037776 7738 leader_election.cc:302] T 6c7bef98784846bcb726071e840076ec P 29d55f61ce314ce2924917d0d54dbffb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 152cd7b5a33b478695dbc5d5aba24d1e, 29d55f61ce314ce2924917d0d54dbffb; no voters:
I20260926 23:31:07.038180 7960 raft_consensus.cc:2781] T 6c7bef98784846bcb726071e840076ec P 29d55f61ce314ce2924917d0d54dbffb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:31:07.038671 7960 raft_consensus.cc:686] T 6c7bef98784846bcb726071e840076ec P 29d55f61ce314ce2924917d0d54dbffb [term 1 LEADER]: Becoming Leader. State: Replica: 29d55f61ce314ce2924917d0d54dbffb, State: Running, Role: LEADER
I20260926 23:31:07.039183 7960 consensus_queue.cc:237] T 6c7bef98784846bcb726071e840076ec P 29d55f61ce314ce2924917d0d54dbffb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152cd7b5a33b478695dbc5d5aba24d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } } peers { permanent_uuid: "29d55f61ce314ce2924917d0d54dbffb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35483 } } peers { permanent_uuid: "b228fb1eb3574b03868c488ff3df2172" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37943 } }
I20260926 23:31:07.039819 7962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:07.039301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.97.3:45223)
0926 23:31:07.039487 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:07.039640 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 23:31:07.044031 7381 catalog_manager.cc:5495] T 6c7bef98784846bcb726071e840076ec P 29d55f61ce314ce2924917d0d54dbffb reported cstate change: term changed from 0 to 1, leader changed from <none> to 29d55f61ce314ce2924917d0d54dbffb (127.0.97.3). New cstate: current_term: 1 leader_uuid: "29d55f61ce314ce2924917d0d54dbffb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152cd7b5a33b478695dbc5d5aba24d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29d55f61ce314ce2924917d0d54dbffb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b228fb1eb3574b03868c488ff3df2172" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37943 } health_report { overall_health: UNKNOWN } } }
I20260926 23:31:07.086153 7949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:07.079637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38718 (local address 127.0.97.3:35483)
0926 23:31:07.079882 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:07.079890 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:07.079914 (+ 24us) server_negotiation.cc:407] Connection header received
0926 23:31:07.080221 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:07.080227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:07.080327 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:07.080508 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:31:07.081000 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:07.081847 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:07.083490 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:07.084016 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:07.084163 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:07.084686 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:31:07.085427 (+ 741us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:31:07.085593 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:07.085716 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 23:31:07.085826 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 23:31:07.091480 7664 raft_consensus.cc:1261] T 6c7bef98784846bcb726071e840076ec P b228fb1eb3574b03868c488ff3df2172 [term 1 FOLLOWER]: Refusing update from remote peer 29d55f61ce314ce2924917d0d54dbffb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:31:07.091866 7515 raft_consensus.cc:1261] T 6c7bef98784846bcb726071e840076ec P 152cd7b5a33b478695dbc5d5aba24d1e [term 1 FOLLOWER]: Refusing update from remote peer 29d55f61ce314ce2924917d0d54dbffb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:31:07.092372 7963 consensus_queue.cc:1035] T 6c7bef98784846bcb726071e840076ec P 29d55f61ce314ce2924917d0d54dbffb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b228fb1eb3574b03868c488ff3df2172" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:31:07.092833 7960 consensus_queue.cc:1035] T 6c7bef98784846bcb726071e840076ec P 29d55f61ce314ce2924917d0d54dbffb [LEADER]: Connected to new peer: Peer: permanent_uuid: "152cd7b5a33b478695dbc5d5aba24d1e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:31:07.095700 7957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:07.095105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.97.1:53907)
0926 23:31:07.095356 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:07.095490 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 23:31:07.095732 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:07.095038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.97.2:36783)
0926 23:31:07.095374 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:07.095517 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199}
I20260926 23:31:07.105304 7966 mvcc.cc:205] Tried to move back new op lower bound from 7333746553197723648 to 7333746552997236736. Current Snapshot: MvccSnapshot[applied={T|T < 7333746553197723648}]
I20260926 23:31:07.106181 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:07.105520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.97.2:46967)
0926 23:31:07.105804 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:07.105927 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260926 23:31:07.107107 7962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:07.106505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.97.3:52573)
0926 23:31:07.106813 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:07.106920 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197}
I20260926 23:31:07.111869 7962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:07.111231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.97.3:58891)
0926 23:31:07.111512 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:07.111645 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260926 23:31:07.116701 7957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:07.116044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.97.1:59391)
0926 23:31:07.116386 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:07.116491 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196}
I20260926 23:31:07.118064 7968 mvcc.cc:205] Tried to move back new op lower bound from 7333746553197723648 to 7333746552997236736. Current Snapshot: MvccSnapshot[applied={T|T < 7333746553197723648}]
I20260926 23:31:07.130919 7382 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:31:07.131326 7382 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:31:07.138589 7382 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8154d80de9b846d7a1fc101eb64b28b8: Sending DeleteTablet for 3 replicas of tablet 6c7bef98784846bcb726071e840076ec
I20260926 23:31:07.139832 7495 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c7bef98784846bcb726071e840076ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:31:07 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:49584
I20260926 23:31:07.139856 7785 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c7bef98784846bcb726071e840076ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:31:07 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:38712
I20260926 23:31:07.140702 7976 tablet_replica.cc:330] T 6c7bef98784846bcb726071e840076ec P 29d55f61ce314ce2924917d0d54dbffb: stopping tablet replica
I20260926 23:31:07.140967 7976 raft_consensus.cc:2229] T 6c7bef98784846bcb726071e840076ec P 29d55f61ce314ce2924917d0d54dbffb [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:31:07.141258 7644 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c7bef98784846bcb726071e840076ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:31:07 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:38554
I20260926 23:31:07.142172 7976 raft_consensus.cc:2258] T 6c7bef98784846bcb726071e840076ec P 29d55f61ce314ce2924917d0d54dbffb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:31:07.143224 7977 tablet_replica.cc:330] T 6c7bef98784846bcb726071e840076ec P 152cd7b5a33b478695dbc5d5aba24d1e: stopping tablet replica
I20260926 23:31:07.143515 7977 raft_consensus.cc:2229] T 6c7bef98784846bcb726071e840076ec P 152cd7b5a33b478695dbc5d5aba24d1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:31:07.143755 7976 ts_tablet_manager.cc:1905] T 6c7bef98784846bcb726071e840076ec P 29d55f61ce314ce2924917d0d54dbffb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:31:07.143997 7977 raft_consensus.cc:2258] T 6c7bef98784846bcb726071e840076ec P 152cd7b5a33b478695dbc5d5aba24d1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:31:07.145411 7977 ts_tablet_manager.cc:1905] T 6c7bef98784846bcb726071e840076ec P 152cd7b5a33b478695dbc5d5aba24d1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:31:07.148859 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 7369
I20260926 23:31:07.154858 7978 tablet_replica.cc:330] T 6c7bef98784846bcb726071e840076ec P b228fb1eb3574b03868c488ff3df2172: stopping tablet replica
I20260926 23:31:07.155155 7978 raft_consensus.cc:2229] T 6c7bef98784846bcb726071e840076ec P b228fb1eb3574b03868c488ff3df2172 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:31:07.155519 7978 raft_consensus.cc:2258] T 6c7bef98784846bcb726071e840076ec P b228fb1eb3574b03868c488ff3df2172 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:31:07.156755 7978 ts_tablet_manager.cc:1905] T 6c7bef98784846bcb726071e840076ec P b228fb1eb3574b03868c488ff3df2172: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:31:07.160410 7977 ts_tablet_manager.cc:1918] T 6c7bef98784846bcb726071e840076ec P 152cd7b5a33b478695dbc5d5aba24d1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:31:07.160663 7977 log.cc:1196] T 6c7bef98784846bcb726071e840076ec P 152cd7b5a33b478695dbc5d5aba24d1e: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-0/wal/wals/6c7bef98784846bcb726071e840076ec
I20260926 23:31:07.161001 7976 ts_tablet_manager.cc:1918] T 6c7bef98784846bcb726071e840076ec P 29d55f61ce314ce2924917d0d54dbffb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:31:07.161180 7976 log.cc:1196] T 6c7bef98784846bcb726071e840076ec P 29d55f61ce314ce2924917d0d54dbffb: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-2/wal/wals/6c7bef98784846bcb726071e840076ec
I20260926 23:31:07.161669 7976 ts_tablet_manager.cc:1939] T 6c7bef98784846bcb726071e840076ec P 29d55f61ce314ce2924917d0d54dbffb: Deleting consensus metadata
I20260926 23:31:07.161881 7977 ts_tablet_manager.cc:1939] T 6c7bef98784846bcb726071e840076ec P 152cd7b5a33b478695dbc5d5aba24d1e: Deleting consensus metadata
I20260926 23:31:07.163317 7367 catalog_manager.cc:4841] TS 29d55f61ce314ce2924917d0d54dbffb (127.0.97.3:35483): tablet 6c7bef98784846bcb726071e840076ec (table test-table [id=a7354ab369804135acaee6cf0999a2e3]) successfully deleted
I20260926 23:31:07.163343 7365 catalog_manager.cc:4841] TS 152cd7b5a33b478695dbc5d5aba24d1e (127.0.97.1:39193): tablet 6c7bef98784846bcb726071e840076ec (table test-table [id=a7354ab369804135acaee6cf0999a2e3]) successfully deleted
I20260926 23:31:07.163825 7978 ts_tablet_manager.cc:1918] T 6c7bef98784846bcb726071e840076ec P b228fb1eb3574b03868c488ff3df2172: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:31:07.163980 7978 log.cc:1196] T 6c7bef98784846bcb726071e840076ec P b228fb1eb3574b03868c488ff3df2172: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790465400694755-388-0/minicluster-data/ts-1/wal/wals/6c7bef98784846bcb726071e840076ec
I20260926 23:31:07.164530 7978 ts_tablet_manager.cc:1939] T 6c7bef98784846bcb726071e840076ec P b228fb1eb3574b03868c488ff3df2172: Deleting consensus metadata
I20260926 23:31:07.165802 7366 catalog_manager.cc:4841] TS b228fb1eb3574b03868c488ff3df2172 (127.0.97.2:37943): tablet 6c7bef98784846bcb726071e840076ec (table test-table [id=a7354ab369804135acaee6cf0999a2e3]) successfully deleted
I20260926 23:31:07.182436 7946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:07.151445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49594 (local address 127.0.97.1:39193)
0926 23:31:07.151686 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:07.151694 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:07.165453 (+ 13759us) server_negotiation.cc:407] Connection header received
0926 23:31:07.165780 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:07.165785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:07.165882 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:07.165985 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:07.167551 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:07.168728 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:07.170080 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:07.170853 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:07.170992 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:07.173956 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:07.173982 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:07.174011 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:07.174078 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:07.178669 (+ 4591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:07.179980 (+ 1311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:07.179990 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:07.179993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:07.180177 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:07.180645 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:07.180649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:07.180651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:07.181220 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:07.181494 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:07.181881 (+ 387us) server_negotiation.cc:299] Negotiation successful
0926 23:31:07.182103 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 23:31:07.183338 7555 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:31:07.278062 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:07.277449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.0.1:33486)
0926 23:31:07.277668 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:07.277817 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 23:31:07.279356 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:07.278943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.0.1:33490)
0926 23:31:07.279099 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:07.279189 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 23:31:07.317428 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 7369
I20260926 23:31:07.340991 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 7572
I20260926 23:31:07.358090 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:07.341989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38564 (local address 127.0.97.2:37943)
0926 23:31:07.342221 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:07.342227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:07.342247 (+ 20us) server_negotiation.cc:407] Connection header received
0926 23:31:07.342452 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:07.342456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:07.342529 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:07.342610 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:07.344436 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:07.345400 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:07.347344 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:07.347794 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:07.347902 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:07.351482 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:07.351501 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:07.351521 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:07.351574 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:07.354914 (+ 3340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:07.355973 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:07.355983 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:07.355985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:07.356138 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:07.356549 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:07.356553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:07.356555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:07.356977 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:07.357241 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:07.357553 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 23:31:07.357774 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 23:31:07.358950 7704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:31:07.460229 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 7572
I20260926 23:31:07.477694 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 7722
I20260926 23:31:07.494025 7949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:07.478604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38734 (local address 127.0.97.3:35483)
0926 23:31:07.478867 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:07.478873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:07.478894 (+ 21us) server_negotiation.cc:407] Connection header received
0926 23:31:07.479071 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:07.479077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:07.479159 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:07.479259 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:07.480714 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:07.481576 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:07.483142 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:07.483647 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:07.483770 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:07.486724 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:07.486749 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:07.486779 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:07.486832 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:07.490347 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:07.491315 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:07.491329 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:07.491331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:07.491530 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:07.492187 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:07.492191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:07.492193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:07.492673 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:07.492962 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:07.493405 (+ 443us) server_negotiation.cc:299] Negotiation successful
0926 23:31:07.493631 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260926 23:31:07.495029 7845 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:31:07.641705 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 7722
I20260926 23:31:07.663651 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 7048
I20260926 23:31:07.665067 7215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:31:07.742352 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:07.741820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.0.1:33506)
0926 23:31:07.742038 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:07.742142 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 23:31:07.743513 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:07.743150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35009 (local address 127.0.0.1:33520)
0926 23:31:07.743296 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:07.743375 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 23:31:07.796741 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 7048
I20260926 23:31:07.817080 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 7147
I20260926 23:31:07.818537 7412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:31:07.954824 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 7147
2026-09-26T23:31:07Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11014 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:31:08 dist-test-slave-gwfx krb5kdc[7987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:31:08 dist-test-slave-gwfx krb5kdc[7987](info): set up 2 sockets
Sep 26 23:31:08 dist-test-slave-gwfx krb5kdc[7987](info): commencing operation
krb5kdc: starting...
W20260926 23:31:08.999897 388 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.009s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:31:09 dist-test-slave-gwfx krb5kdc[7987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:31:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:31:09Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260926 23:31:09.266861 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40013
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:31:09.663702 8000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:31:09.664129 8000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:31:09.664274 8000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:31:09.672724 8000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:31:09.672883 8000 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 23:31:09.672935 8000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:31:09.673025 8000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:31:09.673089 8000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:31:09.673164 8000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:31:09.684607 8000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40013
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:31:09.686349 8000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:31:09.687991 8000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:09.702822 8006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:09.702975 8008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:09.702839 8005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:09.703501 8000 server_base.cc:1034] running on GCE node
I20260926 23:31:09.704735 8000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:09.706790 8000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:09.708103 8000 hybrid_clock.cc:648] HybridClock initialized: now 1790465469707999 us; error 78 us; skew 500 ppm
I20260926 23:31:09.708587 8000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:31:09 dist-test-slave-gwfx krb5kdc[7987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465469, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:31:09.712776 8000 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260926 23:31:09.714802 8000 webserver.cc:460] Webserver started at http://127.0.97.62:42579/ using document root <none> and password file <none>
I20260926 23:31:09.715776 8000 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:09.715905 8000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:09.716295 8000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:31:09.720300 8000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "55da4609224d44989fd4e860066c0076"
format_stamp: "Formatted at 2026-09-26 23:31:09 on dist-test-slave-gwfx"
server_key: "4f6a5d0159048e9d4f194e8fe75d8ceb"
server_key_iv: "ef0f85f28526cb0dca91d0e87c84ef09"
server_key_version: "encryptionkey@0"
I20260926 23:31:09.721397 8000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "55da4609224d44989fd4e860066c0076"
format_stamp: "Formatted at 2026-09-26 23:31:09 on dist-test-slave-gwfx"
server_key: "4f6a5d0159048e9d4f194e8fe75d8ceb"
server_key_iv: "ef0f85f28526cb0dca91d0e87c84ef09"
server_key_version: "encryptionkey@0"
I20260926 23:31:09.728588 8000 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260926 23:31:09.733631 8015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:09.735586 8000 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 23:31:09.735818 8000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "55da4609224d44989fd4e860066c0076"
format_stamp: "Formatted at 2026-09-26 23:31:09 on dist-test-slave-gwfx"
server_key: "4f6a5d0159048e9d4f194e8fe75d8ceb"
server_key_iv: "ef0f85f28526cb0dca91d0e87c84ef09"
server_key_version: "encryptionkey@0"
I20260926 23:31:09.736085 8000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:09.755600 8000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:09.761368 8000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:31:09.761706 8000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:09.786840 8000 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40013
I20260926 23:31:09.786854 8076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40013 every 8 connection(s)
I20260926 23:31:09.788641 8000 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:31:09.793867 8077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:31:09.795492 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 8000
I20260926 23:31:09.795724 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:31:09.796640 388 external_mini_cluster.cc:1428] Setting key 6540772b732ea4b7653364a5cd77a6c1
I20260926 23:31:09.812021 8077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55da4609224d44989fd4e860066c0076: Bootstrap starting.
Sep 26 23:31:09 dist-test-slave-gwfx krb5kdc[7987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 23:31:09.817765 8077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55da4609224d44989fd4e860066c0076: Neither blocks nor log segments found. Creating new log.
I20260926 23:31:09.819475 8077 log.cc:824] T 00000000000000000000000000000000 P 55da4609224d44989fd4e860066c0076: Log is configured to *not* fsync() on all Append() calls
I20260926 23:31:09.824337 8077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55da4609224d44989fd4e860066c0076: No bootstrap required, opened a new log
I20260926 23:31:09.829142 8080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:09.799335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56478 (local address 127.0.97.62:40013)
0926 23:31:09.800365 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:09.800382 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:09.800455 (+ 73us) server_negotiation.cc:407] Connection header received
0926 23:31:09.803057 (+ 2602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:09.803102 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:09.803879 (+ 777us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:09.804374 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:09.806482 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:09.808276 (+ 1794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:09.810430 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:09.811475 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:09.811698 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:09.817669 (+ 5971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:09.817682 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:09.817712 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:09.817787 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:09.822058 (+ 4271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:09.823483 (+ 1425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:09.823494 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:09.823505 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:09.823824 (+ 319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:09.824457 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:09.824463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:09.824466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:09.825479 (+ 1013us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:09.825834 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:09.826415 (+ 581us) server_negotiation.cc:299] Negotiation successful
0926 23:31:09.827138 (+ 723us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":748,"thread_start_us":439,"threads_started":1}
I20260926 23:31:09.834934 8077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55da4609224d44989fd4e860066c0076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55da4609224d44989fd4e860066c0076" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40013 } }
I20260926 23:31:09.835446 8077 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55da4609224d44989fd4e860066c0076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:31:09.835559 8077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55da4609224d44989fd4e860066c0076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55da4609224d44989fd4e860066c0076, State: Initialized, Role: FOLLOWER
I20260926 23:31:09.836287 8077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55da4609224d44989fd4e860066c0076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55da4609224d44989fd4e860066c0076" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40013 } }
I20260926 23:31:09.836545 8077 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55da4609224d44989fd4e860066c0076 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:31:09.836702 8077 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55da4609224d44989fd4e860066c0076 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:31:09.836874 8077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55da4609224d44989fd4e860066c0076 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:31:09.839917 8077 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55da4609224d44989fd4e860066c0076 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55da4609224d44989fd4e860066c0076" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40013 } }
I20260926 23:31:09.840711 8077 leader_election.cc:302] T 00000000000000000000000000000000 P 55da4609224d44989fd4e860066c0076 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55da4609224d44989fd4e860066c0076; no voters:
I20260926 23:31:09.841512 8077 leader_election.cc:288] T 00000000000000000000000000000000 P 55da4609224d44989fd4e860066c0076 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:31:09.841657 8082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55da4609224d44989fd4e860066c0076 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:31:09.842491 8082 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55da4609224d44989fd4e860066c0076 [term 1 LEADER]: Becoming Leader. State: Replica: 55da4609224d44989fd4e860066c0076, State: Running, Role: LEADER
I20260926 23:31:09.843200 8082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55da4609224d44989fd4e860066c0076 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55da4609224d44989fd4e860066c0076" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40013 } }
I20260926 23:31:09.844247 8077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55da4609224d44989fd4e860066c0076 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:31:09.847193 8084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55da4609224d44989fd4e860066c0076 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55da4609224d44989fd4e860066c0076. Latest consensus state: current_term: 1 leader_uuid: "55da4609224d44989fd4e860066c0076" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55da4609224d44989fd4e860066c0076" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40013 } } }
I20260926 23:31:09.847530 8083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55da4609224d44989fd4e860066c0076 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55da4609224d44989fd4e860066c0076" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55da4609224d44989fd4e860066c0076" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40013 } } }
I20260926 23:31:09.847867 8083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55da4609224d44989fd4e860066c0076 [sys.catalog]: This master's current role is: LEADER
I20260926 23:31:09.847875 8084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55da4609224d44989fd4e860066c0076 [sys.catalog]: This master's current role is: LEADER
I20260926 23:31:09.849119 8089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:31:09.856801 8089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:31:09.867471 8089 catalog_manager.cc:1347] Generated new cluster ID: 68473faa93a14b6fae5ff752e7ce8090
I20260926 23:31:09.867651 8089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:31:09.879590 8089 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:31:09.881387 8089 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:31:09.890069 8089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55da4609224d44989fd4e860066c0076: Generated new TSK 0
I20260926 23:31:09.891376 8089 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260926 23:31:09.968011 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40013
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:31:10.408907 8103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:31:10.409308 8103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:31:10.409472 8103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:31:10.418529 8103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:31:10.418721 8103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:31:10.418872 8103 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 23:31:10.418998 8103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:31:10.430790 8103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:31:10.432651 8103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:31:10.434440 8103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:10.450452 8108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:10.450529 8109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:10.450484 8111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:10.451563 8103 server_base.cc:1034] running on GCE node
I20260926 23:31:10.452345 8103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:10.453526 8103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:10.454800 8103 hybrid_clock.cc:648] HybridClock initialized: now 1790465470454705 us; error 68 us; skew 500 ppm
I20260926 23:31:10.455154 8103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:31:10 dist-test-slave-gwfx krb5kdc[7987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465470, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:31:10.465235 8103 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260926 23:31:10.467429 8103 webserver.cc:460] Webserver started at http://127.0.97.1:40389/ using document root <none> and password file <none>
I20260926 23:31:10.468384 8103 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:10.468499 8103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:10.468883 8103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:31:10.472683 8103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-0/data/instance:
uuid: "35f2a2b0541d400998e3e677502dc9e2"
format_stamp: "Formatted at 2026-09-26 23:31:10 on dist-test-slave-gwfx"
server_key: "3449e20be2ed7e4982ca25524f4f1ec1"
server_key_iv: "99ad1105778554730ec2127be9416386"
server_key_version: "encryptionkey@0"
I20260926 23:31:10.473698 8103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-0/wal/instance:
uuid: "35f2a2b0541d400998e3e677502dc9e2"
format_stamp: "Formatted at 2026-09-26 23:31:10 on dist-test-slave-gwfx"
server_key: "3449e20be2ed7e4982ca25524f4f1ec1"
server_key_iv: "99ad1105778554730ec2127be9416386"
server_key_version: "encryptionkey@0"
I20260926 23:31:10.480852 8103 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.007s
I20260926 23:31:10.485709 8118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:10.487777 8103 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 23:31:10.488015 8103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "35f2a2b0541d400998e3e677502dc9e2"
format_stamp: "Formatted at 2026-09-26 23:31:10 on dist-test-slave-gwfx"
server_key: "3449e20be2ed7e4982ca25524f4f1ec1"
server_key_iv: "99ad1105778554730ec2127be9416386"
server_key_version: "encryptionkey@0"
I20260926 23:31:10.488299 8103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:10.528846 8103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:10.534636 8103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:31:10.535033 8103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:10.538262 8103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:31:10.538367 8103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:10.538511 8103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:31:10.538580 8103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 23:31:10 dist-test-slave-gwfx krb5kdc[7987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465470, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 23:31:10.566249 8127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:10.539048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47391 (local address 127.0.97.62:40013)
0926 23:31:10.539632 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:10.539639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:10.541257 (+ 1618us) server_negotiation.cc:407] Connection header received
0926 23:31:10.544911 (+ 3654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:10.544919 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:10.545046 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:10.545179 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:10.548149 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:10.548968 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:10.551386 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:10.552135 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:10.552338 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:10.557439 (+ 5101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:10.557447 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:10.557451 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:10.557498 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:10.561612 (+ 4114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:10.562678 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:10.562713 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:10.562716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:10.562908 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:10.563641 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:10.563649 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:10.563653 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:10.564173 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:10.564360 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:10.565386 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0926 23:31:10.565737 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":299,"threads_started":1}
I20260926 23:31:10.567639 8128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:10.539887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40013 (local address 127.0.97.1:47391)
0926 23:31:10.540981 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:10.541074 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:10.544352 (+ 3278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:10.545595 (+ 1243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:10.545610 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:10.546725 (+ 1115us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:10.547889 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:10.547918 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:10.549154 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:10.549159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:10.551164 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:10.551187 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:10.552424 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:10.552449 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:10.552723 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:10.553560 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:10.553593 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:10.556995 (+ 3402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:10.561851 (+ 4856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:10.561862 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:10.561883 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:10.562519 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:10.563127 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:10.563134 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:10.563137 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:10.563481 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:10.564347 (+ 866us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:10.564356 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:10.564889 (+ 533us) client_negotiation.cc:769] Sending connection context
0926 23:31:10.565231 (+ 342us) client_negotiation.cc:240] Negotiation successful
0926 23:31:10.565792 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":334,"threads_started":1}
I20260926 23:31:10.591922 8127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:10.575005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60517 (local address 127.0.97.62:40013)
0926 23:31:10.575232 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:10.575238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:10.576075 (+ 837us) server_negotiation.cc:407] Connection header received
0926 23:31:10.576346 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:10.576353 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:10.576462 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:10.576585 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:10.578730 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:10.579736 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:10.582067 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:10.582777 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:10.582940 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:10.585104 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:10.585111 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:10.585115 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:10.585154 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:10.588523 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:10.589544 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:10.589549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:10.589552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:10.589715 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:10.590128 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:10.590134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:10.590136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:10.590476 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:10.590638 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:10.591220 (+ 582us) server_negotiation.cc:299] Negotiation successful
0926 23:31:10.591490 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 23:31:10.591991 8189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:10.575030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40013 (local address 127.0.97.1:60517)
0926 23:31:10.575884 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:10.575912 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:10.576155 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:10.576650 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:10.576655 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:10.577730 (+ 1075us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:10.578420 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:10.578438 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:10.579952 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:10.579960 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:10.581816 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:10.581842 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:10.583019 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:10.583028 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:10.583233 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:10.583947 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:10.583967 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:10.584857 (+ 890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:10.588721 (+ 3864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:10.588728 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:10.588733 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:10.589393 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:10.589854 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:10.589860 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:10.589864 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:10.590013 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:10.590620 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:10.590626 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:10.590918 (+ 292us) client_negotiation.cc:769] Sending connection context
0926 23:31:10.591110 (+ 192us) client_negotiation.cc:240] Negotiation successful
0926 23:31:10.591534 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":473,"threads_started":1}
I20260926 23:31:10.604970 8103 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37525
I20260926 23:31:10.605006 8238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37525 every 8 connection(s)
I20260926 23:31:10.606590 8103 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:31:10.609294 8239 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40013
I20260926 23:31:10.609570 8239 heartbeater.cc:464] Registering TS with master...
I20260926 23:31:10.610399 8239 heartbeater.cc:511] Master 127.0.97.62:40013 requested a full tablet report, sending...
I20260926 23:31:10.612960 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 8103
I20260926 23:31:10.613178 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:31:10.613279 8032 ts_manager.cc:194] Registered new tserver with Master: 35f2a2b0541d400998e3e677502dc9e2 (127.0.97.1:37525)
I20260926 23:31:10.614017 388 external_mini_cluster.cc:1428] Setting key 1e63c821c8c75463a8e00f78656534eb
I20260926 23:31:10.615494 8032 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47391
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260926 23:31:10.670747 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40013
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:31:11.070889 8243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:31:11.071313 8243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:31:11.071460 8243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:31:11.079756 8243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:31:11.079927 8243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:31:11.080044 8243 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 23:31:11.080176 8243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:31:11.091595 8243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:31:11.093636 8243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:31:11.095793 8243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:11.114496 8249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:11.114578 8251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:11.114667 8248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:11.116051 8243 server_base.cc:1034] running on GCE node
I20260926 23:31:11.116900 8243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:11.118136 8243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:11.119452 8243 hybrid_clock.cc:648] HybridClock initialized: now 1790465471119356 us; error 64 us; skew 500 ppm
I20260926 23:31:11.119848 8243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:31:11 dist-test-slave-gwfx krb5kdc[7987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465471, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:31:11.124017 8243 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260926 23:31:11.126299 8243 webserver.cc:460] Webserver started at http://127.0.97.2:38985/ using document root <none> and password file <none>
I20260926 23:31:11.127501 8243 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:11.127648 8243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:11.128013 8243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:31:11.131727 8243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-1/data/instance:
uuid: "2c827405b79b4eb9b15a6c0689928d2c"
format_stamp: "Formatted at 2026-09-26 23:31:11 on dist-test-slave-gwfx"
server_key: "b1e91c32e4ff2228069fa8b84850390a"
server_key_iv: "e507c4116f20eb208101f96553d4d4f5"
server_key_version: "encryptionkey@0"
I20260926 23:31:11.132666 8243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2c827405b79b4eb9b15a6c0689928d2c"
format_stamp: "Formatted at 2026-09-26 23:31:11 on dist-test-slave-gwfx"
server_key: "b1e91c32e4ff2228069fa8b84850390a"
server_key_iv: "e507c4116f20eb208101f96553d4d4f5"
server_key_version: "encryptionkey@0"
I20260926 23:31:11.140914 8243 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260926 23:31:11.146363 8258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:11.148973 8243 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260926 23:31:11.149217 8243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "2c827405b79b4eb9b15a6c0689928d2c"
format_stamp: "Formatted at 2026-09-26 23:31:11 on dist-test-slave-gwfx"
server_key: "b1e91c32e4ff2228069fa8b84850390a"
server_key_iv: "e507c4116f20eb208101f96553d4d4f5"
server_key_version: "encryptionkey@0"
I20260926 23:31:11.149477 8243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:11.171082 8243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:11.177994 8243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:31:11.178328 8243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:11.181726 8243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:31:11.181859 8243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:11.181985 8243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:31:11.182045 8243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 23:31:11 dist-test-slave-gwfx krb5kdc[7987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465471, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 23:31:11.216728 8267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:11.182892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60619 (local address 127.0.97.62:40013)
0926 23:31:11.183677 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:11.183685 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:11.185113 (+ 1428us) server_negotiation.cc:407] Connection header received
0926 23:31:11.187851 (+ 2738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:11.187859 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:11.188108 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:11.188275 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:11.192649 (+ 4374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:11.193633 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:11.197804 (+ 4171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:11.198529 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:11.199003 (+ 474us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:11.206484 (+ 7481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:11.206494 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:11.206500 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:11.206567 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:11.211171 (+ 4604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:11.212907 (+ 1736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:11.212916 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:11.212920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:11.213164 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:11.214064 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:11.214070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:11.214073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:11.214528 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:11.214782 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:11.215930 (+ 1148us) server_negotiation.cc:299] Negotiation successful
0926 23:31:11.216290 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":402,"threads_started":1}
I20260926 23:31:11.218369 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:11.183712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40013 (local address 127.0.97.2:60619)
0926 23:31:11.184819 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:11.184882 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:11.187405 (+ 2523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:11.188877 (+ 1472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:11.188897 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:11.190398 (+ 1501us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:11.192200 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:11.192268 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:11.193947 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:11.193956 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:11.197334 (+ 3378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:11.197375 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:11.199264 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:11.199278 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:11.199639 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:11.200753 (+ 1114us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:11.200803 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:11.205878 (+ 5075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:11.211562 (+ 5684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:11.211578 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:11.211618 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:11.212576 (+ 958us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:11.213358 (+ 782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:11.213364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:11.213368 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:11.213853 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:11.214869 (+ 1016us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:11.214878 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:11.215423 (+ 545us) client_negotiation.cc:769] Sending connection context
0926 23:31:11.215754 (+ 331us) client_negotiation.cc:240] Negotiation successful
0926 23:31:11.216475 (+ 721us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"thread_start_us":468,"threads_started":1}
I20260926 23:31:11.236176 8243 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42343
I20260926 23:31:11.236259 8378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42343 every 8 connection(s)
I20260926 23:31:11.237980 8243 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:31:11.240204 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 8243
I20260926 23:31:11.240406 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:31:11.241153 388 external_mini_cluster.cc:1428] Setting key 9bc33618ced508022cb58292627a1320
I20260926 23:31:11.241786 8379 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40013
I20260926 23:31:11.242142 8379 heartbeater.cc:464] Registering TS with master...
I20260926 23:31:11.243402 8379 heartbeater.cc:511] Master 127.0.97.62:40013 requested a full tablet report, sending...
I20260926 23:31:11.245340 8032 ts_manager.cc:194] Registered new tserver with Master: 2c827405b79b4eb9b15a6c0689928d2c (127.0.97.2:42343)
I20260926 23:31:11.246635 8032 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60619
I20260926 23:31:11.252864 8267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:11.230540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39341 (local address 127.0.97.62:40013)
0926 23:31:11.230806 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:11.230814 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:11.232190 (+ 1376us) server_negotiation.cc:407] Connection header received
0926 23:31:11.232411 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:11.232417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:11.232524 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:11.232650 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:11.234473 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:11.235807 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:11.237947 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:11.238672 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:11.238903 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:11.245300 (+ 6397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:11.245309 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:11.245315 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:11.245363 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:11.249170 (+ 3807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:11.250477 (+ 1307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:11.250483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:11.250485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:11.250648 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:11.251113 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:11.251118 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:11.251122 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:11.251497 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:11.251651 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:11.252149 (+ 498us) server_negotiation.cc:299] Negotiation successful
0926 23:31:11.252417 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 23:31:11.252956 8369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:11.230669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40013 (local address 127.0.97.2:39341)
0926 23:31:11.232036 (+ 1367us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:11.232065 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:11.232265 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:11.232790 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:11.232802 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:11.233680 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:11.234202 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:11.234219 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:11.236041 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:11.236046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:11.237702 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:11.237720 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:11.239052 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:11.239060 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:11.239301 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:11.242174 (+ 2873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:11.242208 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:11.243739 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:11.249455 (+ 5716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:11.249462 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:11.249466 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:11.250258 (+ 792us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:11.250821 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:11.250827 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:11.250832 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:11.250989 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:11.251672 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:11.251677 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:11.251890 (+ 213us) client_negotiation.cc:769] Sending connection context
0926 23:31:11.252036 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 23:31:11.252442 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1129,"thread_start_us":925,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260926 23:31:11.299328 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40013
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:31:11.619274 8239 heartbeater.cc:503] Master 127.0.97.62:40013 was elected leader, sending a full tablet report...
W20260926 23:31:11.678507 8383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:31:11.678893 8383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:31:11.679026 8383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:31:11.686714 8383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:31:11.686862 8383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:31:11.686976 8383 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 23:31:11.687103 8383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:31:11.697943 8383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:31:11.699375 8383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:31:11.700847 8383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:11.715816 8388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:11.715833 8389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:11.716315 8391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:11.716662 8383 server_base.cc:1034] running on GCE node
I20260926 23:31:11.717459 8383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:11.718623 8383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:11.719892 8383 hybrid_clock.cc:648] HybridClock initialized: now 1790465471719825 us; error 51 us; skew 500 ppm
I20260926 23:31:11.720232 8383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:31:11 dist-test-slave-gwfx krb5kdc[7987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465471, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:31:11.723553 8383 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260926 23:31:11.725185 8383 webserver.cc:460] Webserver started at http://127.0.97.3:45511/ using document root <none> and password file <none>
I20260926 23:31:11.725900 8383 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:11.725996 8383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:11.726284 8383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:31:11.729388 8383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-2/data/instance:
uuid: "02560cfb3ea043278d48f4866bde3bc3"
format_stamp: "Formatted at 2026-09-26 23:31:11 on dist-test-slave-gwfx"
server_key: "9454b1d4ed26b23ddb5ab6b79eac477c"
server_key_iv: "51698b8111d71998907eb503f53951b9"
server_key_version: "encryptionkey@0"
I20260926 23:31:11.730191 8383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-2/wal/instance:
uuid: "02560cfb3ea043278d48f4866bde3bc3"
format_stamp: "Formatted at 2026-09-26 23:31:11 on dist-test-slave-gwfx"
server_key: "9454b1d4ed26b23ddb5ab6b79eac477c"
server_key_iv: "51698b8111d71998907eb503f53951b9"
server_key_version: "encryptionkey@0"
I20260926 23:31:11.735922 8383 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260926 23:31:11.739944 8398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:11.741771 8383 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 23:31:11.741989 8383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "02560cfb3ea043278d48f4866bde3bc3"
format_stamp: "Formatted at 2026-09-26 23:31:11 on dist-test-slave-gwfx"
server_key: "9454b1d4ed26b23ddb5ab6b79eac477c"
server_key_iv: "51698b8111d71998907eb503f53951b9"
server_key_version: "encryptionkey@0"
I20260926 23:31:11.742240 8383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:11.773769 8383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:11.778386 8383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:31:11.778658 8383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:11.781709 8383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:31:11.781829 8383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:11.781955 8383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:31:11.782016 8383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 23:31:11 dist-test-slave-gwfx krb5kdc[7987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465471, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 23:31:11.811648 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:11.782957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51767 (local address 127.0.97.62:40013)
0926 23:31:11.783570 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:11.783578 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:11.784941 (+ 1363us) server_negotiation.cc:407] Connection header received
0926 23:31:11.788229 (+ 3288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:11.788237 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:11.788345 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:11.788428 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:11.791570 (+ 3142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:11.792880 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:11.796464 (+ 3584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:11.797238 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:11.797476 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:11.803754 (+ 6278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:11.803761 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:11.803764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:11.803810 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:11.807093 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:11.808509 (+ 1416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:11.808515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:11.808518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:11.808701 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:11.809263 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:11.809269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:11.809272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:11.809631 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:11.809804 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:11.810865 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0926 23:31:11.811188 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":267,"threads_started":1}
I20260926 23:31:11.813620 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:11.783704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40013 (local address 127.0.97.3:51767)
0926 23:31:11.784693 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:11.784761 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:11.787785 (+ 3024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:11.789057 (+ 1272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:11.789071 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:11.790058 (+ 987us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:11.791299 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:11.791331 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:11.793207 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:11.793228 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:11.796127 (+ 2899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:11.796153 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:11.797648 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:11.797661 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:11.798137 (+ 476us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:11.799192 (+ 1055us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:11.799244 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:11.803346 (+ 4102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:11.807441 (+ 4095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:11.807455 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:11.807492 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:11.808298 (+ 806us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:11.808846 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:11.808850 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:11.808853 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:11.809120 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:11.809793 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:11.809804 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:11.810331 (+ 527us) client_negotiation.cc:769] Sending connection context
0926 23:31:11.810648 (+ 317us) client_negotiation.cc:240] Negotiation successful
0926 23:31:11.811507 (+ 859us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":389,"threads_started":1}
I20260926 23:31:11.838224 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:11.822462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42843 (local address 127.0.97.62:40013)
0926 23:31:11.822753 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:11.822760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:11.823662 (+ 902us) server_negotiation.cc:407] Connection header received
0926 23:31:11.823884 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:11.823890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:11.823981 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:11.824068 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:11.825193 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:11.825976 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:11.828043 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:11.828598 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:11.828686 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:11.831723 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:11.831731 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:11.831735 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:11.831785 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:11.835370 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:11.836171 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:11.836178 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:11.836182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:11.836323 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:11.836675 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:11.836681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:11.836684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:11.837008 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:11.837145 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:11.837568 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 23:31:11.837833 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 23:31:11.838323 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:11.822679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40013 (local address 127.0.97.3:42843)
0926 23:31:11.823517 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:11.823536 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:11.823742 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:11.824113 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:11.824118 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:11.824668 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:11.825025 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:11.825036 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:11.826131 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:11.826135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:11.827789 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:11.827814 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:11.828979 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:11.828984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:11.829129 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:11.830043 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:11.830063 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:11.831424 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:11.835572 (+ 4148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:11.835576 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:11.835580 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:11.836031 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:11.836431 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:11.836435 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:11.836439 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:11.836569 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:11.837124 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:11.837129 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:11.837326 (+ 197us) client_negotiation.cc:769] Sending connection context
0926 23:31:11.837442 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 23:31:11.837829 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":537,"threads_started":1}
I20260926 23:31:11.842103 8383 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37849
I20260926 23:31:11.842207 8518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37849 every 8 connection(s)
I20260926 23:31:11.843619 8383 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:31:11.845925 8519 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40013
I20260926 23:31:11.846200 8519 heartbeater.cc:464] Registering TS with master...
I20260926 23:31:11.847012 8519 heartbeater.cc:511] Master 127.0.97.62:40013 requested a full tablet report, sending...
I20260926 23:31:11.848523 8032 ts_manager.cc:194] Registered new tserver with Master: 02560cfb3ea043278d48f4866bde3bc3 (127.0.97.3:37849)
I20260926 23:31:11.849099 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 8383
I20260926 23:31:11.849282 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:31:11.849723 8032 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51767
I20260926 23:31:11.849872 388 external_mini_cluster.cc:1428] Setting key be7e9bfec70c9817f1709c9db4866d56
I20260926 23:31:11.853289 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 23:31:11 dist-test-slave-gwfx krb5kdc[7987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465471, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 23:31:11 dist-test-slave-gwfx krb5kdc[7987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:11 dist-test-slave-gwfx krb5kdc[7987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 23:31:11.911192 8407 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:11.899304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56490 (local address 127.0.97.62:40013)
0926 23:31:11.899629 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:11.899636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:11.899914 (+ 278us) server_negotiation.cc:407] Connection header received
0926 23:31:11.900136 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:11.900141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:11.900229 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:11.900340 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:11.901868 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:11.902736 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:11.904057 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:11.904481 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:11.904607 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:11.910908 (+ 6301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56490: BlockingRecv error: recv got EOF from 127.0.0.1:56490 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
Sep 26 23:31:11 dist-test-slave-gwfx krb5kdc[7987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465471, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 23:31:11.930148 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:11.915363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56506 (local address 127.0.97.62:40013)
0926 23:31:11.915636 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:11.915644 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:11.915879 (+ 235us) server_negotiation.cc:407] Connection header received
0926 23:31:11.916087 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:11.916091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:11.916181 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:11.916287 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:11.917617 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:11.918441 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:11.919818 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:11.920316 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:11.920439 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:11.924707 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:11.924720 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:11.924723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:11.924768 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:11.927555 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:11.928243 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:11.928249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:11.928252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:11.928372 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:11.928721 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:11.928726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:11.928728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:11.929021 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:11.929307 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:11.929561 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 23:31:11.929815 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 23:31:11.935230 8032 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:31:11.943116 8032 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:31:11.982234 8538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:11.959378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39226 (local address 127.0.97.1:37525)
0926 23:31:11.972546 (+ 13168us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:11.972556 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:11.972589 (+ 33us) server_negotiation.cc:407] Connection header received
0926 23:31:11.972708 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:11.972717 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:11.973017 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:11.973223 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:31:11.974218 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:11.975760 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:11.978981 (+ 3221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:11.980504 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:11.980648 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:11.980894 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:11.981609 (+ 715us) server_negotiation.cc:299] Negotiation successful
0926 23:31:11.981865 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13015,"thread_start_us":12864,"threads_started":1}
I20260926 23:31:11.982312 8540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:11.960395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49896 (local address 127.0.97.3:37849)
0926 23:31:11.973718 (+ 13323us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:11.973728 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:11.973762 (+ 34us) server_negotiation.cc:407] Connection header received
0926 23:31:11.973860 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:11.973868 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:11.974107 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:11.975099 (+ 992us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:31:11.975411 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:11.976870 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:11.979246 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:11.980547 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:11.981030 (+ 483us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:11.981308 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:11.981698 (+ 390us) server_negotiation.cc:299] Negotiation successful
0926 23:31:11.981947 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13132,"thread_start_us":12944,"threads_started":1}
I20260926 23:31:11.984074 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:11.959436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37849 (local address 127.0.0.1:49896)
0926 23:31:11.972897 (+ 13461us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:11.972917 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:11.973041 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:11.974331 (+ 1290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:11.974336 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:11.974363 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:31:11.974891 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:11.974906 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:11.977022 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:11.977027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:11.979003 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:11.979018 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:11.980792 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:11.980798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:11.981248 (+ 450us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:11.981346 (+ 98us) client_negotiation.cc:769] Sending connection context
0926 23:31:11.981486 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 23:31:11.983285 (+ 1799us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13325,"mutex_wait_us":1676,"thread_start_us":11043,"threads_started":1}
I20260926 23:31:11.984627 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:11.959058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37525 (local address 127.0.0.1:39226)
0926 23:31:11.971619 (+ 12561us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:11.971681 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:11.971951 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:11.973534 (+ 1583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:11.973542 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:11.973586 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:31:11.973982 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:11.973995 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:11.975957 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:11.975962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:11.978761 (+ 2799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:11.978795 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:11.980761 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:11.980771 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:11.981290 (+ 519us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:11.981346 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 23:31:11.981493 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 23:31:11.983269 (+ 1776us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11195,"mutex_wait_us":195,"thread_start_us":13604,"threads_started":1}
I20260926 23:31:11.985044 8541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:11.973781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45690 (local address 127.0.97.2:42343)
0926 23:31:11.974408 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:11.974419 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:11.975300 (+ 881us) server_negotiation.cc:407] Connection header received
0926 23:31:11.975601 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:11.975610 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:11.975923 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:11.976112 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:31:11.977161 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:11.978633 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:11.981248 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:11.983623 (+ 2375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:11.983757 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:11.983981 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:11.984503 (+ 522us) server_negotiation.cc:299] Negotiation successful
0926 23:31:11.984702 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":314,"threads_started":1}
I20260926 23:31:11.985836 8542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:11.974436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42343 (local address 127.0.0.1:45690)
0926 23:31:11.975148 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:11.975169 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:11.975412 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:11.976364 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:11.976367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:11.976388 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:31:11.976897 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:11.976906 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:11.978861 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:11.978867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:11.981043 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:11.981060 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:11.984024 (+ 2964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:11.984031 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:11.984242 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:11.984283 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 23:31:11.985289 (+ 1006us) client_negotiation.cc:240] Negotiation successful
0926 23:31:11.985467 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":122,"thread_start_us":388,"threads_started":1}
I20260926 23:31:11.988142 8449 tablet_service.cc:1467] Processing CreateTablet for tablet 0a0090b9a93b47c9a442cc85424cec28 (DEFAULT_TABLE table=test-table [id=781b60eedb1641679a1016786b5a5f4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:31:11.989652 8449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a0090b9a93b47c9a442cc85424cec28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:31:11.989346 8169 tablet_service.cc:1467] Processing CreateTablet for tablet 0a0090b9a93b47c9a442cc85424cec28 (DEFAULT_TABLE table=test-table [id=781b60eedb1641679a1016786b5a5f4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:31:11.989347 8309 tablet_service.cc:1467] Processing CreateTablet for tablet 0a0090b9a93b47c9a442cc85424cec28 (DEFAULT_TABLE table=test-table [id=781b60eedb1641679a1016786b5a5f4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:31:11.990998 8169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a0090b9a93b47c9a442cc85424cec28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:31:11.991012 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a0090b9a93b47c9a442cc85424cec28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:31:12.002408 8543 tablet_bootstrap.cc:492] T 0a0090b9a93b47c9a442cc85424cec28 P 02560cfb3ea043278d48f4866bde3bc3: Bootstrap starting.
I20260926 23:31:12.005784 8543 tablet_bootstrap.cc:654] T 0a0090b9a93b47c9a442cc85424cec28 P 02560cfb3ea043278d48f4866bde3bc3: Neither blocks nor log segments found. Creating new log.
I20260926 23:31:12.007150 8543 log.cc:824] T 0a0090b9a93b47c9a442cc85424cec28 P 02560cfb3ea043278d48f4866bde3bc3: Log is configured to *not* fsync() on all Append() calls
I20260926 23:31:12.008322 8545 tablet_bootstrap.cc:492] T 0a0090b9a93b47c9a442cc85424cec28 P 35f2a2b0541d400998e3e677502dc9e2: Bootstrap starting.
I20260926 23:31:12.011036 8544 tablet_bootstrap.cc:492] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c: Bootstrap starting.
I20260926 23:31:12.012989 8545 tablet_bootstrap.cc:654] T 0a0090b9a93b47c9a442cc85424cec28 P 35f2a2b0541d400998e3e677502dc9e2: Neither blocks nor log segments found. Creating new log.
I20260926 23:31:12.013520 8543 tablet_bootstrap.cc:492] T 0a0090b9a93b47c9a442cc85424cec28 P 02560cfb3ea043278d48f4866bde3bc3: No bootstrap required, opened a new log
I20260926 23:31:12.013890 8543 ts_tablet_manager.cc:1397] T 0a0090b9a93b47c9a442cc85424cec28 P 02560cfb3ea043278d48f4866bde3bc3: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20260926 23:31:12.014542 8545 log.cc:824] T 0a0090b9a93b47c9a442cc85424cec28 P 35f2a2b0541d400998e3e677502dc9e2: Log is configured to *not* fsync() on all Append() calls
I20260926 23:31:12.015115 8544 tablet_bootstrap.cc:654] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c: Neither blocks nor log segments found. Creating new log.
I20260926 23:31:12.016599 8544 log.cc:824] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c: Log is configured to *not* fsync() on all Append() calls
I20260926 23:31:12.019482 8545 tablet_bootstrap.cc:492] T 0a0090b9a93b47c9a442cc85424cec28 P 35f2a2b0541d400998e3e677502dc9e2: No bootstrap required, opened a new log
I20260926 23:31:12.019832 8545 ts_tablet_manager.cc:1397] T 0a0090b9a93b47c9a442cc85424cec28 P 35f2a2b0541d400998e3e677502dc9e2: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260926 23:31:12.022179 8544 tablet_bootstrap.cc:492] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c: No bootstrap required, opened a new log
I20260926 23:31:12.022517 8544 ts_tablet_manager.cc:1397] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260926 23:31:12.023445 8543 raft_consensus.cc:348] T 0a0090b9a93b47c9a442cc85424cec28 P 02560cfb3ea043278d48f4866bde3bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } }
I20260926 23:31:12.023926 8543 raft_consensus.cc:374] T 0a0090b9a93b47c9a442cc85424cec28 P 02560cfb3ea043278d48f4866bde3bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:31:12.024026 8543 raft_consensus.cc:729] T 0a0090b9a93b47c9a442cc85424cec28 P 02560cfb3ea043278d48f4866bde3bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02560cfb3ea043278d48f4866bde3bc3, State: Initialized, Role: FOLLOWER
I20260926 23:31:12.024729 8543 consensus_queue.cc:260] T 0a0090b9a93b47c9a442cc85424cec28 P 02560cfb3ea043278d48f4866bde3bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } }
I20260926 23:31:12.026057 8519 heartbeater.cc:503] Master 127.0.97.62:40013 was elected leader, sending a full tablet report...
I20260926 23:31:12.026630 8543 ts_tablet_manager.cc:1428] T 0a0090b9a93b47c9a442cc85424cec28 P 02560cfb3ea043278d48f4866bde3bc3: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260926 23:31:12.030745 8545 raft_consensus.cc:348] T 0a0090b9a93b47c9a442cc85424cec28 P 35f2a2b0541d400998e3e677502dc9e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } }
I20260926 23:31:12.031283 8545 raft_consensus.cc:374] T 0a0090b9a93b47c9a442cc85424cec28 P 35f2a2b0541d400998e3e677502dc9e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:31:12.031404 8545 raft_consensus.cc:729] T 0a0090b9a93b47c9a442cc85424cec28 P 35f2a2b0541d400998e3e677502dc9e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f2a2b0541d400998e3e677502dc9e2, State: Initialized, Role: FOLLOWER
I20260926 23:31:12.032166 8545 consensus_queue.cc:260] T 0a0090b9a93b47c9a442cc85424cec28 P 35f2a2b0541d400998e3e677502dc9e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } }
I20260926 23:31:12.034335 8545 ts_tablet_manager.cc:1428] T 0a0090b9a93b47c9a442cc85424cec28 P 35f2a2b0541d400998e3e677502dc9e2: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260926 23:31:12.035905 8544 raft_consensus.cc:348] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } }
I20260926 23:31:12.036289 8544 raft_consensus.cc:374] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:31:12.036381 8544 raft_consensus.cc:729] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c827405b79b4eb9b15a6c0689928d2c, State: Initialized, Role: FOLLOWER
I20260926 23:31:12.036927 8544 consensus_queue.cc:260] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } }
I20260926 23:31:12.038143 8379 heartbeater.cc:503] Master 127.0.97.62:40013 was elected leader, sending a full tablet report...
I20260926 23:31:12.038595 8544 ts_tablet_manager.cc:1428] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c: Time spent starting tablet: real 0.016s user 0.012s sys 0.000s
I20260926 23:31:12.089814 8551 raft_consensus.cc:482] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:31:12.090133 8551 raft_consensus.cc:504] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } }
I20260926 23:31:12.092170 8551 leader_election.cc:288] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35f2a2b0541d400998e3e677502dc9e2 (127.0.97.1:37525), 02560cfb3ea043278d48f4866bde3bc3 (127.0.97.3:37849)
W20260926 23:31:12.095305 8520 tablet.cc:2369] T 0a0090b9a93b47c9a442cc85424cec28 P 02560cfb3ea043278d48f4866bde3bc3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:31:12.100167 8538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:12.092690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37031 (local address 127.0.97.1:37525)
0926 23:31:12.092961 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:12.092969 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:12.093483 (+ 514us) server_negotiation.cc:407] Connection header received
0926 23:31:12.093769 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:12.093773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:12.093874 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:12.093960 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:31:12.094653 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:12.095535 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:12.097723 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:12.099003 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:12.099153 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:12.099281 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:12.099724 (+ 443us) server_negotiation.cc:299] Negotiation successful
0926 23:31:12.099908 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 23:31:12.100242 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:12.092483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37525 (local address 127.0.97.2:37031)
0926 23:31:12.093331 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:12.093360 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:12.093615 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:12.093991 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:12.093995 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:12.094030 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:31:12.094481 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:12.094495 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:12.095708 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:12.095714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:12.097496 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:12.097517 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:12.099262 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:12.099268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:12.099501 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:12.099531 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 23:31:12.099635 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 23:31:12.099792 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":33,"spinlock_wait_cycles":18816,"thread_start_us":718,"threads_started":1}
I20260926 23:31:12.101655 8540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:12.092983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53881 (local address 127.0.97.3:37849)
0926 23:31:12.093193 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:12.093200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:12.094033 (+ 833us) server_negotiation.cc:407] Connection header received
0926 23:31:12.094397 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:12.094404 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:12.094484 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:12.094594 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:31:12.096426 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:12.097155 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:12.099612 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:12.100563 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:12.100690 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:12.100798 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:12.101215 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 23:31:12.101373 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 23:31:12.101500 8194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0090b9a93b47c9a442cc85424cec28" candidate_uuid: "2c827405b79b4eb9b15a6c0689928d2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f2a2b0541d400998e3e677502dc9e2" is_pre_election: true
I20260926 23:31:12.102092 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:12.093029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37849 (local address 127.0.97.2:53881)
0926 23:31:12.093887 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:12.093908 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:12.094220 (+ 312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:12.095735 (+ 1515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:12.095739 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:12.095767 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:31:12.096229 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:12.096238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:12.097295 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:12.097301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:12.099443 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:12.099456 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:12.100807 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:12.100814 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:12.101036 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:12.101065 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 23:31:12.101151 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 23:31:12.101272 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":48,"thread_start_us":465,"threads_started":1}
I20260926 23:31:12.102233 8194 raft_consensus.cc:2454] T 0a0090b9a93b47c9a442cc85424cec28 P 35f2a2b0541d400998e3e677502dc9e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c827405b79b4eb9b15a6c0689928d2c in term 0.
I20260926 23:31:12.103157 8261 leader_election.cc:302] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c827405b79b4eb9b15a6c0689928d2c, 35f2a2b0541d400998e3e677502dc9e2; no voters:
I20260926 23:31:12.103410 8469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0090b9a93b47c9a442cc85424cec28" candidate_uuid: "2c827405b79b4eb9b15a6c0689928d2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02560cfb3ea043278d48f4866bde3bc3" is_pre_election: true
I20260926 23:31:12.103655 8551 raft_consensus.cc:2781] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:31:12.103826 8551 raft_consensus.cc:482] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:31:12.103922 8551 raft_consensus.cc:3037] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:31:12.104008 8469 raft_consensus.cc:2454] T 0a0090b9a93b47c9a442cc85424cec28 P 02560cfb3ea043278d48f4866bde3bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c827405b79b4eb9b15a6c0689928d2c in term 0.
I20260926 23:31:12.106506 8551 raft_consensus.cc:504] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } }
I20260926 23:31:12.107227 8551 leader_election.cc:288] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c [CANDIDATE]: Term 1 election: Requested vote from peers 35f2a2b0541d400998e3e677502dc9e2 (127.0.97.1:37525), 02560cfb3ea043278d48f4866bde3bc3 (127.0.97.3:37849)
I20260926 23:31:12.107825 8469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0090b9a93b47c9a442cc85424cec28" candidate_uuid: "2c827405b79b4eb9b15a6c0689928d2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02560cfb3ea043278d48f4866bde3bc3"
I20260926 23:31:12.107856 8194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0090b9a93b47c9a442cc85424cec28" candidate_uuid: "2c827405b79b4eb9b15a6c0689928d2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f2a2b0541d400998e3e677502dc9e2"
I20260926 23:31:12.108047 8469 raft_consensus.cc:3037] T 0a0090b9a93b47c9a442cc85424cec28 P 02560cfb3ea043278d48f4866bde3bc3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:31:12.108072 8194 raft_consensus.cc:3037] T 0a0090b9a93b47c9a442cc85424cec28 P 35f2a2b0541d400998e3e677502dc9e2 [term 0 FOLLOWER]: Advancing to term 1
W20260926 23:31:12.109680 8240 tablet.cc:2369] T 0a0090b9a93b47c9a442cc85424cec28 P 35f2a2b0541d400998e3e677502dc9e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:31:12.110924 8194 raft_consensus.cc:2454] T 0a0090b9a93b47c9a442cc85424cec28 P 35f2a2b0541d400998e3e677502dc9e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c827405b79b4eb9b15a6c0689928d2c in term 1.
I20260926 23:31:12.111095 8469 raft_consensus.cc:2454] T 0a0090b9a93b47c9a442cc85424cec28 P 02560cfb3ea043278d48f4866bde3bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c827405b79b4eb9b15a6c0689928d2c in term 1.
I20260926 23:31:12.111404 8261 leader_election.cc:302] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c827405b79b4eb9b15a6c0689928d2c, 35f2a2b0541d400998e3e677502dc9e2; no voters:
I20260926 23:31:12.111775 8551 raft_consensus.cc:2781] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:31:12.112274 8551 raft_consensus.cc:686] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c [term 1 LEADER]: Becoming Leader. State: Replica: 2c827405b79b4eb9b15a6c0689928d2c, State: Running, Role: LEADER
I20260926 23:31:12.112839 8551 consensus_queue.cc:237] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } }
I20260926 23:31:12.118404 8032 catalog_manager.cc:5495] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c827405b79b4eb9b15a6c0689928d2c (127.0.97.2). New cstate: current_term: 1 leader_uuid: "2c827405b79b4eb9b15a6c0689928d2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } health_report { overall_health: HEALTHY } } }
I20260926 23:31:12.147547 8541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:12.141420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45694 (local address 127.0.97.2:42343)
0926 23:31:12.141663 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:12.141669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:12.141691 (+ 22us) server_negotiation.cc:407] Connection header received
0926 23:31:12.141977 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:12.141982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:12.142073 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:12.142257 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:31:12.142604 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:12.143435 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:12.144921 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:12.145454 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:12.145576 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:12.146108 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:31:12.146914 (+ 806us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:31:12.147060 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:12.147137 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 23:31:12.147241 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 23:31:12.157017 8469 raft_consensus.cc:1261] T 0a0090b9a93b47c9a442cc85424cec28 P 02560cfb3ea043278d48f4866bde3bc3 [term 1 FOLLOWER]: Refusing update from remote peer 2c827405b79b4eb9b15a6c0689928d2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:31:12.157332 8194 raft_consensus.cc:1261] T 0a0090b9a93b47c9a442cc85424cec28 P 35f2a2b0541d400998e3e677502dc9e2 [term 1 FOLLOWER]: Refusing update from remote peer 2c827405b79b4eb9b15a6c0689928d2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:31:12.158051 8554 consensus_queue.cc:1035] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:31:12.158463 8551 consensus_queue.cc:1035] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:31:12.175105 8559 mvcc.cc:205] Tried to move back new op lower bound from 7333746573944639488 to 7333746573778391040. Current Snapshot: MvccSnapshot[applied={T|T < 7333746573944639488}]
I20260926 23:31:12.175455 8558 mvcc.cc:205] Tried to move back new op lower bound from 7333746573944639488 to 7333746573778391040. Current Snapshot: MvccSnapshot[applied={T|T < 7333746573944639488}]
I20260926 23:31:12.190264 8032 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:31:12.190552 8032 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:31:12.194659 8032 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 55da4609224d44989fd4e860066c0076: Sending DeleteTablet for 3 replicas of tablet 0a0090b9a93b47c9a442cc85424cec28
I20260926 23:31:12.196190 8449 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a0090b9a93b47c9a442cc85424cec28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:31:12 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49896
I20260926 23:31:12.196214 8169 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a0090b9a93b47c9a442cc85424cec28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:31:12 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39226
I20260926 23:31:12.197129 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a0090b9a93b47c9a442cc85424cec28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:31:12 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45690
I20260926 23:31:12.201853 8032 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:31:12.202932 8032 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:31:12.208729 8568 tablet_replica.cc:330] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c: stopping tablet replica
I20260926 23:31:12.210137 8568 raft_consensus.cc:2229] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:31:12.211623 8568 raft_consensus.cc:2258] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:31:12.212659 8567 tablet_replica.cc:330] T 0a0090b9a93b47c9a442cc85424cec28 P 35f2a2b0541d400998e3e677502dc9e2: stopping tablet replica
I20260926 23:31:12.213617 8567 raft_consensus.cc:2229] T 0a0090b9a93b47c9a442cc85424cec28 P 35f2a2b0541d400998e3e677502dc9e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:31:12.214290 8567 raft_consensus.cc:2258] T 0a0090b9a93b47c9a442cc85424cec28 P 35f2a2b0541d400998e3e677502dc9e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:31:12.215799 8566 tablet_replica.cc:330] T 0a0090b9a93b47c9a442cc85424cec28 P 02560cfb3ea043278d48f4866bde3bc3: stopping tablet replica
I20260926 23:31:12.216259 8568 ts_tablet_manager.cc:1905] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:31:12.216315 8566 raft_consensus.cc:2229] T 0a0090b9a93b47c9a442cc85424cec28 P 02560cfb3ea043278d48f4866bde3bc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:31:12.216997 8566 raft_consensus.cc:2258] T 0a0090b9a93b47c9a442cc85424cec28 P 02560cfb3ea043278d48f4866bde3bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:31:12.217418 8448 tablet_service.cc:1467] Processing CreateTablet for tablet 9e3a0412d11c42c48d21d3e1929b16ac (DEFAULT_TABLE table=test-table [id=e8b8544ae3a34f0a89d0cebe1d8e4643]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:31:12.218044 8448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e3a0412d11c42c48d21d3e1929b16ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:31:12.219372 8169 tablet_service.cc:1467] Processing CreateTablet for tablet 9e3a0412d11c42c48d21d3e1929b16ac (DEFAULT_TABLE table=test-table [id=e8b8544ae3a34f0a89d0cebe1d8e4643]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:31:12.219906 8169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e3a0412d11c42c48d21d3e1929b16ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:31:12.223953 8566 ts_tablet_manager.cc:1905] T 0a0090b9a93b47c9a442cc85424cec28 P 02560cfb3ea043278d48f4866bde3bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:31:12.224655 8568 ts_tablet_manager.cc:1918] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:31:12.224853 8568 log.cc:1196] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-1/wal/wals/0a0090b9a93b47c9a442cc85424cec28
I20260926 23:31:12.225364 8568 ts_tablet_manager.cc:1939] T 0a0090b9a93b47c9a442cc85424cec28 P 2c827405b79b4eb9b15a6c0689928d2c: Deleting consensus metadata
I20260926 23:31:12.226200 8309 tablet_service.cc:1467] Processing CreateTablet for tablet 9e3a0412d11c42c48d21d3e1929b16ac (DEFAULT_TABLE table=test-table [id=e8b8544ae3a34f0a89d0cebe1d8e4643]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:31:12.226738 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e3a0412d11c42c48d21d3e1929b16ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:31:12.227465 8018 catalog_manager.cc:4841] TS 2c827405b79b4eb9b15a6c0689928d2c (127.0.97.2:42343): tablet 0a0090b9a93b47c9a442cc85424cec28 (table test-table [id=781b60eedb1641679a1016786b5a5f4b]) successfully deleted
I20260926 23:31:12.228318 8545 tablet_bootstrap.cc:492] T 9e3a0412d11c42c48d21d3e1929b16ac P 35f2a2b0541d400998e3e677502dc9e2: Bootstrap starting.
I20260926 23:31:12.230924 8545 tablet_bootstrap.cc:654] T 9e3a0412d11c42c48d21d3e1929b16ac P 35f2a2b0541d400998e3e677502dc9e2: Neither blocks nor log segments found. Creating new log.
I20260926 23:31:12.233496 8543 tablet_bootstrap.cc:492] T 9e3a0412d11c42c48d21d3e1929b16ac P 02560cfb3ea043278d48f4866bde3bc3: Bootstrap starting.
I20260926 23:31:12.235571 8543 tablet_bootstrap.cc:654] T 9e3a0412d11c42c48d21d3e1929b16ac P 02560cfb3ea043278d48f4866bde3bc3: Neither blocks nor log segments found. Creating new log.
I20260926 23:31:12.237202 8566 ts_tablet_manager.cc:1918] T 0a0090b9a93b47c9a442cc85424cec28 P 02560cfb3ea043278d48f4866bde3bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:31:12.237449 8566 log.cc:1196] T 0a0090b9a93b47c9a442cc85424cec28 P 02560cfb3ea043278d48f4866bde3bc3: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-2/wal/wals/0a0090b9a93b47c9a442cc85424cec28
I20260926 23:31:12.238073 8566 ts_tablet_manager.cc:1939] T 0a0090b9a93b47c9a442cc85424cec28 P 02560cfb3ea043278d48f4866bde3bc3: Deleting consensus metadata
I20260926 23:31:12.239677 8567 ts_tablet_manager.cc:1905] T 0a0090b9a93b47c9a442cc85424cec28 P 35f2a2b0541d400998e3e677502dc9e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:31:12.240710 8017 catalog_manager.cc:4841] TS 02560cfb3ea043278d48f4866bde3bc3 (127.0.97.3:37849): tablet 0a0090b9a93b47c9a442cc85424cec28 (table test-table [id=781b60eedb1641679a1016786b5a5f4b]) successfully deleted
I20260926 23:31:12.247313 8545 tablet_bootstrap.cc:492] T 9e3a0412d11c42c48d21d3e1929b16ac P 35f2a2b0541d400998e3e677502dc9e2: No bootstrap required, opened a new log
I20260926 23:31:12.247515 8545 ts_tablet_manager.cc:1397] T 9e3a0412d11c42c48d21d3e1929b16ac P 35f2a2b0541d400998e3e677502dc9e2: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.005s
I20260926 23:31:12.248401 8545 raft_consensus.cc:348] T 9e3a0412d11c42c48d21d3e1929b16ac P 35f2a2b0541d400998e3e677502dc9e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:12.248669 8545 raft_consensus.cc:374] T 9e3a0412d11c42c48d21d3e1929b16ac P 35f2a2b0541d400998e3e677502dc9e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:31:12.248770 8545 raft_consensus.cc:729] T 9e3a0412d11c42c48d21d3e1929b16ac P 35f2a2b0541d400998e3e677502dc9e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f2a2b0541d400998e3e677502dc9e2, State: Initialized, Role: FOLLOWER
I20260926 23:31:12.250528 8545 consensus_queue.cc:260] T 9e3a0412d11c42c48d21d3e1929b16ac P 35f2a2b0541d400998e3e677502dc9e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:12.251739 8545 ts_tablet_manager.cc:1428] T 9e3a0412d11c42c48d21d3e1929b16ac P 35f2a2b0541d400998e3e677502dc9e2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260926 23:31:12.252557 8544 tablet_bootstrap.cc:492] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c: Bootstrap starting.
I20260926 23:31:12.253372 8543 tablet_bootstrap.cc:492] T 9e3a0412d11c42c48d21d3e1929b16ac P 02560cfb3ea043278d48f4866bde3bc3: No bootstrap required, opened a new log
I20260926 23:31:12.253643 8543 ts_tablet_manager.cc:1397] T 9e3a0412d11c42c48d21d3e1929b16ac P 02560cfb3ea043278d48f4866bde3bc3: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.011s
I20260926 23:31:12.254769 8543 raft_consensus.cc:348] T 9e3a0412d11c42c48d21d3e1929b16ac P 02560cfb3ea043278d48f4866bde3bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:12.255152 8543 raft_consensus.cc:374] T 9e3a0412d11c42c48d21d3e1929b16ac P 02560cfb3ea043278d48f4866bde3bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:31:12.255295 8543 raft_consensus.cc:729] T 9e3a0412d11c42c48d21d3e1929b16ac P 02560cfb3ea043278d48f4866bde3bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02560cfb3ea043278d48f4866bde3bc3, State: Initialized, Role: FOLLOWER
I20260926 23:31:12.255910 8543 consensus_queue.cc:260] T 9e3a0412d11c42c48d21d3e1929b16ac P 02560cfb3ea043278d48f4866bde3bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:12.256739 8543 ts_tablet_manager.cc:1428] T 9e3a0412d11c42c48d21d3e1929b16ac P 02560cfb3ea043278d48f4866bde3bc3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260926 23:31:12.258460 8544 tablet_bootstrap.cc:654] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c: Neither blocks nor log segments found. Creating new log.
I20260926 23:31:12.259770 8567 ts_tablet_manager.cc:1918] T 0a0090b9a93b47c9a442cc85424cec28 P 35f2a2b0541d400998e3e677502dc9e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:31:12.259979 8567 log.cc:1196] T 0a0090b9a93b47c9a442cc85424cec28 P 35f2a2b0541d400998e3e677502dc9e2: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-0/wal/wals/0a0090b9a93b47c9a442cc85424cec28
I20260926 23:31:12.260562 8567 ts_tablet_manager.cc:1939] T 0a0090b9a93b47c9a442cc85424cec28 P 35f2a2b0541d400998e3e677502dc9e2: Deleting consensus metadata
I20260926 23:31:12.264034 8018 catalog_manager.cc:4841] TS 35f2a2b0541d400998e3e677502dc9e2 (127.0.97.1:37525): tablet 0a0090b9a93b47c9a442cc85424cec28 (table test-table [id=781b60eedb1641679a1016786b5a5f4b]) successfully deleted
I20260926 23:31:12.264972 8544 tablet_bootstrap.cc:492] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c: No bootstrap required, opened a new log
I20260926 23:31:12.265194 8544 ts_tablet_manager.cc:1397] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260926 23:31:12.266211 8544 raft_consensus.cc:348] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:12.266520 8544 raft_consensus.cc:374] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:31:12.266786 8544 raft_consensus.cc:729] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c827405b79b4eb9b15a6c0689928d2c, State: Initialized, Role: FOLLOWER
I20260926 23:31:12.267138 8544 consensus_queue.cc:260] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:12.267808 8544 ts_tablet_manager.cc:1428] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260926 23:31:12.496258 8380 tablet.cc:2369] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:31:12.515954 8554 raft_consensus.cc:482] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:31:12.516289 8554 raft_consensus.cc:504] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:12.517127 8554 leader_election.cc:288] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35f2a2b0541d400998e3e677502dc9e2 (127.0.97.1:37525), 02560cfb3ea043278d48f4866bde3bc3 (127.0.97.3:37849)
I20260926 23:31:12.517940 8194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3a0412d11c42c48d21d3e1929b16ac" candidate_uuid: "2c827405b79b4eb9b15a6c0689928d2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f2a2b0541d400998e3e677502dc9e2" is_pre_election: true
I20260926 23:31:12.517987 8469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3a0412d11c42c48d21d3e1929b16ac" candidate_uuid: "2c827405b79b4eb9b15a6c0689928d2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02560cfb3ea043278d48f4866bde3bc3" is_pre_election: true
I20260926 23:31:12.518214 8194 raft_consensus.cc:2454] T 9e3a0412d11c42c48d21d3e1929b16ac P 35f2a2b0541d400998e3e677502dc9e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c827405b79b4eb9b15a6c0689928d2c in term 0.
I20260926 23:31:12.518340 8469 raft_consensus.cc:2454] T 9e3a0412d11c42c48d21d3e1929b16ac P 02560cfb3ea043278d48f4866bde3bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c827405b79b4eb9b15a6c0689928d2c in term 0.
I20260926 23:31:12.518710 8261 leader_election.cc:302] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c827405b79b4eb9b15a6c0689928d2c, 35f2a2b0541d400998e3e677502dc9e2; no voters:
I20260926 23:31:12.519058 8554 raft_consensus.cc:2781] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:31:12.519222 8554 raft_consensus.cc:482] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:31:12.519353 8554 raft_consensus.cc:3037] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:31:12.522902 8554 raft_consensus.cc:504] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:12.523777 8554 leader_election.cc:288] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c [CANDIDATE]: Term 1 election: Requested vote from peers 35f2a2b0541d400998e3e677502dc9e2 (127.0.97.1:37525), 02560cfb3ea043278d48f4866bde3bc3 (127.0.97.3:37849)
I20260926 23:31:12.524451 8469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3a0412d11c42c48d21d3e1929b16ac" candidate_uuid: "2c827405b79b4eb9b15a6c0689928d2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02560cfb3ea043278d48f4866bde3bc3"
I20260926 23:31:12.524523 8194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3a0412d11c42c48d21d3e1929b16ac" candidate_uuid: "2c827405b79b4eb9b15a6c0689928d2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f2a2b0541d400998e3e677502dc9e2"
I20260926 23:31:12.524755 8469 raft_consensus.cc:3037] T 9e3a0412d11c42c48d21d3e1929b16ac P 02560cfb3ea043278d48f4866bde3bc3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:31:12.524755 8194 raft_consensus.cc:3037] T 9e3a0412d11c42c48d21d3e1929b16ac P 35f2a2b0541d400998e3e677502dc9e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:31:12.528086 8194 raft_consensus.cc:2454] T 9e3a0412d11c42c48d21d3e1929b16ac P 35f2a2b0541d400998e3e677502dc9e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c827405b79b4eb9b15a6c0689928d2c in term 1.
I20260926 23:31:12.528083 8469 raft_consensus.cc:2454] T 9e3a0412d11c42c48d21d3e1929b16ac P 02560cfb3ea043278d48f4866bde3bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c827405b79b4eb9b15a6c0689928d2c in term 1.
I20260926 23:31:12.528759 8260 leader_election.cc:302] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02560cfb3ea043278d48f4866bde3bc3, 2c827405b79b4eb9b15a6c0689928d2c; no voters:
I20260926 23:31:12.529227 8554 raft_consensus.cc:2781] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:31:12.529438 8554 raft_consensus.cc:686] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c [term 1 LEADER]: Becoming Leader. State: Replica: 2c827405b79b4eb9b15a6c0689928d2c, State: Running, Role: LEADER
I20260926 23:31:12.529804 8554 consensus_queue.cc:237] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:12.533227 8031 catalog_manager.cc:5495] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c827405b79b4eb9b15a6c0689928d2c (127.0.97.2). New cstate: current_term: 1 leader_uuid: "2c827405b79b4eb9b15a6c0689928d2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } health_report { overall_health: UNKNOWN } } }
Sep 26 23:31:12 dist-test-slave-gwfx krb5kdc[7987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465469, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 23:31:12.590251 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:12.573668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40013 (local address 127.0.0.1:56522)
0926 23:31:12.574516 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:12.574544 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:12.574775 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:12.575140 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:12.575148 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:12.575933 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:12.576152 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:12.576161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:12.577263 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:12.577267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:12.578232 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:12.578243 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:12.579044 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:12.579050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:12.579233 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:12.580158 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:12.580177 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:12.583293 (+ 3116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:12.586911 (+ 3618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:12.586930 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:12.586940 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:12.587721 (+ 781us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:12.588144 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:12.588149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:12.588153 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:12.588351 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:12.588933 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:12.588943 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:12.589174 (+ 231us) client_negotiation.cc:769] Sending connection context
0926 23:31:12.589390 (+ 216us) client_negotiation.cc:240] Negotiation successful
0926 23:31:12.589793 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":5,"thread_start_us":432,"threads_started":1}
I20260926 23:31:12.590685 8579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:12.573798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56522 (local address 127.0.97.62:40013)
0926 23:31:12.574876 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:12.574882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:12.574903 (+ 21us) server_negotiation.cc:407] Connection header received
0926 23:31:12.574950 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:12.574953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:12.575009 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:12.575098 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:12.576317 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:12.577132 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:12.578357 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:12.578882 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:12.578993 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:12.583614 (+ 4621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:12.583620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:12.583624 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:12.583662 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:12.586645 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:12.587880 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:12.587885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:12.587887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:12.588023 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:12.588474 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:12.588477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:12.588479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:12.588799 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:12.588916 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:12.589477 (+ 561us) server_negotiation.cc:299] Negotiation successful
0926 23:31:12.589974 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":242,"server-negotiator.queue_time_us":948,"thread_start_us":639,"threads_started":1}
I20260926 23:31:12.597810 8031 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56522:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 23:31:12.599308 8031 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:31:12.611533 8169 tablet_service.cc:1467] Processing CreateTablet for tablet 4216e885f2704cdab2ea5363f43913f8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=00064c2979f4406b9f6bb00327ec660f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 23:31:12.611502 8448 tablet_service.cc:1467] Processing CreateTablet for tablet 4216e885f2704cdab2ea5363f43913f8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=00064c2979f4406b9f6bb00327ec660f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 23:31:12.611465 8309 tablet_service.cc:1467] Processing CreateTablet for tablet 4216e885f2704cdab2ea5363f43913f8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=00064c2979f4406b9f6bb00327ec660f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 23:31:12.612210 8169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4216e885f2704cdab2ea5363f43913f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:31:12.612222 8448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4216e885f2704cdab2ea5363f43913f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:31:12.612252 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4216e885f2704cdab2ea5363f43913f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:31:12.620869 8545 tablet_bootstrap.cc:492] T 4216e885f2704cdab2ea5363f43913f8 P 35f2a2b0541d400998e3e677502dc9e2: Bootstrap starting.
I20260926 23:31:12.622812 8545 tablet_bootstrap.cc:654] T 4216e885f2704cdab2ea5363f43913f8 P 35f2a2b0541d400998e3e677502dc9e2: Neither blocks nor log segments found. Creating new log.
I20260926 23:31:12.623517 8543 tablet_bootstrap.cc:492] T 4216e885f2704cdab2ea5363f43913f8 P 02560cfb3ea043278d48f4866bde3bc3: Bootstrap starting.
I20260926 23:31:12.625519 8543 tablet_bootstrap.cc:654] T 4216e885f2704cdab2ea5363f43913f8 P 02560cfb3ea043278d48f4866bde3bc3: Neither blocks nor log segments found. Creating new log.
I20260926 23:31:12.627852 8544 tablet_bootstrap.cc:492] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c: Bootstrap starting.
I20260926 23:31:12.629786 8544 tablet_bootstrap.cc:654] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c: Neither blocks nor log segments found. Creating new log.
I20260926 23:31:12.637687 8543 tablet_bootstrap.cc:492] T 4216e885f2704cdab2ea5363f43913f8 P 02560cfb3ea043278d48f4866bde3bc3: No bootstrap required, opened a new log
I20260926 23:31:12.638041 8543 ts_tablet_manager.cc:1397] T 4216e885f2704cdab2ea5363f43913f8 P 02560cfb3ea043278d48f4866bde3bc3: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.001s
I20260926 23:31:12.639027 8543 raft_consensus.cc:348] T 4216e885f2704cdab2ea5363f43913f8 P 02560cfb3ea043278d48f4866bde3bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:12.639425 8543 raft_consensus.cc:374] T 4216e885f2704cdab2ea5363f43913f8 P 02560cfb3ea043278d48f4866bde3bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:31:12.639566 8543 raft_consensus.cc:729] T 4216e885f2704cdab2ea5363f43913f8 P 02560cfb3ea043278d48f4866bde3bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02560cfb3ea043278d48f4866bde3bc3, State: Initialized, Role: FOLLOWER
I20260926 23:31:12.639950 8543 consensus_queue.cc:260] T 4216e885f2704cdab2ea5363f43913f8 P 02560cfb3ea043278d48f4866bde3bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:12.640569 8549 tablet_replica.cc:439] T 4216e885f2704cdab2ea5363f43913f8 P 02560cfb3ea043278d48f4866bde3bc3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } } }
I20260926 23:31:12.640888 8549 tablet_replica.cc:442] T 4216e885f2704cdab2ea5363f43913f8 P 02560cfb3ea043278d48f4866bde3bc3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:31:12.644881 8543 ts_tablet_manager.cc:1428] T 4216e885f2704cdab2ea5363f43913f8 P 02560cfb3ea043278d48f4866bde3bc3: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260926 23:31:12.647258 8544 tablet_bootstrap.cc:492] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c: No bootstrap required, opened a new log
I20260926 23:31:12.647603 8544 ts_tablet_manager.cc:1397] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.005s
I20260926 23:31:12.648900 8544 raft_consensus.cc:348] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:12.649308 8544 raft_consensus.cc:374] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:31:12.649402 8544 raft_consensus.cc:729] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c827405b79b4eb9b15a6c0689928d2c, State: Initialized, Role: FOLLOWER
I20260926 23:31:12.649737 8544 consensus_queue.cc:260] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:12.650282 8554 tablet_replica.cc:439] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } } }
I20260926 23:31:12.650571 8554 tablet_replica.cc:442] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:31:12.650556 8545 tablet_bootstrap.cc:492] T 4216e885f2704cdab2ea5363f43913f8 P 35f2a2b0541d400998e3e677502dc9e2: No bootstrap required, opened a new log
I20260926 23:31:12.650807 8545 ts_tablet_manager.cc:1397] T 4216e885f2704cdab2ea5363f43913f8 P 35f2a2b0541d400998e3e677502dc9e2: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.006s
I20260926 23:31:12.651247 8544 ts_tablet_manager.cc:1428] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 23:31:12.652767 8545 raft_consensus.cc:348] T 4216e885f2704cdab2ea5363f43913f8 P 35f2a2b0541d400998e3e677502dc9e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:12.653057 8545 raft_consensus.cc:374] T 4216e885f2704cdab2ea5363f43913f8 P 35f2a2b0541d400998e3e677502dc9e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:31:12.653151 8545 raft_consensus.cc:729] T 4216e885f2704cdab2ea5363f43913f8 P 35f2a2b0541d400998e3e677502dc9e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f2a2b0541d400998e3e677502dc9e2, State: Initialized, Role: FOLLOWER
I20260926 23:31:12.653461 8545 consensus_queue.cc:260] T 4216e885f2704cdab2ea5363f43913f8 P 35f2a2b0541d400998e3e677502dc9e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:12.654048 8550 tablet_replica.cc:439] T 4216e885f2704cdab2ea5363f43913f8 P 35f2a2b0541d400998e3e677502dc9e2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } } }
I20260926 23:31:12.655519 8545 ts_tablet_manager.cc:1428] T 4216e885f2704cdab2ea5363f43913f8 P 35f2a2b0541d400998e3e677502dc9e2: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260926 23:31:12.655833 8550 tablet_replica.cc:442] T 4216e885f2704cdab2ea5363f43913f8 P 35f2a2b0541d400998e3e677502dc9e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:31:12.692368 8554 raft_consensus.cc:482] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:31:12.692618 8554 raft_consensus.cc:504] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:12.693387 8554 leader_election.cc:288] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35f2a2b0541d400998e3e677502dc9e2 (127.0.97.1:37525), 02560cfb3ea043278d48f4866bde3bc3 (127.0.97.3:37849)
I20260926 23:31:12.694046 8194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4216e885f2704cdab2ea5363f43913f8" candidate_uuid: "2c827405b79b4eb9b15a6c0689928d2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f2a2b0541d400998e3e677502dc9e2" is_pre_election: true
I20260926 23:31:12.694141 8469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4216e885f2704cdab2ea5363f43913f8" candidate_uuid: "2c827405b79b4eb9b15a6c0689928d2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02560cfb3ea043278d48f4866bde3bc3" is_pre_election: true
I20260926 23:31:12.694355 8194 raft_consensus.cc:2454] T 4216e885f2704cdab2ea5363f43913f8 P 35f2a2b0541d400998e3e677502dc9e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c827405b79b4eb9b15a6c0689928d2c in term 0.
I20260926 23:31:12.694518 8469 raft_consensus.cc:2454] T 4216e885f2704cdab2ea5363f43913f8 P 02560cfb3ea043278d48f4866bde3bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c827405b79b4eb9b15a6c0689928d2c in term 0.
I20260926 23:31:12.694902 8261 leader_election.cc:302] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c827405b79b4eb9b15a6c0689928d2c, 35f2a2b0541d400998e3e677502dc9e2; no voters:
I20260926 23:31:12.695245 8554 raft_consensus.cc:2781] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:31:12.695381 8554 raft_consensus.cc:482] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:31:12.695473 8554 raft_consensus.cc:3037] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:31:12.697656 8554 raft_consensus.cc:504] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:12.698266 8554 leader_election.cc:288] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c [CANDIDATE]: Term 1 election: Requested vote from peers 35f2a2b0541d400998e3e677502dc9e2 (127.0.97.1:37525), 02560cfb3ea043278d48f4866bde3bc3 (127.0.97.3:37849)
I20260926 23:31:12.699093 8194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4216e885f2704cdab2ea5363f43913f8" candidate_uuid: "2c827405b79b4eb9b15a6c0689928d2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f2a2b0541d400998e3e677502dc9e2"
I20260926 23:31:12.699167 8469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4216e885f2704cdab2ea5363f43913f8" candidate_uuid: "2c827405b79b4eb9b15a6c0689928d2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02560cfb3ea043278d48f4866bde3bc3"
I20260926 23:31:12.699445 8194 raft_consensus.cc:3037] T 4216e885f2704cdab2ea5363f43913f8 P 35f2a2b0541d400998e3e677502dc9e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:31:12.699443 8469 raft_consensus.cc:3037] T 4216e885f2704cdab2ea5363f43913f8 P 02560cfb3ea043278d48f4866bde3bc3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:31:12.702644 8194 raft_consensus.cc:2454] T 4216e885f2704cdab2ea5363f43913f8 P 35f2a2b0541d400998e3e677502dc9e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c827405b79b4eb9b15a6c0689928d2c in term 1.
I20260926 23:31:12.702644 8469 raft_consensus.cc:2454] T 4216e885f2704cdab2ea5363f43913f8 P 02560cfb3ea043278d48f4866bde3bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c827405b79b4eb9b15a6c0689928d2c in term 1.
I20260926 23:31:12.703399 8260 leader_election.cc:302] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02560cfb3ea043278d48f4866bde3bc3, 2c827405b79b4eb9b15a6c0689928d2c; no voters:
I20260926 23:31:12.703866 8554 raft_consensus.cc:2781] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:31:12.704067 8554 raft_consensus.cc:686] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c [term 1 LEADER]: Becoming Leader. State: Replica: 2c827405b79b4eb9b15a6c0689928d2c, State: Running, Role: LEADER
I20260926 23:31:12.704384 8554 consensus_queue.cc:237] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:12.706032 8551 tablet_replica.cc:439] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c: TxnStatusTablet state changed. Reason: New leader 2c827405b79b4eb9b15a6c0689928d2c. Latest consensus state: current_term: 1 leader_uuid: "2c827405b79b4eb9b15a6c0689928d2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } } }
I20260926 23:31:12.706384 8551 tablet_replica.cc:442] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c: This TxnStatusTablet replica's current role is: LEADER
I20260926 23:31:12.707041 8584 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 23:31:12.707289 8584 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 23:31:12.707947 8032 catalog_manager.cc:5495] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c827405b79b4eb9b15a6c0689928d2c (127.0.97.2). New cstate: current_term: 1 leader_uuid: "2c827405b79b4eb9b15a6c0689928d2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } health_report { overall_health: UNKNOWN } } }
I20260926 23:31:12.743372 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:12.736554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42343 (local address 127.0.0.1:45704)
0926 23:31:12.736919 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:12.736946 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:12.737144 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:12.737628 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:12.737633 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:12.737648 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:31:12.737859 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:12.737873 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:12.739234 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:12.739241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:12.740972 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:12.740994 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:12.741932 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:12.741939 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:12.742118 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:12.742265 (+ 147us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:31:12.742737 (+ 472us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:31:12.742760 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 23:31:12.742837 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 23:31:12.743026 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 23:31:12.743428 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:12.736619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45704 (local address 127.0.97.2:42343)
0926 23:31:12.737274 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:12.737280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:12.737300 (+ 20us) server_negotiation.cc:407] Connection header received
0926 23:31:12.737375 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:12.737379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:12.737456 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:12.737566 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:31:12.738069 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:12.739073 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:12.741166 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:12.741683 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:12.741882 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:12.742482 (+ 600us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:31:12.742612 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:31:12.742735 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:12.742872 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 23:31:12.742962 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":359,"threads_started":1}
I20260926 23:31:12.748878 8469 raft_consensus.cc:1261] T 4216e885f2704cdab2ea5363f43913f8 P 02560cfb3ea043278d48f4866bde3bc3 [term 1 FOLLOWER]: Refusing update from remote peer 2c827405b79b4eb9b15a6c0689928d2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:31:12.748878 8194 raft_consensus.cc:1261] T 4216e885f2704cdab2ea5363f43913f8 P 35f2a2b0541d400998e3e677502dc9e2 [term 1 FOLLOWER]: Refusing update from remote peer 2c827405b79b4eb9b15a6c0689928d2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:31:12.749642 8551 consensus_queue.cc:1035] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:31:12.749969 8554 consensus_queue.cc:1035] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:31:12.761353 8550 tablet_replica.cc:439] T 4216e885f2704cdab2ea5363f43913f8 P 35f2a2b0541d400998e3e677502dc9e2: TxnStatusTablet state changed. Reason: New leader 2c827405b79b4eb9b15a6c0689928d2c. Latest consensus state: current_term: 1 leader_uuid: "2c827405b79b4eb9b15a6c0689928d2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } } }
I20260926 23:31:12.761701 8550 tablet_replica.cc:442] T 4216e885f2704cdab2ea5363f43913f8 P 35f2a2b0541d400998e3e677502dc9e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:31:12.764580 8549 tablet_replica.cc:439] T 4216e885f2704cdab2ea5363f43913f8 P 02560cfb3ea043278d48f4866bde3bc3: TxnStatusTablet state changed. Reason: New leader 2c827405b79b4eb9b15a6c0689928d2c. Latest consensus state: current_term: 1 leader_uuid: "2c827405b79b4eb9b15a6c0689928d2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } } }
I20260926 23:31:12.764940 8549 tablet_replica.cc:442] T 4216e885f2704cdab2ea5363f43913f8 P 02560cfb3ea043278d48f4866bde3bc3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:31:12.787647 8549 tablet_replica.cc:439] T 4216e885f2704cdab2ea5363f43913f8 P 02560cfb3ea043278d48f4866bde3bc3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c827405b79b4eb9b15a6c0689928d2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } } }
I20260926 23:31:12.787739 8550 tablet_replica.cc:439] T 4216e885f2704cdab2ea5363f43913f8 P 35f2a2b0541d400998e3e677502dc9e2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c827405b79b4eb9b15a6c0689928d2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } } }
I20260926 23:31:12.787999 8549 tablet_replica.cc:442] T 4216e885f2704cdab2ea5363f43913f8 P 02560cfb3ea043278d48f4866bde3bc3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:31:12.788018 8550 tablet_replica.cc:442] T 4216e885f2704cdab2ea5363f43913f8 P 35f2a2b0541d400998e3e677502dc9e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 23:31:12.791307 8592 tablet_replica.cc:439] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c827405b79b4eb9b15a6c0689928d2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } } }
I20260926 23:31:12.791803 8592 tablet_replica.cc:442] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c: This TxnStatusTablet replica's current role is: LEADER
I20260926 23:31:12.792181 8554 tablet_replica.cc:439] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c827405b79b4eb9b15a6c0689928d2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } } }
I20260926 23:31:12.792464 8554 tablet_replica.cc:442] T 4216e885f2704cdab2ea5363f43913f8 P 2c827405b79b4eb9b15a6c0689928d2c: This TxnStatusTablet replica's current role is: LEADER
I20260926 23:31:12.817420 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:12.808733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52677 (local address 127.0.97.2:42343)
0926 23:31:12.808961 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:12.808970 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:12.809686 (+ 716us) server_negotiation.cc:407] Connection header received
0926 23:31:12.809919 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:12.809925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:12.810032 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:12.810171 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:31:12.810890 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:12.812236 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:12.814653 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:12.815280 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:12.815423 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:12.816247 (+ 824us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:31:12.816420 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:31:12.816501 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:12.816740 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 23:31:12.816995 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":108,"server-negotiator.queue_time_us":59}
I20260926 23:31:12.817633 8598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:12.808566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42343 (local address 127.0.97.2:52677)
0926 23:31:12.809524 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:12.809548 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:12.809757 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:12.810229 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:12.810235 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:12.810251 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:31:12.810616 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:12.810629 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:12.812446 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:12.812453 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:12.814407 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:12.814431 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:12.815516 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:12.815524 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:12.815778 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:12.815880 (+ 102us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:31:12.816544 (+ 664us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:31:12.816557 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 23:31:12.816671 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 23:31:12.816995 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"mutex_wait_us":5,"thread_start_us":443,"threads_started":1}
I20260926 23:31:12.829489 8469 raft_consensus.cc:1261] T 9e3a0412d11c42c48d21d3e1929b16ac P 02560cfb3ea043278d48f4866bde3bc3 [term 1 FOLLOWER]: Refusing update from remote peer 2c827405b79b4eb9b15a6c0689928d2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:31:12.829574 8194 raft_consensus.cc:1261] T 9e3a0412d11c42c48d21d3e1929b16ac P 35f2a2b0541d400998e3e677502dc9e2 [term 1 FOLLOWER]: Refusing update from remote peer 2c827405b79b4eb9b15a6c0689928d2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:31:12.830111 8554 consensus_queue.cc:1035] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:31:12.830441 8592 consensus_queue.cc:1035] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:31:12.843859 8587 mvcc.cc:205] Tried to move back new op lower bound from 7333746576703909888 to 7333746575484739584. Current Snapshot: MvccSnapshot[applied={T|T < 7333746576703909888}]
I20260926 23:31:12.934726 8032 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:31:12.935082 8032 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:31:12.939508 8032 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 55da4609224d44989fd4e860066c0076: Sending DeleteTablet for 3 replicas of tablet 9e3a0412d11c42c48d21d3e1929b16ac
I20260926 23:31:12.940490 8169 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e3a0412d11c42c48d21d3e1929b16ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:31:12 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39226
I20260926 23:31:12.940766 8448 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e3a0412d11c42c48d21d3e1929b16ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:31:12 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49896
I20260926 23:31:12.947324 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e3a0412d11c42c48d21d3e1929b16ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:31:12 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45690
I20260926 23:31:12.952117 8604 tablet_replica.cc:330] T 9e3a0412d11c42c48d21d3e1929b16ac P 02560cfb3ea043278d48f4866bde3bc3: stopping tablet replica
I20260926 23:31:12.952522 8604 raft_consensus.cc:2229] T 9e3a0412d11c42c48d21d3e1929b16ac P 02560cfb3ea043278d48f4866bde3bc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:31:12.952876 8604 raft_consensus.cc:2258] T 9e3a0412d11c42c48d21d3e1929b16ac P 02560cfb3ea043278d48f4866bde3bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:31:12.954309 8604 ts_tablet_manager.cc:1905] T 9e3a0412d11c42c48d21d3e1929b16ac P 02560cfb3ea043278d48f4866bde3bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:31:12.961457 8605 tablet_replica.cc:330] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c: stopping tablet replica
I20260926 23:31:12.961932 8605 raft_consensus.cc:2229] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:31:12.962491 8604 ts_tablet_manager.cc:1918] T 9e3a0412d11c42c48d21d3e1929b16ac P 02560cfb3ea043278d48f4866bde3bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:31:12.962536 8605 raft_consensus.cc:2258] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:31:12.962718 8604 log.cc:1196] T 9e3a0412d11c42c48d21d3e1929b16ac P 02560cfb3ea043278d48f4866bde3bc3: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-2/wal/wals/9e3a0412d11c42c48d21d3e1929b16ac
I20260926 23:31:12.963161 8604 ts_tablet_manager.cc:1939] T 9e3a0412d11c42c48d21d3e1929b16ac P 02560cfb3ea043278d48f4866bde3bc3: Deleting consensus metadata
I20260926 23:31:12.964589 8605 ts_tablet_manager.cc:1905] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:31:12.964622 8017 catalog_manager.cc:4841] TS 02560cfb3ea043278d48f4866bde3bc3 (127.0.97.3:37849): tablet 9e3a0412d11c42c48d21d3e1929b16ac (table test-table [id=e8b8544ae3a34f0a89d0cebe1d8e4643]) successfully deleted
I20260926 23:31:12.971045 8603 tablet_replica.cc:330] T 9e3a0412d11c42c48d21d3e1929b16ac P 35f2a2b0541d400998e3e677502dc9e2: stopping tablet replica
I20260926 23:31:12.971427 8603 raft_consensus.cc:2229] T 9e3a0412d11c42c48d21d3e1929b16ac P 35f2a2b0541d400998e3e677502dc9e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:31:12.971856 8603 raft_consensus.cc:2258] T 9e3a0412d11c42c48d21d3e1929b16ac P 35f2a2b0541d400998e3e677502dc9e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:31:12.973060 8603 ts_tablet_manager.cc:1905] T 9e3a0412d11c42c48d21d3e1929b16ac P 35f2a2b0541d400998e3e677502dc9e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:31:12.975556 8605 ts_tablet_manager.cc:1918] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:31:12.975737 8605 log.cc:1196] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-1/wal/wals/9e3a0412d11c42c48d21d3e1929b16ac
I20260926 23:31:12.976224 8605 ts_tablet_manager.cc:1939] T 9e3a0412d11c42c48d21d3e1929b16ac P 2c827405b79b4eb9b15a6c0689928d2c: Deleting consensus metadata
I20260926 23:31:12.977664 8018 catalog_manager.cc:4841] TS 2c827405b79b4eb9b15a6c0689928d2c (127.0.97.2:42343): tablet 9e3a0412d11c42c48d21d3e1929b16ac (table test-table [id=e8b8544ae3a34f0a89d0cebe1d8e4643]) successfully deleted
I20260926 23:31:12.980559 8603 ts_tablet_manager.cc:1918] T 9e3a0412d11c42c48d21d3e1929b16ac P 35f2a2b0541d400998e3e677502dc9e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:31:12.980803 8603 log.cc:1196] T 9e3a0412d11c42c48d21d3e1929b16ac P 35f2a2b0541d400998e3e677502dc9e2: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-0/wal/wals/9e3a0412d11c42c48d21d3e1929b16ac
I20260926 23:31:12.981284 8603 ts_tablet_manager.cc:1939] T 9e3a0412d11c42c48d21d3e1929b16ac P 35f2a2b0541d400998e3e677502dc9e2: Deleting consensus metadata
I20260926 23:31:12.982509 8018 catalog_manager.cc:4841] TS 35f2a2b0541d400998e3e677502dc9e2 (127.0.97.1:37525): tablet 9e3a0412d11c42c48d21d3e1929b16ac (table test-table [id=e8b8544ae3a34f0a89d0cebe1d8e4643]) successfully deleted
Sep 26 23:31:12 dist-test-slave-gwfx krb5kdc[7987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 23:31:13 dist-test-slave-gwfx krb5kdc[7987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:13 dist-test-slave-gwfx krb5kdc[7987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 23:31:13.022387 8579 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:13.006815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56530 (local address 127.0.97.62:40013)
0926 23:31:13.007071 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:13.007077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:13.007646 (+ 569us) server_negotiation.cc:407] Connection header received
0926 23:31:13.007910 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:13.007915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:13.008006 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:13.008454 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:13.010876 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:13.011753 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:13.014383 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:13.014981 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:13.015136 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:13.022073 (+ 6937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56530: BlockingRecv error: recv got EOF from 127.0.0.1:56530 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 26 23:31:13 dist-test-slave-gwfx krb5kdc[7987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 23:31:13.044241 8579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:13.026149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56544 (local address 127.0.97.62:40013)
0926 23:31:13.026321 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:13.026327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:13.026678 (+ 351us) server_negotiation.cc:407] Connection header received
0926 23:31:13.026892 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:13.026896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:13.026970 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:13.027053 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:13.028754 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:13.029552 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:13.031016 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:13.031457 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:13.031575 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:13.037227 (+ 5652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:13.037233 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:13.037237 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:13.037293 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:13.040633 (+ 3340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:13.041451 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:13.041457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:13.041461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:13.041607 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:13.042162 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:13.042170 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:13.042174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:13.042779 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:13.043109 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:13.043485 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 23:31:13.043844 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 23:31:13.049069 8032 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:31:13.050094 8032 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:31:13.060732 8169 tablet_service.cc:1467] Processing CreateTablet for tablet a58674195b564589a1707c36138e6ba9 (DEFAULT_TABLE table=test-table [id=56f99fc31aeb4fccb65be521f2e61a07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:31:13.061343 8169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a58674195b564589a1707c36138e6ba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:31:13.061422 8309 tablet_service.cc:1467] Processing CreateTablet for tablet a58674195b564589a1707c36138e6ba9 (DEFAULT_TABLE table=test-table [id=56f99fc31aeb4fccb65be521f2e61a07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:31:13.061673 8448 tablet_service.cc:1467] Processing CreateTablet for tablet a58674195b564589a1707c36138e6ba9 (DEFAULT_TABLE table=test-table [id=56f99fc31aeb4fccb65be521f2e61a07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:31:13.061868 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a58674195b564589a1707c36138e6ba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:31:13.062275 8448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a58674195b564589a1707c36138e6ba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:31:13.068580 8545 tablet_bootstrap.cc:492] T a58674195b564589a1707c36138e6ba9 P 35f2a2b0541d400998e3e677502dc9e2: Bootstrap starting.
I20260926 23:31:13.070225 8543 tablet_bootstrap.cc:492] T a58674195b564589a1707c36138e6ba9 P 02560cfb3ea043278d48f4866bde3bc3: Bootstrap starting.
I20260926 23:31:13.070946 8545 tablet_bootstrap.cc:654] T a58674195b564589a1707c36138e6ba9 P 35f2a2b0541d400998e3e677502dc9e2: Neither blocks nor log segments found. Creating new log.
I20260926 23:31:13.071659 8544 tablet_bootstrap.cc:492] T a58674195b564589a1707c36138e6ba9 P 2c827405b79b4eb9b15a6c0689928d2c: Bootstrap starting.
I20260926 23:31:13.072304 8543 tablet_bootstrap.cc:654] T a58674195b564589a1707c36138e6ba9 P 02560cfb3ea043278d48f4866bde3bc3: Neither blocks nor log segments found. Creating new log.
I20260926 23:31:13.073370 8544 tablet_bootstrap.cc:654] T a58674195b564589a1707c36138e6ba9 P 2c827405b79b4eb9b15a6c0689928d2c: Neither blocks nor log segments found. Creating new log.
I20260926 23:31:13.075246 8543 tablet_bootstrap.cc:492] T a58674195b564589a1707c36138e6ba9 P 02560cfb3ea043278d48f4866bde3bc3: No bootstrap required, opened a new log
I20260926 23:31:13.075428 8543 ts_tablet_manager.cc:1397] T a58674195b564589a1707c36138e6ba9 P 02560cfb3ea043278d48f4866bde3bc3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 23:31:13.076632 8545 tablet_bootstrap.cc:492] T a58674195b564589a1707c36138e6ba9 P 35f2a2b0541d400998e3e677502dc9e2: No bootstrap required, opened a new log
I20260926 23:31:13.076558 8543 raft_consensus.cc:348] T a58674195b564589a1707c36138e6ba9 P 02560cfb3ea043278d48f4866bde3bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:13.076875 8544 tablet_bootstrap.cc:492] T a58674195b564589a1707c36138e6ba9 P 2c827405b79b4eb9b15a6c0689928d2c: No bootstrap required, opened a new log
I20260926 23:31:13.076892 8543 raft_consensus.cc:374] T a58674195b564589a1707c36138e6ba9 P 02560cfb3ea043278d48f4866bde3bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:31:13.076988 8543 raft_consensus.cc:729] T a58674195b564589a1707c36138e6ba9 P 02560cfb3ea043278d48f4866bde3bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02560cfb3ea043278d48f4866bde3bc3, State: Initialized, Role: FOLLOWER
I20260926 23:31:13.077040 8544 ts_tablet_manager.cc:1397] T a58674195b564589a1707c36138e6ba9 P 2c827405b79b4eb9b15a6c0689928d2c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260926 23:31:13.077135 8545 ts_tablet_manager.cc:1397] T a58674195b564589a1707c36138e6ba9 P 35f2a2b0541d400998e3e677502dc9e2: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20260926 23:31:13.077304 8543 consensus_queue.cc:260] T a58674195b564589a1707c36138e6ba9 P 02560cfb3ea043278d48f4866bde3bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:13.077998 8544 raft_consensus.cc:348] T a58674195b564589a1707c36138e6ba9 P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:13.078289 8544 raft_consensus.cc:374] T a58674195b564589a1707c36138e6ba9 P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:31:13.078392 8544 raft_consensus.cc:729] T a58674195b564589a1707c36138e6ba9 P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c827405b79b4eb9b15a6c0689928d2c, State: Initialized, Role: FOLLOWER
I20260926 23:31:13.078191 8545 raft_consensus.cc:348] T a58674195b564589a1707c36138e6ba9 P 35f2a2b0541d400998e3e677502dc9e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:13.078717 8545 raft_consensus.cc:374] T a58674195b564589a1707c36138e6ba9 P 35f2a2b0541d400998e3e677502dc9e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:31:13.078853 8545 raft_consensus.cc:729] T a58674195b564589a1707c36138e6ba9 P 35f2a2b0541d400998e3e677502dc9e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f2a2b0541d400998e3e677502dc9e2, State: Initialized, Role: FOLLOWER
I20260926 23:31:13.079137 8545 consensus_queue.cc:260] T a58674195b564589a1707c36138e6ba9 P 35f2a2b0541d400998e3e677502dc9e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:13.079231 8544 consensus_queue.cc:260] T a58674195b564589a1707c36138e6ba9 P 2c827405b79b4eb9b15a6c0689928d2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:13.079833 8545 ts_tablet_manager.cc:1428] T a58674195b564589a1707c36138e6ba9 P 35f2a2b0541d400998e3e677502dc9e2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 23:31:13.080672 8544 ts_tablet_manager.cc:1428] T a58674195b564589a1707c36138e6ba9 P 2c827405b79b4eb9b15a6c0689928d2c: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260926 23:31:13.084733 8543 ts_tablet_manager.cc:1428] T a58674195b564589a1707c36138e6ba9 P 02560cfb3ea043278d48f4866bde3bc3: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20260926 23:31:13.105708 8551 raft_consensus.cc:482] T a58674195b564589a1707c36138e6ba9 P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:31:13.105985 8551 raft_consensus.cc:504] T a58674195b564589a1707c36138e6ba9 P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:13.106837 8551 leader_election.cc:288] T a58674195b564589a1707c36138e6ba9 P 2c827405b79b4eb9b15a6c0689928d2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35f2a2b0541d400998e3e677502dc9e2 (127.0.97.1:37525), 02560cfb3ea043278d48f4866bde3bc3 (127.0.97.3:37849)
I20260926 23:31:13.107523 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a58674195b564589a1707c36138e6ba9" candidate_uuid: "2c827405b79b4eb9b15a6c0689928d2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f2a2b0541d400998e3e677502dc9e2" is_pre_election: true
I20260926 23:31:13.107563 8468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a58674195b564589a1707c36138e6ba9" candidate_uuid: "2c827405b79b4eb9b15a6c0689928d2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02560cfb3ea043278d48f4866bde3bc3" is_pre_election: true
I20260926 23:31:13.107868 8468 raft_consensus.cc:2454] T a58674195b564589a1707c36138e6ba9 P 02560cfb3ea043278d48f4866bde3bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c827405b79b4eb9b15a6c0689928d2c in term 0.
I20260926 23:31:13.107872 8193 raft_consensus.cc:2454] T a58674195b564589a1707c36138e6ba9 P 35f2a2b0541d400998e3e677502dc9e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c827405b79b4eb9b15a6c0689928d2c in term 0.
I20260926 23:31:13.108465 8261 leader_election.cc:302] T a58674195b564589a1707c36138e6ba9 P 2c827405b79b4eb9b15a6c0689928d2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 02560cfb3ea043278d48f4866bde3bc3, 2c827405b79b4eb9b15a6c0689928d2c, 35f2a2b0541d400998e3e677502dc9e2; no voters:
I20260926 23:31:13.108873 8551 raft_consensus.cc:2781] T a58674195b564589a1707c36138e6ba9 P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:31:13.108979 8551 raft_consensus.cc:482] T a58674195b564589a1707c36138e6ba9 P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:31:13.109097 8551 raft_consensus.cc:3037] T a58674195b564589a1707c36138e6ba9 P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:31:13.112759 8551 raft_consensus.cc:504] T a58674195b564589a1707c36138e6ba9 P 2c827405b79b4eb9b15a6c0689928d2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:13.113633 8551 leader_election.cc:288] T a58674195b564589a1707c36138e6ba9 P 2c827405b79b4eb9b15a6c0689928d2c [CANDIDATE]: Term 1 election: Requested vote from peers 35f2a2b0541d400998e3e677502dc9e2 (127.0.97.1:37525), 02560cfb3ea043278d48f4866bde3bc3 (127.0.97.3:37849)
I20260926 23:31:13.114297 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a58674195b564589a1707c36138e6ba9" candidate_uuid: "2c827405b79b4eb9b15a6c0689928d2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f2a2b0541d400998e3e677502dc9e2"
I20260926 23:31:13.114311 8468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a58674195b564589a1707c36138e6ba9" candidate_uuid: "2c827405b79b4eb9b15a6c0689928d2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02560cfb3ea043278d48f4866bde3bc3"
I20260926 23:31:13.114549 8193 raft_consensus.cc:3037] T a58674195b564589a1707c36138e6ba9 P 35f2a2b0541d400998e3e677502dc9e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:31:13.114583 8468 raft_consensus.cc:3037] T a58674195b564589a1707c36138e6ba9 P 02560cfb3ea043278d48f4866bde3bc3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:31:13.117455 8193 raft_consensus.cc:2454] T a58674195b564589a1707c36138e6ba9 P 35f2a2b0541d400998e3e677502dc9e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c827405b79b4eb9b15a6c0689928d2c in term 1.
I20260926 23:31:13.117951 8468 raft_consensus.cc:2454] T a58674195b564589a1707c36138e6ba9 P 02560cfb3ea043278d48f4866bde3bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c827405b79b4eb9b15a6c0689928d2c in term 1.
I20260926 23:31:13.118074 8261 leader_election.cc:302] T a58674195b564589a1707c36138e6ba9 P 2c827405b79b4eb9b15a6c0689928d2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c827405b79b4eb9b15a6c0689928d2c, 35f2a2b0541d400998e3e677502dc9e2; no voters:
I20260926 23:31:13.118471 8551 raft_consensus.cc:2781] T a58674195b564589a1707c36138e6ba9 P 2c827405b79b4eb9b15a6c0689928d2c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:31:13.118732 8551 raft_consensus.cc:686] T a58674195b564589a1707c36138e6ba9 P 2c827405b79b4eb9b15a6c0689928d2c [term 1 LEADER]: Becoming Leader. State: Replica: 2c827405b79b4eb9b15a6c0689928d2c, State: Running, Role: LEADER
I20260926 23:31:13.119148 8551 consensus_queue.cc:237] T a58674195b564589a1707c36138e6ba9 P 2c827405b79b4eb9b15a6c0689928d2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:13.122622 8031 catalog_manager.cc:5495] T a58674195b564589a1707c36138e6ba9 P 2c827405b79b4eb9b15a6c0689928d2c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c827405b79b4eb9b15a6c0689928d2c (127.0.97.2). New cstate: current_term: 1 leader_uuid: "2c827405b79b4eb9b15a6c0689928d2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } health_report { overall_health: UNKNOWN } } }
I20260926 23:31:13.144459 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:13.137858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45720 (local address 127.0.97.2:42343)
0926 23:31:13.138082 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:13.138090 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:13.138112 (+ 22us) server_negotiation.cc:407] Connection header received
0926 23:31:13.138394 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:13.138398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:13.138473 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:13.138559 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:31:13.139089 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:13.139999 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:13.142314 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:13.142904 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:13.143024 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:13.143555 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:31:13.143693 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:31:13.143763 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:13.143938 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 23:31:13.144019 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 23:31:13.148007 8193 raft_consensus.cc:1261] T a58674195b564589a1707c36138e6ba9 P 35f2a2b0541d400998e3e677502dc9e2 [term 1 FOLLOWER]: Refusing update from remote peer 2c827405b79b4eb9b15a6c0689928d2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:31:13.148021 8468 raft_consensus.cc:1261] T a58674195b564589a1707c36138e6ba9 P 02560cfb3ea043278d48f4866bde3bc3 [term 1 FOLLOWER]: Refusing update from remote peer 2c827405b79b4eb9b15a6c0689928d2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:31:13.148746 8551 consensus_queue.cc:1035] T a58674195b564589a1707c36138e6ba9 P 2c827405b79b4eb9b15a6c0689928d2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:31:13.149122 8592 consensus_queue.cc:1035] T a58674195b564589a1707c36138e6ba9 P 2c827405b79b4eb9b15a6c0689928d2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:31:13.170413 8032 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:31:13.170826 8032 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:31:13.174502 8032 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 55da4609224d44989fd4e860066c0076: Sending DeleteTablet for 3 replicas of tablet a58674195b564589a1707c36138e6ba9
I20260926 23:31:13.175881 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet a58674195b564589a1707c36138e6ba9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:31:13 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45690
I20260926 23:31:13.175848 8448 tablet_service.cc:1514] Processing DeleteTablet for tablet a58674195b564589a1707c36138e6ba9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:31:13 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49896
I20260926 23:31:13.176196 8604 tablet_replica.cc:330] T a58674195b564589a1707c36138e6ba9 P 02560cfb3ea043278d48f4866bde3bc3: stopping tablet replica
I20260926 23:31:13.176384 8605 tablet_replica.cc:330] T a58674195b564589a1707c36138e6ba9 P 2c827405b79b4eb9b15a6c0689928d2c: stopping tablet replica
I20260926 23:31:13.176395 8604 raft_consensus.cc:2229] T a58674195b564589a1707c36138e6ba9 P 02560cfb3ea043278d48f4866bde3bc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:31:13.176637 8604 raft_consensus.cc:2258] T a58674195b564589a1707c36138e6ba9 P 02560cfb3ea043278d48f4866bde3bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:31:13.176620 8605 raft_consensus.cc:2229] T a58674195b564589a1707c36138e6ba9 P 2c827405b79b4eb9b15a6c0689928d2c [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:31:13.177026 8605 raft_consensus.cc:2258] T a58674195b564589a1707c36138e6ba9 P 2c827405b79b4eb9b15a6c0689928d2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:31:13.176966 8032 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 23:31:13.177716 8169 tablet_service.cc:1514] Processing DeleteTablet for tablet a58674195b564589a1707c36138e6ba9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:31:13 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39226
W20260926 23:31:13.177798 8032 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:31:13.178033 8603 tablet_replica.cc:330] T a58674195b564589a1707c36138e6ba9 P 35f2a2b0541d400998e3e677502dc9e2: stopping tablet replica
I20260926 23:31:13.178248 8603 raft_consensus.cc:2229] T a58674195b564589a1707c36138e6ba9 P 35f2a2b0541d400998e3e677502dc9e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:31:13.178349 8605 ts_tablet_manager.cc:1905] T a58674195b564589a1707c36138e6ba9 P 2c827405b79b4eb9b15a6c0689928d2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:31:13.178471 8603 raft_consensus.cc:2258] T a58674195b564589a1707c36138e6ba9 P 35f2a2b0541d400998e3e677502dc9e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:31:13.180011 8603 ts_tablet_manager.cc:1905] T a58674195b564589a1707c36138e6ba9 P 35f2a2b0541d400998e3e677502dc9e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:31:13.184228 8605 ts_tablet_manager.cc:1918] T a58674195b564589a1707c36138e6ba9 P 2c827405b79b4eb9b15a6c0689928d2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:31:13.184659 8605 log.cc:1196] T a58674195b564589a1707c36138e6ba9 P 2c827405b79b4eb9b15a6c0689928d2c: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-1/wal/wals/a58674195b564589a1707c36138e6ba9
I20260926 23:31:13.185099 8605 ts_tablet_manager.cc:1939] T a58674195b564589a1707c36138e6ba9 P 2c827405b79b4eb9b15a6c0689928d2c: Deleting consensus metadata
I20260926 23:31:13.185722 8604 ts_tablet_manager.cc:1905] T a58674195b564589a1707c36138e6ba9 P 02560cfb3ea043278d48f4866bde3bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:31:13.187207 8018 catalog_manager.cc:4841] TS 2c827405b79b4eb9b15a6c0689928d2c (127.0.97.2:42343): tablet a58674195b564589a1707c36138e6ba9 (table test-table [id=56f99fc31aeb4fccb65be521f2e61a07]) successfully deleted
I20260926 23:31:13.192255 8604 ts_tablet_manager.cc:1918] T a58674195b564589a1707c36138e6ba9 P 02560cfb3ea043278d48f4866bde3bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:31:13.192524 8604 log.cc:1196] T a58674195b564589a1707c36138e6ba9 P 02560cfb3ea043278d48f4866bde3bc3: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-2/wal/wals/a58674195b564589a1707c36138e6ba9
I20260926 23:31:13.192605 8448 tablet_service.cc:1467] Processing CreateTablet for tablet 1ff7f2a3fcaf401c9fe04966ab7992cb (DEFAULT_TABLE table=test-table [id=4abd275b9dff4da181047948f29fae32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:31:13.193082 8604 ts_tablet_manager.cc:1939] T a58674195b564589a1707c36138e6ba9 P 02560cfb3ea043278d48f4866bde3bc3: Deleting consensus metadata
I20260926 23:31:13.193110 8448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ff7f2a3fcaf401c9fe04966ab7992cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:31:13.193480 8603 ts_tablet_manager.cc:1918] T a58674195b564589a1707c36138e6ba9 P 35f2a2b0541d400998e3e677502dc9e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:31:13.193747 8603 log.cc:1196] T a58674195b564589a1707c36138e6ba9 P 35f2a2b0541d400998e3e677502dc9e2: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-0/wal/wals/a58674195b564589a1707c36138e6ba9
I20260926 23:31:13.194268 8603 ts_tablet_manager.cc:1939] T a58674195b564589a1707c36138e6ba9 P 35f2a2b0541d400998e3e677502dc9e2: Deleting consensus metadata
I20260926 23:31:13.194736 8169 tablet_service.cc:1467] Processing CreateTablet for tablet 1ff7f2a3fcaf401c9fe04966ab7992cb (DEFAULT_TABLE table=test-table [id=4abd275b9dff4da181047948f29fae32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:31:13.195171 8169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ff7f2a3fcaf401c9fe04966ab7992cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:31:13.195924 8309 tablet_service.cc:1467] Processing CreateTablet for tablet 1ff7f2a3fcaf401c9fe04966ab7992cb (DEFAULT_TABLE table=test-table [id=4abd275b9dff4da181047948f29fae32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:31:13.196420 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ff7f2a3fcaf401c9fe04966ab7992cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:31:13.199056 8018 catalog_manager.cc:4841] TS 35f2a2b0541d400998e3e677502dc9e2 (127.0.97.1:37525): tablet a58674195b564589a1707c36138e6ba9 (table test-table [id=56f99fc31aeb4fccb65be521f2e61a07]) successfully deleted
I20260926 23:31:13.199569 8543 tablet_bootstrap.cc:492] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 02560cfb3ea043278d48f4866bde3bc3: Bootstrap starting.
I20260926 23:31:13.201342 8543 tablet_bootstrap.cc:654] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 02560cfb3ea043278d48f4866bde3bc3: Neither blocks nor log segments found. Creating new log.
I20260926 23:31:13.203678 8544 tablet_bootstrap.cc:492] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 2c827405b79b4eb9b15a6c0689928d2c: Bootstrap starting.
I20260926 23:31:13.205742 8544 tablet_bootstrap.cc:654] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 2c827405b79b4eb9b15a6c0689928d2c: Neither blocks nor log segments found. Creating new log.
I20260926 23:31:13.206009 8545 tablet_bootstrap.cc:492] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 35f2a2b0541d400998e3e677502dc9e2: Bootstrap starting.
I20260926 23:31:13.207644 8545 tablet_bootstrap.cc:654] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 35f2a2b0541d400998e3e677502dc9e2: Neither blocks nor log segments found. Creating new log.
I20260926 23:31:13.211434 8017 catalog_manager.cc:4841] TS 02560cfb3ea043278d48f4866bde3bc3 (127.0.97.3:37849): tablet a58674195b564589a1707c36138e6ba9 (table test-table [id=56f99fc31aeb4fccb65be521f2e61a07]) successfully deleted
I20260926 23:31:13.213382 8543 tablet_bootstrap.cc:492] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 02560cfb3ea043278d48f4866bde3bc3: No bootstrap required, opened a new log
I20260926 23:31:13.213604 8543 ts_tablet_manager.cc:1397] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 02560cfb3ea043278d48f4866bde3bc3: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260926 23:31:13.214641 8543 raft_consensus.cc:348] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 02560cfb3ea043278d48f4866bde3bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:13.214996 8543 raft_consensus.cc:374] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 02560cfb3ea043278d48f4866bde3bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:31:13.215209 8543 raft_consensus.cc:729] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 02560cfb3ea043278d48f4866bde3bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02560cfb3ea043278d48f4866bde3bc3, State: Initialized, Role: FOLLOWER
I20260926 23:31:13.215514 8543 consensus_queue.cc:260] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 02560cfb3ea043278d48f4866bde3bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:13.217546 8543 ts_tablet_manager.cc:1428] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 02560cfb3ea043278d48f4866bde3bc3: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 23:31:13.218633 8545 tablet_bootstrap.cc:492] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 35f2a2b0541d400998e3e677502dc9e2: No bootstrap required, opened a new log
I20260926 23:31:13.218883 8544 tablet_bootstrap.cc:492] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 2c827405b79b4eb9b15a6c0689928d2c: No bootstrap required, opened a new log
I20260926 23:31:13.218921 8545 ts_tablet_manager.cc:1397] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 35f2a2b0541d400998e3e677502dc9e2: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.006s
I20260926 23:31:13.219089 8544 ts_tablet_manager.cc:1397] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 2c827405b79b4eb9b15a6c0689928d2c: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.005s
I20260926 23:31:13.219801 8544 raft_consensus.cc:348] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:13.220067 8544 raft_consensus.cc:374] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:31:13.219959 8545 raft_consensus.cc:348] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 35f2a2b0541d400998e3e677502dc9e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:13.220155 8544 raft_consensus.cc:729] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c827405b79b4eb9b15a6c0689928d2c, State: Initialized, Role: FOLLOWER
I20260926 23:31:13.220223 8545 raft_consensus.cc:374] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 35f2a2b0541d400998e3e677502dc9e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:31:13.220311 8545 raft_consensus.cc:729] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 35f2a2b0541d400998e3e677502dc9e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f2a2b0541d400998e3e677502dc9e2, State: Initialized, Role: FOLLOWER
I20260926 23:31:13.220430 8544 consensus_queue.cc:260] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 2c827405b79b4eb9b15a6c0689928d2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:13.220558 8545 consensus_queue.cc:260] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 35f2a2b0541d400998e3e677502dc9e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:13.220984 8544 ts_tablet_manager.cc:1428] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 2c827405b79b4eb9b15a6c0689928d2c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 23:31:13.221107 8545 ts_tablet_manager.cc:1428] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 35f2a2b0541d400998e3e677502dc9e2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 23:31:13.279347 8549 raft_consensus.cc:482] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 02560cfb3ea043278d48f4866bde3bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:31:13.279657 8549 raft_consensus.cc:504] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 02560cfb3ea043278d48f4866bde3bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:13.281634 8549 leader_election.cc:288] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 02560cfb3ea043278d48f4866bde3bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35f2a2b0541d400998e3e677502dc9e2 (127.0.97.1:37525), 2c827405b79b4eb9b15a6c0689928d2c (127.0.97.2:42343)
I20260926 23:31:13.288769 8628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:13.281927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37525 (local address 127.0.97.3:48833)
0926 23:31:13.282580 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:13.282600 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:13.282807 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:13.283251 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:13.283257 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:13.283289 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:31:13.283601 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:13.283611 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:13.284830 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:13.284835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:13.286443 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:13.286458 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:13.287892 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:13.287898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:13.288112 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:13.288149 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 23:31:13.288260 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 23:31:13.288423 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":349,"threads_started":1}
I20260926 23:31:13.288751 8629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:13.282026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48833 (local address 127.0.97.1:37525)
0926 23:31:13.282721 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:13.282729 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:13.282752 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:31:13.282977 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:13.282984 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:13.283068 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:13.283178 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:31:13.283792 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:13.284671 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:13.286627 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:13.287671 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:13.287811 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:13.287957 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:13.288306 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 23:31:13.288433 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":341,"threads_started":1}
I20260926 23:31:13.289642 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ff7f2a3fcaf401c9fe04966ab7992cb" candidate_uuid: "02560cfb3ea043278d48f4866bde3bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f2a2b0541d400998e3e677502dc9e2" is_pre_election: true
I20260926 23:31:13.289817 8630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:13.282755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42343 (local address 127.0.97.3:34287)
0926 23:31:13.283457 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:13.283472 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:13.283620 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:13.283926 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:13.283930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:13.283955 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:31:13.284284 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:13.284294 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:13.285278 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:13.285284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:13.286797 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:13.286809 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:13.288120 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:13.288126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:13.288808 (+ 682us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:13.289257 (+ 449us) client_negotiation.cc:769] Sending connection context
0926 23:31:13.289357 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 23:31:13.289519 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":373,"threads_started":1}
I20260926 23:31:13.289896 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:13.282921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34287 (local address 127.0.97.2:42343)
0926 23:31:13.283146 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:13.283152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:13.283574 (+ 422us) server_negotiation.cc:407] Connection header received
0926 23:31:13.283751 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:13.283756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:13.283821 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:13.283899 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:31:13.284425 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:13.285153 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:13.286951 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:13.287895 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:13.288013 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:13.288119 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:13.289438 (+ 1319us) server_negotiation.cc:299] Negotiation successful
0926 23:31:13.289620 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 23:31:13.289984 8193 raft_consensus.cc:2454] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 35f2a2b0541d400998e3e677502dc9e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02560cfb3ea043278d48f4866bde3bc3 in term 0.
I20260926 23:31:13.290553 8401 leader_election.cc:302] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 02560cfb3ea043278d48f4866bde3bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02560cfb3ea043278d48f4866bde3bc3, 35f2a2b0541d400998e3e677502dc9e2; no voters:
I20260926 23:31:13.290737 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ff7f2a3fcaf401c9fe04966ab7992cb" candidate_uuid: "02560cfb3ea043278d48f4866bde3bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c827405b79b4eb9b15a6c0689928d2c" is_pre_election: true
I20260926 23:31:13.291018 8549 raft_consensus.cc:2781] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 02560cfb3ea043278d48f4866bde3bc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:31:13.291136 8549 raft_consensus.cc:482] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 02560cfb3ea043278d48f4866bde3bc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:31:13.291225 8549 raft_consensus.cc:3037] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 02560cfb3ea043278d48f4866bde3bc3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:31:13.291182 8329 raft_consensus.cc:2454] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02560cfb3ea043278d48f4866bde3bc3 in term 0.
I20260926 23:31:13.293963 8549 raft_consensus.cc:504] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 02560cfb3ea043278d48f4866bde3bc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:13.294647 8549 leader_election.cc:288] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 02560cfb3ea043278d48f4866bde3bc3 [CANDIDATE]: Term 1 election: Requested vote from peers 35f2a2b0541d400998e3e677502dc9e2 (127.0.97.1:37525), 2c827405b79b4eb9b15a6c0689928d2c (127.0.97.2:42343)
I20260926 23:31:13.295384 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ff7f2a3fcaf401c9fe04966ab7992cb" candidate_uuid: "02560cfb3ea043278d48f4866bde3bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f2a2b0541d400998e3e677502dc9e2"
I20260926 23:31:13.295383 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ff7f2a3fcaf401c9fe04966ab7992cb" candidate_uuid: "02560cfb3ea043278d48f4866bde3bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c827405b79b4eb9b15a6c0689928d2c"
I20260926 23:31:13.295634 8329 raft_consensus.cc:3037] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 2c827405b79b4eb9b15a6c0689928d2c [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:31:13.295630 8193 raft_consensus.cc:3037] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 35f2a2b0541d400998e3e677502dc9e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:31:13.298872 8329 raft_consensus.cc:2454] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 2c827405b79b4eb9b15a6c0689928d2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02560cfb3ea043278d48f4866bde3bc3 in term 1.
I20260926 23:31:13.298872 8193 raft_consensus.cc:2454] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 35f2a2b0541d400998e3e677502dc9e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02560cfb3ea043278d48f4866bde3bc3 in term 1.
I20260926 23:31:13.299458 8401 leader_election.cc:302] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 02560cfb3ea043278d48f4866bde3bc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02560cfb3ea043278d48f4866bde3bc3, 2c827405b79b4eb9b15a6c0689928d2c; no voters:
I20260926 23:31:13.299850 8549 raft_consensus.cc:2781] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 02560cfb3ea043278d48f4866bde3bc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:31:13.300346 8549 raft_consensus.cc:686] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 02560cfb3ea043278d48f4866bde3bc3 [term 1 LEADER]: Becoming Leader. State: Replica: 02560cfb3ea043278d48f4866bde3bc3, State: Running, Role: LEADER
I20260926 23:31:13.300783 8549 consensus_queue.cc:237] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 02560cfb3ea043278d48f4866bde3bc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } }
I20260926 23:31:13.305335 8031 catalog_manager.cc:5495] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 02560cfb3ea043278d48f4866bde3bc3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02560cfb3ea043278d48f4866bde3bc3 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "02560cfb3ea043278d48f4866bde3bc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02560cfb3ea043278d48f4866bde3bc3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37849 } health_report { overall_health: HEALTHY } } }
I20260926 23:31:13.359916 8633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:13.352684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49900 (local address 127.0.97.3:37849)
0926 23:31:13.353416 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:13.353422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:13.353441 (+ 19us) server_negotiation.cc:407] Connection header received
0926 23:31:13.353554 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:13.353558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:13.353673 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:13.353855 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:31:13.354362 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:13.355320 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:13.356987 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:13.357478 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:13.357607 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:13.358196 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:31:13.359086 (+ 890us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:31:13.359212 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:13.359364 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 23:31:13.359455 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":439,"threads_started":1}
I20260926 23:31:13.378300 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:13.372486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35799 (local address 127.0.97.2:42343)
0926 23:31:13.372719 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:13.372727 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:13.373304 (+ 577us) server_negotiation.cc:407] Connection header received
0926 23:31:13.373514 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:13.373520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:13.373604 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:13.373689 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:31:13.374158 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:13.374982 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:13.376425 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:13.376864 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:13.376965 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:13.377484 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:31:13.377640 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:31:13.377729 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:13.377893 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 23:31:13.377975 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 23:31:13.378338 8636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:13.372412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42343 (local address 127.0.97.3:35799)
0926 23:31:13.373130 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:13.373148 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:13.373347 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:13.373713 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:13.373717 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:13.373727 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:31:13.374018 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:13.374027 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:13.375098 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:13.375102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:13.376279 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:13.376293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:13.377028 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:13.377032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:13.377165 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:13.377268 (+ 103us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:31:13.377746 (+ 478us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:31:13.377756 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 23:31:13.377830 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 23:31:13.377973 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":372,"threads_started":1}
I20260926 23:31:13.388051 8193 raft_consensus.cc:1261] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 35f2a2b0541d400998e3e677502dc9e2 [term 1 FOLLOWER]: Refusing update from remote peer 02560cfb3ea043278d48f4866bde3bc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:31:13.388330 8329 raft_consensus.cc:1261] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 2c827405b79b4eb9b15a6c0689928d2c [term 1 FOLLOWER]: Refusing update from remote peer 02560cfb3ea043278d48f4866bde3bc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:31:13.388689 8631 consensus_queue.cc:1035] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 02560cfb3ea043278d48f4866bde3bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35f2a2b0541d400998e3e677502dc9e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:31:13.389043 8549 consensus_queue.cc:1035] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 02560cfb3ea043278d48f4866bde3bc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c827405b79b4eb9b15a6c0689928d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:31:13.419119 8633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:13.412904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47771 (local address 127.0.97.3:37849)
0926 23:31:13.413139 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:13.413147 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:13.413586 (+ 439us) server_negotiation.cc:407] Connection header received
0926 23:31:13.413776 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:13.413781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:13.413854 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:13.413926 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:31:13.414561 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:13.415410 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:13.417130 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:13.417715 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:13.417819 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:13.418275 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:31:13.418429 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:31:13.418531 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:13.418671 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 23:31:13.418802 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 23:31:13.419140 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:13.412853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37849 (local address 127.0.97.2:47771)
0926 23:31:13.413481 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:13.413498 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:13.413643 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:13.413950 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:13.413955 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:13.413966 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 23:31:13.414231 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:13.414244 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:13.415540 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:13.415546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:13.416897 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:13.416914 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:13.417856 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:13.417861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:13.418058 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:13.418108 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 23:31:13.418526 (+ 418us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 23:31:13.418538 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 23:31:13.418631 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 23:31:13.418834 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":309,"threads_started":1}
I20260926 23:31:13.460623 8030 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:31:13.460918 8030 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:31:13.464958 8030 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 55da4609224d44989fd4e860066c0076: Sending DeleteTablet for 3 replicas of tablet 1ff7f2a3fcaf401c9fe04966ab7992cb
I20260926 23:31:13.466146 8169 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ff7f2a3fcaf401c9fe04966ab7992cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:31:13 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39226
I20260926 23:31:13.466180 8448 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ff7f2a3fcaf401c9fe04966ab7992cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:31:13 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49896
I20260926 23:31:13.466527 8603 tablet_replica.cc:330] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 35f2a2b0541d400998e3e677502dc9e2: stopping tablet replica
I20260926 23:31:13.466810 8604 tablet_replica.cc:330] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 02560cfb3ea043278d48f4866bde3bc3: stopping tablet replica
I20260926 23:31:13.466794 8603 raft_consensus.cc:2229] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 35f2a2b0541d400998e3e677502dc9e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:31:13.467001 8604 raft_consensus.cc:2229] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 02560cfb3ea043278d48f4866bde3bc3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:31:13.467043 8603 raft_consensus.cc:2258] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 35f2a2b0541d400998e3e677502dc9e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:31:13.467613 8604 raft_consensus.cc:2258] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 02560cfb3ea043278d48f4866bde3bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:31:13.468379 8603 ts_tablet_manager.cc:1905] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 35f2a2b0541d400998e3e677502dc9e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:31:13.468851 8604 ts_tablet_manager.cc:1905] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 02560cfb3ea043278d48f4866bde3bc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:31:13.469209 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 8103
I20260926 23:31:13.470805 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ff7f2a3fcaf401c9fe04966ab7992cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:31:13 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45690
I20260926 23:31:13.471342 8605 tablet_replica.cc:330] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 2c827405b79b4eb9b15a6c0689928d2c: stopping tablet replica
I20260926 23:31:13.471577 8605 raft_consensus.cc:2229] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 2c827405b79b4eb9b15a6c0689928d2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:31:13.471830 8605 raft_consensus.cc:2258] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 2c827405b79b4eb9b15a6c0689928d2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:31:13.473143 8605 ts_tablet_manager.cc:1905] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 2c827405b79b4eb9b15a6c0689928d2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:31:13.477229 8603 ts_tablet_manager.cc:1918] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 35f2a2b0541d400998e3e677502dc9e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:31:13.477236 8604 ts_tablet_manager.cc:1918] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 02560cfb3ea043278d48f4866bde3bc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:31:13.477429 8604 log.cc:1196] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 02560cfb3ea043278d48f4866bde3bc3: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-2/wal/wals/1ff7f2a3fcaf401c9fe04966ab7992cb
I20260926 23:31:13.477644 8603 log.cc:1196] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 35f2a2b0541d400998e3e677502dc9e2: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-0/wal/wals/1ff7f2a3fcaf401c9fe04966ab7992cb
I20260926 23:31:13.477950 8604 ts_tablet_manager.cc:1939] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 02560cfb3ea043278d48f4866bde3bc3: Deleting consensus metadata
I20260926 23:31:13.478271 8603 ts_tablet_manager.cc:1939] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 35f2a2b0541d400998e3e677502dc9e2: Deleting consensus metadata
I20260926 23:31:13.479416 8017 catalog_manager.cc:4841] TS 02560cfb3ea043278d48f4866bde3bc3 (127.0.97.3:37849): tablet 1ff7f2a3fcaf401c9fe04966ab7992cb (table test-table [id=4abd275b9dff4da181047948f29fae32]) successfully deleted
I20260926 23:31:13.480355 8018 catalog_manager.cc:4841] TS 35f2a2b0541d400998e3e677502dc9e2 (127.0.97.1:37525): tablet 1ff7f2a3fcaf401c9fe04966ab7992cb (table test-table [id=4abd275b9dff4da181047948f29fae32]) successfully deleted
I20260926 23:31:13.482795 8605 ts_tablet_manager.cc:1918] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 2c827405b79b4eb9b15a6c0689928d2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 23:31:13.482963 8605 log.cc:1196] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 2c827405b79b4eb9b15a6c0689928d2c: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790465400694755-388-0/minicluster-data/ts-1/wal/wals/1ff7f2a3fcaf401c9fe04966ab7992cb
I20260926 23:31:13.483316 8605 ts_tablet_manager.cc:1939] T 1ff7f2a3fcaf401c9fe04966ab7992cb P 2c827405b79b4eb9b15a6c0689928d2c: Deleting consensus metadata
I20260926 23:31:13.484607 8018 catalog_manager.cc:4841] TS 2c827405b79b4eb9b15a6c0689928d2c (127.0.97.2:42343): tablet 1ff7f2a3fcaf401c9fe04966ab7992cb (table test-table [id=4abd275b9dff4da181047948f29fae32]) successfully deleted
Sep 26 23:31:13 dist-test-slave-gwfx krb5kdc[7987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260926 23:31:13.498804 8629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:13.470768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39232 (local address 127.0.97.1:37525)
0926 23:31:13.470983 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:13.470990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:13.479916 (+ 8926us) server_negotiation.cc:407] Connection header received
0926 23:31:13.480276 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:13.480282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:13.480388 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:13.480461 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:13.482403 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:13.483619 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:13.485287 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:13.485974 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:13.486115 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:13.490454 (+ 4339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:13.490484 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:13.490514 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:13.490579 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:13.495106 (+ 4527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:13.496097 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:13.496110 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:13.496114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:13.496332 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:13.496747 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:13.496753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:13.496757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:13.497349 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:13.497703 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:13.497953 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 23:31:13.498296 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 23:31:13.499646 8234 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:31:13.605559 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 8103
I20260926 23:31:13.632934 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 8243
Sep 26 23:31:13 dist-test-slave-gwfx krb5kdc[7987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260926 23:31:13.654218 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:13.634010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45722 (local address 127.0.97.2:42343)
0926 23:31:13.634306 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:13.634313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:13.634340 (+ 27us) server_negotiation.cc:407] Connection header received
0926 23:31:13.634428 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:13.634433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:13.634537 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:13.634732 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:13.636230 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:13.637391 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:13.639476 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:13.640261 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:13.640465 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:13.645383 (+ 4918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:13.645413 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:13.645441 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:13.645517 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:13.650409 (+ 4892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:13.651470 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:13.651482 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:13.651486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:13.651723 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:13.652214 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:13.652220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:13.652223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:13.652792 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:13.653168 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:13.653476 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 23:31:13.653794 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 23:31:13.655260 8374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:31:13.839496 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 8243
I20260926 23:31:13.878726 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 8383
Sep 26 23:31:13 dist-test-slave-gwfx krb5kdc[7987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260926 23:31:13.899843 8633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:13.880007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49908 (local address 127.0.97.3:37849)
0926 23:31:13.880191 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:13.880197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:13.880218 (+ 21us) server_negotiation.cc:407] Connection header received
0926 23:31:13.880305 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:13.880311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:13.880404 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:13.880517 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:13.882416 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:13.883829 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:13.885777 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:13.886535 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:13.886665 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:13.892496 (+ 5831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:13.892516 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:13.892539 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:13.892597 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:13.896445 (+ 3848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:13.897379 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:13.897388 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:13.897391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:13.897578 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:13.897997 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:13.898001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:13.898003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:13.898473 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:13.898856 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:13.899200 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 23:31:13.899474 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 23:31:13.900640 8514 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:31:14.050328 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 8383
I20260926 23:31:14.071632 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 8000
I20260926 23:31:14.073191 8072 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:31:14.202914 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 8000
2026-09-26T23:31:14Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6249 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:31:14 dist-test-slave-gwfx krb5kdc[8651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:31:14 dist-test-slave-gwfx krb5kdc[8651](info): set up 2 sockets
Sep 26 23:31:14 dist-test-slave-gwfx krb5kdc[8651](info): commencing operation
krb5kdc: starting...
W20260926 23:31:15.267928 388 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:31:15 dist-test-slave-gwfx krb5kdc[8651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:31:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:31:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:31:15.539435 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40183
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:31:15.924173 8664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:31:15.924618 8664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:31:15.924758 8664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:31:15.933080 8664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:31:15.933260 8664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:31:15.933339 8664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:31:15.933427 8664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:31:15.933498 8664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:31:15.945050 8664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:31:15.946826 8664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:31:15.948515 8664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:15.962656 8669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:15.962709 8670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:15.962739 8672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:15.963176 8664 server_base.cc:1034] running on GCE node
I20260926 23:31:15.964398 8664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:15.966388 8664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:15.967705 8664 hybrid_clock.cc:648] HybridClock initialized: now 1790465475967636 us; error 51 us; skew 500 ppm
I20260926 23:31:15.968226 8664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T23:31:16Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1837 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:31:16 dist-test-slave-gwfx krb5kdc[8678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:31:16 dist-test-slave-gwfx krb5kdc[8678](info): set up 2 sockets
Sep 26 23:31:16 dist-test-slave-gwfx krb5kdc[8678](info): commencing operation
krb5kdc: starting...
W20260926 23:31:17.087914 388 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:31:17 dist-test-slave-gwfx krb5kdc[8678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:31:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:31:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:31:17.363713 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34729
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34729
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:31:17.762115 8691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:31:17.762548 8691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:31:17.762655 8691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:31:17.771311 8691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:31:17.771476 8691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:31:17.771526 8691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:31:17.771590 8691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:31:17.771637 8691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:31:17.783032 8691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:31:17.784910 8691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:31:17.786500 8691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:17.799945 8696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:17.799945 8697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:17.799947 8699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:17.800927 8691 server_base.cc:1034] running on GCE node
I20260926 23:31:17.801980 8691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:17.803828 8691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:17.805064 8691 hybrid_clock.cc:648] HybridClock initialized: now 1790465477805009 us; error 41 us; skew 500 ppm
I20260926 23:31:17.805527 8691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:31:17 dist-test-slave-gwfx krb5kdc[8678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:31:17.809263 8691 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:31:17.811308 8691 webserver.cc:460] Webserver started at http://127.0.97.62:35179/ using document root <none> and password file <none>
I20260926 23:31:17.812239 8691 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:17.812373 8691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:17.812749 8691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:31:17.816778 8691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "3491e2c162f64c71b5aec7d228cfcd86"
format_stamp: "Formatted at 2026-09-26 23:31:17 on dist-test-slave-gwfx"
server_key: "f326789d7301f3cfe6ef70b8a11db12b"
server_key_iv: "60f0ec15ba2b7a6a3db0fc5882652480"
server_key_version: "encryptionkey@0"
I20260926 23:31:17.817884 8691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "3491e2c162f64c71b5aec7d228cfcd86"
format_stamp: "Formatted at 2026-09-26 23:31:17 on dist-test-slave-gwfx"
server_key: "f326789d7301f3cfe6ef70b8a11db12b"
server_key_iv: "60f0ec15ba2b7a6a3db0fc5882652480"
server_key_version: "encryptionkey@0"
I20260926 23:31:17.824920 8691 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20260926 23:31:17.830371 8706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:17.832583 8691 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260926 23:31:17.832840 8691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "3491e2c162f64c71b5aec7d228cfcd86"
format_stamp: "Formatted at 2026-09-26 23:31:17 on dist-test-slave-gwfx"
server_key: "f326789d7301f3cfe6ef70b8a11db12b"
server_key_iv: "60f0ec15ba2b7a6a3db0fc5882652480"
server_key_version: "encryptionkey@0"
I20260926 23:31:17.833105 8691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:17.859400 8691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:17.864972 8691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:31:17.865370 8691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:17.886319 8691 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34729
I20260926 23:31:17.886339 8757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34729 every 8 connection(s)
I20260926 23:31:17.887956 8691 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:31:17.892840 8758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:31:17.892894 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 8691
I20260926 23:31:17.893105 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:31:17.893819 388 external_mini_cluster.cc:1428] Setting key d90c52b7592bd9e5ccc55a928b379b01
I20260926 23:31:17.910241 8758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86: Bootstrap starting.
Sep 26 23:31:17 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:31:17.914438 8758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86: Neither blocks nor log segments found. Creating new log.
I20260926 23:31:17.916126 8758 log.cc:824] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86: Log is configured to *not* fsync() on all Append() calls
I20260926 23:31:17.920217 8758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86: No bootstrap required, opened a new log
I20260926 23:31:17.922775 8761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:17.896278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56264 (local address 127.0.97.62:34729)
0926 23:31:17.897212 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:17.897231 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:17.897297 (+ 66us) server_negotiation.cc:407] Connection header received
0926 23:31:17.900294 (+ 2997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:17.900344 (+ 50us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:17.901384 (+ 1040us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:17.902143 (+ 759us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:17.903808 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:17.906138 (+ 2330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:17.907763 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:17.909055 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:17.909280 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:17.913618 (+ 4338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:17.913631 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:17.913655 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:17.913717 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:17.917655 (+ 3938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:17.918467 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:17.918477 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:17.918487 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:17.918746 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:17.919094 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:17.919098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:17.919101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:17.919874 (+ 773us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:17.920190 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:17.920596 (+ 406us) server_negotiation.cc:299] Negotiation successful
0926 23:31:17.921178 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":375,"threads_started":1}
I20260926 23:31:17.930011 8758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3491e2c162f64c71b5aec7d228cfcd86" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34729 } }
I20260926 23:31:17.930490 8758 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:31:17.930570 8758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3491e2c162f64c71b5aec7d228cfcd86, State: Initialized, Role: FOLLOWER
I20260926 23:31:17.931284 8758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3491e2c162f64c71b5aec7d228cfcd86" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34729 } }
I20260926 23:31:17.931546 8758 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:31:17.931694 8758 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:31:17.931845 8758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:31:17.934803 8758 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3491e2c162f64c71b5aec7d228cfcd86" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34729 } }
I20260926 23:31:17.935456 8758 leader_election.cc:302] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3491e2c162f64c71b5aec7d228cfcd86; no voters:
I20260926 23:31:17.936116 8758 leader_election.cc:288] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:31:17.936292 8763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:31:17.937345 8763 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [term 1 LEADER]: Becoming Leader. State: Replica: 3491e2c162f64c71b5aec7d228cfcd86, State: Running, Role: LEADER
I20260926 23:31:17.938084 8763 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3491e2c162f64c71b5aec7d228cfcd86" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34729 } }
I20260926 23:31:17.938618 8758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:31:17.941715 8764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3491e2c162f64c71b5aec7d228cfcd86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3491e2c162f64c71b5aec7d228cfcd86" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34729 } } }
I20260926 23:31:17.942444 8764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [sys.catalog]: This master's current role is: LEADER
I20260926 23:31:17.941870 8765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3491e2c162f64c71b5aec7d228cfcd86. Latest consensus state: current_term: 1 leader_uuid: "3491e2c162f64c71b5aec7d228cfcd86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3491e2c162f64c71b5aec7d228cfcd86" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34729 } } }
I20260926 23:31:17.942754 8765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [sys.catalog]: This master's current role is: LEADER
I20260926 23:31:17.944702 8771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:31:17.950727 8771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:31:17.962570 8771 catalog_manager.cc:1347] Generated new cluster ID: 18aaaf2f6bac46709b0781a5a33d3022
I20260926 23:31:17.962769 8771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:31:17.976572 8771 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:31:17.978474 8771 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:31:17.987418 8771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86: Generated new TSK 0
I20260926 23:31:17.988811 8771 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:31:18.070647 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34729
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:31:18.522264 8784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:31:18.522647 8784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:31:18.522807 8784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:31:18.530901 8784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:31:18.531065 8784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:31:18.531229 8784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:31:18.542785 8784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:31:18.544626 8784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:31:18.546442 8784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:18.563334 8789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:18.563287 8792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:18.563256 8790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:18.564368 8784 server_base.cc:1034] running on GCE node
I20260926 23:31:18.565244 8784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:18.566463 8784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:18.567718 8784 hybrid_clock.cc:648] HybridClock initialized: now 1790465478567657 us; error 66 us; skew 500 ppm
I20260926 23:31:18.568192 8784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:31:18 dist-test-slave-gwfx krb5kdc[8678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:31:18.572367 8784 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:31:18.574347 8784 webserver.cc:460] Webserver started at http://127.0.97.1:36871/ using document root <none> and password file <none>
I20260926 23:31:18.575297 8784 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:18.575443 8784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:18.575806 8784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:31:18.579536 8784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-0/data/instance:
uuid: "a0477807ae4648c2a052c9d3c70c0ce7"
format_stamp: "Formatted at 2026-09-26 23:31:18 on dist-test-slave-gwfx"
server_key: "f20dcdbf58d967f59d01de32ee152bdf"
server_key_iv: "9bc9fd56b1e9cb71e53a73dedb18bd68"
server_key_version: "encryptionkey@0"
I20260926 23:31:18.580530 8784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a0477807ae4648c2a052c9d3c70c0ce7"
format_stamp: "Formatted at 2026-09-26 23:31:18 on dist-test-slave-gwfx"
server_key: "f20dcdbf58d967f59d01de32ee152bdf"
server_key_iv: "9bc9fd56b1e9cb71e53a73dedb18bd68"
server_key_version: "encryptionkey@0"
I20260926 23:31:18.587276 8784 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260926 23:31:18.591888 8799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:18.593717 8784 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 23:31:18.593924 8784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "a0477807ae4648c2a052c9d3c70c0ce7"
format_stamp: "Formatted at 2026-09-26 23:31:18 on dist-test-slave-gwfx"
server_key: "f20dcdbf58d967f59d01de32ee152bdf"
server_key_iv: "9bc9fd56b1e9cb71e53a73dedb18bd68"
server_key_version: "encryptionkey@0"
I20260926 23:31:18.594173 8784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:18.609745 8784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:18.615448 8784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:31:18.615823 8784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:18.617005 8784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:31:18.618836 8784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:31:18.618929 8784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:18.619078 8784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:31:18.619140 8784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:18.655751 8784 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:32877
I20260926 23:31:18.655759 8911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:32877 every 8 connection(s)
I20260926 23:31:18.657454 8784 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:31:18.666253 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 8784
I20260926 23:31:18.666481 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:31:18.667387 388 external_mini_cluster.cc:1428] Setting key d827e79572f34ddfb72bf418c43f01f5
Sep 26 23:31:18 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:31:18.684651 8914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:18.661043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42441 (local address 127.0.97.62:34729)
0926 23:31:18.661607 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:18.661613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:18.663191 (+ 1578us) server_negotiation.cc:407] Connection header received
0926 23:31:18.665557 (+ 2366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:18.665565 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:18.665737 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:18.665914 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:18.668413 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:18.669368 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:18.671655 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:18.672107 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:18.672250 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:18.677140 (+ 4890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:18.677148 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:18.677153 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:18.677200 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:18.680613 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:18.681656 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:18.681662 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:18.681666 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:18.681847 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:18.682369 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:18.682373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:18.682376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:18.682818 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:18.682956 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:18.683952 (+ 996us) server_negotiation.cc:299] Negotiation successful
0926 23:31:18.684245 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":282,"threads_started":1}
I20260926 23:31:18.685966 8915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:18.661903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.1:42441)
0926 23:31:18.662908 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:18.662961 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:18.665045 (+ 2084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:18.666316 (+ 1271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:18.666330 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:18.667198 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:18.668110 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:18.668148 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:18.669610 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:18.669615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:18.671422 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:18.671444 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:18.672416 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:18.672425 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:18.672731 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:18.673581 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:18.673610 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:18.676731 (+ 3121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:18.680877 (+ 4146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:18.680887 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:18.680913 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:18.681489 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:18.681949 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:18.681952 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:18.681955 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:18.682254 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:18.683030 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:18.683038 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:18.683520 (+ 482us) client_negotiation.cc:769] Sending connection context
0926 23:31:18.683806 (+ 286us) client_negotiation.cc:240] Negotiation successful
0926 23:31:18.684370 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":373,"threads_started":1}
I20260926 23:31:18.688125 8912 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34729
I20260926 23:31:18.688545 8912 heartbeater.cc:464] Registering TS with master...
I20260926 23:31:18.689522 8912 heartbeater.cc:511] Master 127.0.97.62:34729 requested a full tablet report, sending...
I20260926 23:31:18.692440 8723 ts_manager.cc:194] Registered new tserver with Master: a0477807ae4648c2a052c9d3c70c0ce7 (127.0.97.1:32877)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 23:31:18.694278 8723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:31:18.731544 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34729
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:31:19.140937 8918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:31:19.141378 8918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:31:19.141542 8918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:31:19.149974 8918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:31:19.150149 8918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:31:19.150321 8918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:31:19.162189 8918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:31:19.163965 8918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:31:19.165582 8918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:19.181528 8924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:19.181677 8923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:19.181571 8926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:19.182480 8918 server_base.cc:1034] running on GCE node
I20260926 23:31:19.183284 8918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:19.184544 8918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:19.185842 8918 hybrid_clock.cc:648] HybridClock initialized: now 1790465479185756 us; error 71 us; skew 500 ppm
I20260926 23:31:19.186545 8918 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:31:19 dist-test-slave-gwfx krb5kdc[8678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:31:19.190737 8918 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:31:19.192689 8918 webserver.cc:460] Webserver started at http://127.0.97.2:45285/ using document root <none> and password file <none>
I20260926 23:31:19.193626 8918 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:19.193764 8918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:19.194129 8918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:31:19.198006 8918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-1/data/instance:
uuid: "d46dda7151d44a6cafc6efb8945fc361"
format_stamp: "Formatted at 2026-09-26 23:31:19 on dist-test-slave-gwfx"
server_key: "f339b36763b7a4ee0404799e5a10171d"
server_key_iv: "7f87ebea46575a1bd9301a540e42f4dc"
server_key_version: "encryptionkey@0"
I20260926 23:31:19.199129 8918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d46dda7151d44a6cafc6efb8945fc361"
format_stamp: "Formatted at 2026-09-26 23:31:19 on dist-test-slave-gwfx"
server_key: "f339b36763b7a4ee0404799e5a10171d"
server_key_iv: "7f87ebea46575a1bd9301a540e42f4dc"
server_key_version: "encryptionkey@0"
I20260926 23:31:19.206734 8918 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s
I20260926 23:31:19.211982 8933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:19.213894 8918 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260926 23:31:19.214190 8918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "d46dda7151d44a6cafc6efb8945fc361"
format_stamp: "Formatted at 2026-09-26 23:31:19 on dist-test-slave-gwfx"
server_key: "f339b36763b7a4ee0404799e5a10171d"
server_key_iv: "7f87ebea46575a1bd9301a540e42f4dc"
server_key_version: "encryptionkey@0"
I20260926 23:31:19.214450 8918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:19.235793 8918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:19.241448 8918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:31:19.241796 8918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:19.243023 8918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:31:19.244733 8918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:31:19.244823 8918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:19.244963 8918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:31:19.245024 8918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:19.279558 8918 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40639
I20260926 23:31:19.279592 9045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40639 every 8 connection(s)
I20260926 23:31:19.281123 8918 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:31:19.281507 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 8918
I20260926 23:31:19.281697 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:31:19.282346 388 external_mini_cluster.cc:1428] Setting key d913994d499d8ec42e2e53b4703a3d37
Sep 26 23:31:19 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:31:19.306597 9048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:19.284316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59817 (local address 127.0.97.62:34729)
0926 23:31:19.284873 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:19.284879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:19.286484 (+ 1605us) server_negotiation.cc:407] Connection header received
0926 23:31:19.289050 (+ 2566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:19.289056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:19.289169 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:19.289306 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:19.291577 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:19.292663 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:19.294558 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:19.295102 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:19.295212 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:19.299709 (+ 4497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:19.299717 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:19.299723 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:19.299768 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:19.302952 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:19.303836 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:19.303841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:19.303845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:19.304000 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:19.304430 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:19.304434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:19.304437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:19.304830 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:19.304982 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:19.305857 (+ 875us) server_negotiation.cc:299] Negotiation successful
0926 23:31:19.306136 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":282,"threads_started":1}
I20260926 23:31:19.307724 9049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:19.285264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.2:59817)
0926 23:31:19.286290 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:19.286344 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:19.288627 (+ 2283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:19.289611 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:19.289624 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:19.290436 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:19.291347 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:19.291374 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:19.292820 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:19.292824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:19.294387 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:19.294401 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:19.295364 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:19.295371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:19.295687 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:19.296480 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:19.296526 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:19.299420 (+ 2894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:19.303131 (+ 3711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:19.303140 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:19.303158 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:19.303701 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:19.304109 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:19.304112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:19.304114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:19.304325 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:19.304993 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:19.305001 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:19.305433 (+ 432us) client_negotiation.cc:769] Sending connection context
0926 23:31:19.305715 (+ 282us) client_negotiation.cc:240] Negotiation successful
0926 23:31:19.306224 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":447,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 23:31:19.309691 9046 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34729
I20260926 23:31:19.310112 9046 heartbeater.cc:464] Registering TS with master...
I20260926 23:31:19.311237 9046 heartbeater.cc:511] Master 127.0.97.62:34729 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 23:31:19.313407 8723 ts_manager.cc:194] Registered new tserver with Master: d46dda7151d44a6cafc6efb8945fc361 (127.0.97.2:40639)
I20260926 23:31:19.314487 8723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59817
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:31:19.342072 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34729
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:31:19.699163 8912 heartbeater.cc:503] Master 127.0.97.62:34729 was elected leader, sending a full tablet report...
W20260926 23:31:19.751577 9052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:31:19.751986 9052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:31:19.752130 9052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:31:19.761224 9052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:31:19.761411 9052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:31:19.761583 9052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:31:19.773186 9052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:31:19.774988 9052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:31:19.776747 9052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:19.793200 9057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:19.793098 9060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:19.793208 9058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:19.794173 9052 server_base.cc:1034] running on GCE node
I20260926 23:31:19.795040 9052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:19.796469 9052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:19.797701 9052 hybrid_clock.cc:648] HybridClock initialized: now 1790465479797628 us; error 62 us; skew 500 ppm
I20260926 23:31:19.798058 9052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:31:19 dist-test-slave-gwfx krb5kdc[8678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:31:19.801939 9052 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:31:19.803993 9052 webserver.cc:460] Webserver started at http://127.0.97.3:40825/ using document root <none> and password file <none>
I20260926 23:31:19.804941 9052 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:19.805081 9052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:19.805470 9052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:31:19.809580 9052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-2/data/instance:
uuid: "7c9c9c92411544b5aeef2eba3c33df3a"
format_stamp: "Formatted at 2026-09-26 23:31:19 on dist-test-slave-gwfx"
server_key: "5673f93a1bcafcca27160376388c9ad2"
server_key_iv: "1f532860dd55cbe88dfd8c7a627edd85"
server_key_version: "encryptionkey@0"
I20260926 23:31:19.810725 9052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7c9c9c92411544b5aeef2eba3c33df3a"
format_stamp: "Formatted at 2026-09-26 23:31:19 on dist-test-slave-gwfx"
server_key: "5673f93a1bcafcca27160376388c9ad2"
server_key_iv: "1f532860dd55cbe88dfd8c7a627edd85"
server_key_version: "encryptionkey@0"
I20260926 23:31:19.818569 9052 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s
I20260926 23:31:19.826040 9067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:19.828385 9052 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260926 23:31:19.828673 9052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "7c9c9c92411544b5aeef2eba3c33df3a"
format_stamp: "Formatted at 2026-09-26 23:31:19 on dist-test-slave-gwfx"
server_key: "5673f93a1bcafcca27160376388c9ad2"
server_key_iv: "1f532860dd55cbe88dfd8c7a627edd85"
server_key_version: "encryptionkey@0"
I20260926 23:31:19.828941 9052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:19.846217 9052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:19.851783 9052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:31:19.852131 9052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:19.853348 9052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:31:19.855065 9052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:31:19.855149 9052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:19.855288 9052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:31:19.855341 9052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:19.890142 9052 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45307
I20260926 23:31:19.890298 9179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45307 every 8 connection(s)
I20260926 23:31:19.891861 9052 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:31:19.901006 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 9052
I20260926 23:31:19.901190 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:31:19.901711 388 external_mini_cluster.cc:1428] Setting key 7c59d31031e0d6e00d3c295c12a6b0f8
Sep 26 23:31:19 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:31:19.917927 9182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:19.895424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41365 (local address 127.0.97.62:34729)
0926 23:31:19.895938 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:19.895944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:19.897302 (+ 1358us) server_negotiation.cc:407] Connection header received
0926 23:31:19.899678 (+ 2376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:19.899685 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:19.899795 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:19.899903 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:19.902056 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:19.903077 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:19.905119 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:19.905800 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:19.905917 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:19.910534 (+ 4617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:19.910542 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:19.910546 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:19.910598 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:19.913575 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:19.914599 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:19.914605 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:19.914607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:19.914845 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:19.915465 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:19.915470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:19.915472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:19.915797 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:19.915923 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:19.917272 (+ 1349us) server_negotiation.cc:299] Negotiation successful
0926 23:31:19.917588 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":245,"threads_started":1}
I20260926 23:31:19.919795 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:19.896071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.3:41365)
0926 23:31:19.897077 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:19.897137 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:19.899296 (+ 2159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:19.900227 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:19.900238 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:19.900921 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:19.901736 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:19.901758 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:19.903261 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:19.903268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:19.904927 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:19.904944 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:19.906041 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:19.906047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:19.906316 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:19.907204 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:19.907234 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:19.910190 (+ 2956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:19.913816 (+ 3626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:19.913828 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:19.913848 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:19.914422 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:19.915043 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:19.915048 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:19.915050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:19.915328 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:19.916071 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:19.916083 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:19.916661 (+ 578us) client_negotiation.cc:769] Sending connection context
0926 23:31:19.917113 (+ 452us) client_negotiation.cc:240] Negotiation successful
0926 23:31:19.917860 (+ 747us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":402,"threads_started":1}
I20260926 23:31:19.921965 9180 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34729
I20260926 23:31:19.922435 9180 heartbeater.cc:464] Registering TS with master...
I20260926 23:31:19.923493 9180 heartbeater.cc:511] Master 127.0.97.62:34729 requested a full tablet report, sending...
I20260926 23:31:19.925585 8723 ts_manager.cc:194] Registered new tserver with Master: 7c9c9c92411544b5aeef2eba3c33df3a (127.0.97.3:45307)
I20260926 23:31:19.926642 8723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41365
I20260926 23:31:19.932848 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260926 23:31:20.111085 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 8784
Sep 26 23:31:20 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:31:20.132408 9191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:20.112380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60474 (local address 127.0.97.1:32877)
0926 23:31:20.113050 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:20.113058 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:20.113117 (+ 59us) server_negotiation.cc:407] Connection header received
0926 23:31:20.113329 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:20.113338 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:20.113622 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:20.113806 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:20.115582 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:20.116916 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:20.118665 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:20.119679 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:20.119872 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:20.125073 (+ 5201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:20.125095 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:20.125171 (+ 76us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:20.125234 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:20.128484 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:20.129735 (+ 1251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:20.129746 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:20.129749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:20.129938 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:20.130488 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:20.130496 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:20.130499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:20.131048 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:20.131403 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:20.131736 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 23:31:20.131996 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":305,"threads_started":1}
I20260926 23:31:20.134063 8907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:31:20.226867 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 8784
I20260926 23:31:20.243944 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 8918
Sep 26 23:31:20 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:31:20.261631 9193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:20.244924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37488 (local address 127.0.97.2:40639)
0926 23:31:20.245633 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:20.245641 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:20.245672 (+ 31us) server_negotiation.cc:407] Connection header received
0926 23:31:20.245742 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:20.245749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:20.245992 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:20.246155 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:20.247488 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:20.248642 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:20.250196 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:20.250917 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:20.251051 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:20.255461 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:20.255479 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:20.255498 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:20.255546 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:20.258564 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:20.259441 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:20.259450 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:20.259452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:20.259590 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:20.259972 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:20.259976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:20.259978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:20.260373 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:20.260672 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:20.260993 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 23:31:20.261273 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":343,"threads_started":1}
I20260926 23:31:20.263531 9041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:31:20.354423 9046 heartbeater.cc:503] Master 127.0.97.62:34729 was elected leader, sending a full tablet report...
I20260926 23:31:20.355718 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 8918
W20260926 23:31:20.371254 8708 connection.cc:535] server connection from 127.0.97.2:59817 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 23:31:20.371562 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 9052
Sep 26 23:31:20 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:31:20.391531 9195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:20.372683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50338 (local address 127.0.97.3:45307)
0926 23:31:20.373450 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:20.373458 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:20.373522 (+ 64us) server_negotiation.cc:407] Connection header received
0926 23:31:20.373622 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:20.373632 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:20.373902 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:20.374047 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:20.375677 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:20.376738 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:20.378233 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:20.378922 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:20.379043 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:20.384195 (+ 5152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:20.384216 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:20.384239 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:20.384292 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:20.387825 (+ 3533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:20.388793 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:20.388803 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:20.388806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:20.389052 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:20.389473 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:20.389477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:20.389479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:20.390059 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:20.390338 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:20.390743 (+ 405us) server_negotiation.cc:299] Negotiation successful
0926 23:31:20.391101 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":327,"threads_started":1}
I20260926 23:31:20.393289 9175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:31:20.487321 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 9052
I20260926 23:31:20.505000 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 8691
I20260926 23:31:20.506394 8753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:31:20.610890 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 8691
I20260926 23:31:20.632082 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34729
--webserver_interface=127.0.97.62
--webserver_port=35179
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34729
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:31:21.000082 9198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:31:21.000432 9198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:31:21.000563 9198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:31:21.008641 9198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:31:21.008811 9198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:31:21.008891 9198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:31:21.008966 9198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:31:21.009032 9198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:31:21.019387 9198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=35179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:31:21.020828 9198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:31:21.022236 9198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:21.036171 9206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:21.036196 9204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:21.036334 9203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:21.036773 9198 server_base.cc:1034] running on GCE node
I20260926 23:31:21.038038 9198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:21.040126 9198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:21.041448 9198 hybrid_clock.cc:648] HybridClock initialized: now 1790465481041385 us; error 56 us; skew 500 ppm
I20260926 23:31:21.041950 9198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:31:21 dist-test-slave-gwfx krb5kdc[8678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:31:21.050125 9198 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:31:21.052416 9198 webserver.cc:460] Webserver started at http://127.0.97.62:35179/ using document root <none> and password file <none>
I20260926 23:31:21.053421 9198 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:21.053593 9198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:21.062367 9198 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 23:31:21.066862 9213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:21.069149 9198 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260926 23:31:21.069442 9198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "3491e2c162f64c71b5aec7d228cfcd86"
format_stamp: "Formatted at 2026-09-26 23:31:17 on dist-test-slave-gwfx"
server_key: "f326789d7301f3cfe6ef70b8a11db12b"
server_key_iv: "60f0ec15ba2b7a6a3db0fc5882652480"
server_key_version: "encryptionkey@0"
I20260926 23:31:21.070799 9198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:21.094926 9198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:21.100265 9198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:31:21.100543 9198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:21.123879 9198 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34729
I20260926 23:31:21.123890 9264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34729 every 8 connection(s)
I20260926 23:31:21.125852 9198 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:31:21.130455 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 9198
I20260926 23:31:21.131430 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:32877
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:34729
--webserver_port=36871
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:31:21.133780 9265 sys_catalog.cc:263] Verifying existing consensus state
I20260926 23:31:21.137332 9265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86: Bootstrap starting.
I20260926 23:31:21.158613 9265 log.cc:824] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86: Log is configured to *not* fsync() on all Append() calls
I20260926 23:31:21.168373 9265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 23:31:21.168951 9265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86: Bootstrap complete.
I20260926 23:31:21.178835 9265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3491e2c162f64c71b5aec7d228cfcd86" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34729 } }
I20260926 23:31:21.179833 9265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3491e2c162f64c71b5aec7d228cfcd86, State: Initialized, Role: FOLLOWER
I20260926 23:31:21.180572 9265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3491e2c162f64c71b5aec7d228cfcd86" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34729 } }
I20260926 23:31:21.180814 9265 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:31:21.180934 9265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:31:21.181108 9265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [term 1 FOLLOWER]: Advancing to term 2
I20260926 23:31:21.185307 9265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3491e2c162f64c71b5aec7d228cfcd86" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34729 } }
I20260926 23:31:21.185941 9265 leader_election.cc:302] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3491e2c162f64c71b5aec7d228cfcd86; no voters:
I20260926 23:31:21.186755 9265 leader_election.cc:288] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 23:31:21.186895 9269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 23:31:21.188508 9269 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [term 2 LEADER]: Becoming Leader. State: Replica: 3491e2c162f64c71b5aec7d228cfcd86, State: Running, Role: LEADER
I20260926 23:31:21.189141 9269 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3491e2c162f64c71b5aec7d228cfcd86" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34729 } }
I20260926 23:31:21.189457 9265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:31:21.192437 9271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3491e2c162f64c71b5aec7d228cfcd86. Latest consensus state: current_term: 2 leader_uuid: "3491e2c162f64c71b5aec7d228cfcd86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3491e2c162f64c71b5aec7d228cfcd86" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34729 } } }
I20260926 23:31:21.192525 9270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3491e2c162f64c71b5aec7d228cfcd86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3491e2c162f64c71b5aec7d228cfcd86" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34729 } } }
I20260926 23:31:21.192991 9271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [sys.catalog]: This master's current role is: LEADER
I20260926 23:31:21.193042 9270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86 [sys.catalog]: This master's current role is: LEADER
I20260926 23:31:21.194028 9276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:31:21.212406 9276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:31:21.214093 9276 catalog_manager.cc:1259] Loaded cluster ID: 18aaaf2f6bac46709b0781a5a33d3022
I20260926 23:31:21.214241 9276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:31:21.218631 9276 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:31:21.221307 9276 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3491e2c162f64c71b5aec7d228cfcd86: Loaded TSK: 0
I20260926 23:31:21.223335 9276 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 23:31:21.569380 9267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:31:21.569810 9267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:31:21.569957 9267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:31:21.578474 9267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:31:21.578648 9267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:31:21.578853 9267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:31:21.590246 9267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:32877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36871
--tserver_master_addrs=127.0.97.62:34729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:31:21.592092 9267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:31:21.593720 9267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:21.610896 9292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:21.610941 9291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:21.611562 9294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:21.611910 9267 server_base.cc:1034] running on GCE node
I20260926 23:31:21.612747 9267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:21.613958 9267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:21.615193 9267 hybrid_clock.cc:648] HybridClock initialized: now 1790465481615128 us; error 55 us; skew 500 ppm
I20260926 23:31:21.615607 9267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:31:21 dist-test-slave-gwfx krb5kdc[8678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465481, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:31:21.619486 9267 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260926 23:31:21.621613 9267 webserver.cc:460] Webserver started at http://127.0.97.1:36871/ using document root <none> and password file <none>
I20260926 23:31:21.622561 9267 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:21.622776 9267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:21.629999 9267 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.002s
I20260926 23:31:21.633694 9301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:21.635635 9267 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:31:21.635864 9267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "a0477807ae4648c2a052c9d3c70c0ce7"
format_stamp: "Formatted at 2026-09-26 23:31:18 on dist-test-slave-gwfx"
server_key: "f20dcdbf58d967f59d01de32ee152bdf"
server_key_iv: "9bc9fd56b1e9cb71e53a73dedb18bd68"
server_key_version: "encryptionkey@0"
I20260926 23:31:21.637100 9267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:21.659456 9267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:21.665143 9267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:31:21.665454 9267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:21.666654 9267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:31:21.668356 9267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:31:21.668478 9267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:21.668594 9267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:31:21.668673 9267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:21.703161 9267 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:32877
I20260926 23:31:21.703229 9413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:32877 every 8 connection(s)
I20260926 23:31:21.704952 9267 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:31:21.713289 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 9267
I20260926 23:31:21.714037 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40639
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:34729
--webserver_port=45285
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:31:21 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:21 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 23:31:21.728368 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:21.708595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55729 (local address 127.0.97.62:34729)
0926 23:31:21.709378 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:21.709395 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:21.709789 (+ 394us) server_negotiation.cc:407] Connection header received
0926 23:31:21.712268 (+ 2479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:21.712280 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:21.712782 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:21.713252 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:21.715219 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:21.717105 (+ 1886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:21.719089 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:21.720165 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:21.720372 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:21.727155 (+ 6783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55729: BlockingRecv error: recv got EOF from 127.0.97.1:55729 (error 108)
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":298,"threads_started":1}
I20260926 23:31:21.728381 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:21.708705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.1:55729)
0926 23:31:21.709597 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:21.709645 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:21.711651 (+ 2006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:21.713400 (+ 1749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:21.713413 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:21.714133 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:21.714951 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:21.714978 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:21.717345 (+ 2367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:21.717351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:21.718914 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:21.718935 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:21.720512 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:21.720521 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:21.720800 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:21.721733 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:21.721770 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:21.726647 (+ 4877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":408,"threads_started":1}
W20260926 23:31:21.729445 9414 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34729 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:34729: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 26 23:31:21 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:21 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:21.739042 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:21.730154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.1:60573)
0926 23:31:21.730354 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:21.730377 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:21.730513 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:21.730919 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:21.730924 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:21.731326 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:21.731562 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:21.731574 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:21.732627 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:21.732631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:21.733659 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:21.733674 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:21.734412 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:21.734419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:21.734580 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:21.735297 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:21.735318 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:21.738672 (+ 3354us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 23:31:21.739218 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:21.730305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60573 (local address 127.0.97.62:34729)
0926 23:31:21.730462 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:21.730467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:21.730489 (+ 22us) server_negotiation.cc:407] Connection header received
0926 23:31:21.730648 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:21.730654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:21.730775 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:21.730857 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:21.731721 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:21.732477 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:21.733799 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:21.734225 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:21.734338 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:21.738974 (+ 4636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60573: BlockingRecv error: recv got EOF from 127.0.97.1:60573 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 23:31:21 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:21 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:21.750085 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:21.739874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.1:45627)
0926 23:31:21.740107 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:21.740131 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:21.740254 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:21.740569 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:21.740574 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:21.741000 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:21.741230 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:21.741242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:21.742434 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:21.742441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:21.744182 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:21.744201 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:21.744942 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:21.744948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:21.745143 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:21.745829 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:21.745853 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:21.749643 (+ 3790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 23:31:21.750097 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:21.739988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45627 (local address 127.0.97.62:34729)
0926 23:31:21.740155 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:21.740159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:21.740180 (+ 21us) server_negotiation.cc:407] Connection header received
0926 23:31:21.740374 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:21.740378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:21.740442 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:21.740521 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:21.741378 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:21.742302 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:21.744341 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:21.744801 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:21.744902 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:21.749899 (+ 4997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45627: BlockingRecv error: recv got EOF from 127.0.97.1:45627 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 23:31:21.750634 9414 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 23:31:22.107615 9418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:31:22.107987 9418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:31:22.108120 9418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:31:22.116074 9418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:31:22.116230 9418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:31:22.116393 9418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:31:22.127645 9418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=45285
--tserver_master_addrs=127.0.97.62:34729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:31:22.129390 9418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:31:22.131102 9418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:22.147500 9424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:22.147475 9423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:22.147408 9426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:22.148682 9418 server_base.cc:1034] running on GCE node
I20260926 23:31:22.149478 9418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:22.150882 9418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:22.152215 9418 hybrid_clock.cc:648] HybridClock initialized: now 1790465482152177 us; error 99 us; skew 500 ppm
I20260926 23:31:22.152634 9418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:31:22 dist-test-slave-gwfx krb5kdc[8678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465482, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:31:22.156782 9418 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260926 23:31:22.158751 9418 webserver.cc:460] Webserver started at http://127.0.97.2:45285/ using document root <none> and password file <none>
I20260926 23:31:22.159662 9418 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:22.159811 9418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:22.166565 9418 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 23:31:22.170065 9433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:22.171716 9418 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:31:22.171891 9418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "d46dda7151d44a6cafc6efb8945fc361"
format_stamp: "Formatted at 2026-09-26 23:31:19 on dist-test-slave-gwfx"
server_key: "f339b36763b7a4ee0404799e5a10171d"
server_key_iv: "7f87ebea46575a1bd9301a540e42f4dc"
server_key_version: "encryptionkey@0"
I20260926 23:31:22.173030 9418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:22.197852 9418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:22.203419 9418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:31:22.203783 9418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:22.205165 9418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:31:22.206954 9418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:31:22.207079 9418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:22.207180 9418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:31:22.207237 9418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:22.242137 9418 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40639
I20260926 23:31:22.242184 9545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40639 every 8 connection(s)
I20260926 23:31:22.244107 9418 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:31:22.244412 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 9418
I20260926 23:31:22.245139 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:45307
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:34729
--webserver_port=40825
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:31:22 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:22 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 23:31:22.263823 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:22.247508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41187 (local address 127.0.97.62:34729)
0926 23:31:22.247751 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:22.247757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:22.249382 (+ 1625us) server_negotiation.cc:407] Connection header received
0926 23:31:22.252218 (+ 2836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:22.252224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:22.252334 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:22.252433 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:22.254967 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:22.255741 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:22.257491 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:22.257931 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:22.258053 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:22.263505 (+ 5452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:41187: BlockingRecv error: recv got EOF from 127.0.97.2:41187 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 23:31:22.264783 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:22.248128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.2:41187)
0926 23:31:22.249126 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:22.249192 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:22.251719 (+ 2527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:22.252800 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:22.252814 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:22.253675 (+ 861us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:22.254681 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:22.254752 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:22.255920 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:22.255926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:22.257323 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:22.257338 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:22.258218 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:22.258227 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:22.258481 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:22.259266 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:22.259298 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:22.263253 (+ 3955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":389,"threads_started":1}
W20260926 23:31:22.265631 9546 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34729 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:34729: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 26 23:31:22 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:22 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:22.275063 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:22.266407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.2:49563)
0926 23:31:22.266605 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:22.266628 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:22.266791 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:22.267166 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:22.267171 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:22.267524 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:22.267691 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:22.267700 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:22.268766 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:22.268770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:22.269714 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:22.269727 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:22.270427 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:22.270434 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:22.270608 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:22.271378 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:22.271397 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:22.274676 (+ 3279us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 23:31:22.275347 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:22.266471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49563 (local address 127.0.97.62:34729)
0926 23:31:22.266702 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:22.266708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:22.266731 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:31:22.266922 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:22.266926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:22.266993 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:22.267076 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:22.267837 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:22.268624 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:22.269847 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:22.270282 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:22.270405 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:22.275036 (+ 4631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:49563: BlockingRecv error: recv got EOF from 127.0.97.2:49563 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 23:31:22 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:22 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:22.286118 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:22.276025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.2:48255)
0926 23:31:22.276318 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:22.276338 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:22.276463 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:22.276791 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:22.276796 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:22.277291 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:22.277514 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:22.277526 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:22.278807 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:22.278813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:22.280409 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:22.280427 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:22.281195 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:22.281201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:22.281389 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:22.282076 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:22.282099 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:22.285746 (+ 3647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170}
W20260926 23:31:22.286267 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:22.276106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48255 (local address 127.0.97.62:34729)
0926 23:31:22.276286 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:22.276291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:22.276399 (+ 108us) server_negotiation.cc:407] Connection header received
0926 23:31:22.276576 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:22.276580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:22.276663 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:22.276747 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:22.277648 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:22.278647 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:22.280583 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:22.281046 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:22.281152 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:22.286029 (+ 4877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:48255: BlockingRecv error: recv got EOF from 127.0.97.2:48255 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 23:31:22.286569 9546 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 23:31:22.687331 9549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:31:22.687736 9549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:31:22.687885 9549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:31:22.696151 9549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:31:22.696300 9549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:31:22.696468 9549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:31:22.707875 9549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:45307
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=40825
--tserver_master_addrs=127.0.97.62:34729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:31:22.709580 9549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:31:22.711402 9549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:22.727331 9554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:22.727253 9555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:22.727291 9557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:22.727900 9549 server_base.cc:1034] running on GCE node
I20260926 23:31:22.728705 9549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:22.729933 9549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:22.731154 9549 hybrid_clock.cc:648] HybridClock initialized: now 1790465482731075 us; error 72 us; skew 500 ppm
I20260926 23:31:22.731511 9549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:31:22 dist-test-slave-gwfx krb5kdc[8678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465482, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:31:22.735119 9549 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260926 23:31:22.736877 9549 webserver.cc:460] Webserver started at http://127.0.97.3:40825/ using document root <none> and password file <none>
I20260926 23:31:22.737731 9549 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:22.737871 9549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:22.744935 9549 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.002s
I20260926 23:31:22.748346 9564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:22.750172 9549 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 23:31:22.750360 9549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "7c9c9c92411544b5aeef2eba3c33df3a"
format_stamp: "Formatted at 2026-09-26 23:31:19 on dist-test-slave-gwfx"
server_key: "5673f93a1bcafcca27160376388c9ad2"
server_key_iv: "1f532860dd55cbe88dfd8c7a627edd85"
server_key_version: "encryptionkey@0"
I20260926 23:31:22.751823 9549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 26 23:31:22 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:22 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:22.761596 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:22.751524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.1:54631)
0926 23:31:22.752041 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:22.752065 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:22.752285 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:22.752734 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:22.752738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:22.753227 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:22.753464 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:22.753475 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:22.754874 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:22.754878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:22.755929 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:22.755943 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:22.757001 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:22.757007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:22.757152 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:22.757811 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:22.757831 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:22.761315 (+ 3484us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":335,"threads_started":1}
W20260926 23:31:22.761911 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:22.751638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54631 (local address 127.0.97.62:34729)
0926 23:31:22.751853 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:22.751860 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:22.752169 (+ 309us) server_negotiation.cc:407] Connection header received
0926 23:31:22.752441 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:22.752447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:22.752559 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:22.752677 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:22.753630 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:22.754708 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:22.756093 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:22.756785 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:22.756917 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:22.761637 (+ 4720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54631: BlockingRecv error: recv got EOF from 127.0.97.1:54631 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 23:31:22.815150 9549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:22.821110 9549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:31:22.821455 9549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:22.822875 9549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:31:22.824802 9549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:31:22.824934 9549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:22.825043 9549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:31:22.825137 9549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:22.861462 9549 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45307
I20260926 23:31:22.861526 9677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45307 every 8 connection(s)
I20260926 23:31:22.863402 9549 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:31:22.869577 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 9549
Sep 26 23:31:22 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:22 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 23:31:22.883872 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:22.866807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52845 (local address 127.0.97.62:34729)
0926 23:31:22.866995 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:22.867000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:22.868470 (+ 1470us) server_negotiation.cc:407] Connection header received
0926 23:31:22.871356 (+ 2886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:22.871369 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:22.871484 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:22.871568 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:22.873692 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:22.874551 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:22.876531 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:22.877046 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:22.877180 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:22.883514 (+ 6334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:52845: BlockingRecv error: recv got EOF from 127.0.97.3:52845 (error 108)
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":60}
I20260926 23:31:22.884555 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:22.867313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.3:52845)
0926 23:31:22.868256 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:22.868311 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:22.870862 (+ 2551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:22.872128 (+ 1266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:22.872138 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:22.872774 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:22.873502 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:22.873523 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:22.874766 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:22.874770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:22.876391 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:22.876407 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:22.877288 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:22.877295 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:22.877659 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:22.878441 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:22.878479 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:22.883230 (+ 4751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":357,"threads_started":1}
W20260926 23:31:22.885442 9678 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34729 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:34729: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260926 23:31:22.888139 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:22.870789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56266 (local address 127.0.97.62:34729)
0926 23:31:22.871703 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:22.871708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:22.871726 (+ 18us) server_negotiation.cc:407] Connection header received
0926 23:31:22.871770 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:22.871773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:22.871833 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:22.871921 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:22.873636 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:22.874562 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:22.875867 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:22.876400 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:22.876522 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:22.880171 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:22.880186 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:22.880216 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:22.880275 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:22.883889 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:22.884827 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:22.884837 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:22.884847 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:22.885133 (+ 286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:22.885624 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:22.885629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:22.885632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:22.886274 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:22.886615 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:22.887284 (+ 669us) server_negotiation.cc:299] Negotiation successful
0926 23:31:22.887763 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":780,"thread_start_us":646,"threads_started":1}
Sep 26 23:31:22 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:22 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:22.895757 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:22.886162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.3:54111)
0926 23:31:22.886360 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:22.886383 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:22.886494 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:22.886888 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:22.886891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:22.887433 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:22.887634 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:22.887648 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:22.889005 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:22.889009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:22.890371 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:22.890388 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:22.891335 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:22.891340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:22.891525 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:22.892266 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:22.892287 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:22.895508 (+ 3221us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 23:31:22.896111 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:22.886304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54111 (local address 127.0.97.62:34729)
0926 23:31:22.886569 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:22.886575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:22.886591 (+ 16us) server_negotiation.cc:407] Connection header received
0926 23:31:22.886642 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:22.886648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:22.886755 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:22.886856 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:22.887776 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:22.888876 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:22.890576 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:22.891181 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:22.891304 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:22.895776 (+ 4472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:54111: BlockingRecv error: recv got EOF from 127.0.97.3:54111 (error 108)
Metrics: {"server-negotiator.queue_time_us":138}
Sep 26 23:31:22 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:22 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:22.905289 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:22.896593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.3:53317)
0926 23:31:22.896789 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:22.896810 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:22.896938 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:22.897272 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:22.897277 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:22.897622 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:22.897802 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:22.897812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:22.898865 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:22.898870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:22.899832 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:22.899845 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:22.900549 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:22.900554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:22.900729 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:22.901313 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:22.901331 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:22.904916 (+ 3585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 23:31:22.905385 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:22.896682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53317 (local address 127.0.97.62:34729)
0926 23:31:22.896850 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:22.896855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:22.896874 (+ 19us) server_negotiation.cc:407] Connection header received
0926 23:31:22.897056 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:22.897061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:22.897141 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:22.897226 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:22.897938 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:22.898732 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:22.899965 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:22.900404 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:22.900514 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:22.905139 (+ 4625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:53317: BlockingRecv error: recv got EOF from 127.0.97.3:53317 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 23:31:22.905701 9678 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 23:31:23 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:23 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:23.297581 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:23.287395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.2:42843)
0926 23:31:23.287999 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:23.288017 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:23.288229 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:23.288610 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:23.288614 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:23.289165 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:23.289431 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:23.289447 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:23.290640 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:23.290645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:23.291772 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:23.291789 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:23.292591 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:23.292597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:23.292761 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:23.293505 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:23.293530 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:23.297202 (+ 3672us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":316,"threads_started":1}
W20260926 23:31:23.297775 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:23.287514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42843 (local address 127.0.97.62:34729)
0926 23:31:23.287801 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:23.287807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:23.288106 (+ 299us) server_negotiation.cc:407] Connection header received
0926 23:31:23.288361 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:23.288367 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:23.288439 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:23.288541 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:23.289630 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:23.290460 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:23.291925 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:23.292362 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:23.292468 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:23.297496 (+ 5028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:42843: BlockingRecv error: recv got EOF from 127.0.97.2:42843 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 26 23:31:23 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:23 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:23.773314 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:23.763080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.1:45745)
0926 23:31:23.763711 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:23.763738 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:23.763911 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:23.764348 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:23.764352 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:23.764829 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:23.765021 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:23.765033 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:23.766296 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:23.766302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:23.767559 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:23.767577 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:23.768471 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:23.768505 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:23.768691 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:23.769431 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:23.769452 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:23.773016 (+ 3564us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":340,"threads_started":1}
W20260926 23:31:23.773643 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:23.763194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45745 (local address 127.0.97.62:34729)
0926 23:31:23.763427 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:23.763432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:23.763909 (+ 477us) server_negotiation.cc:407] Connection header received
0926 23:31:23.764050 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:23.764054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:23.764160 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:23.764281 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:23.765230 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:23.766083 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:23.767708 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:23.768272 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:23.768432 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:23.773324 (+ 4892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45745: BlockingRecv error: recv got EOF from 127.0.97.1:45745 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 26 23:31:23 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:23 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:23.916465 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:23.906555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.3:52475)
0926 23:31:23.907229 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:23.907249 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:23.907391 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:23.907738 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:23.907741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:23.908282 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:23.908508 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:23.908519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:23.909731 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:23.909734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:23.911002 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:23.911019 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:23.912030 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:23.912037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:23.912218 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:23.912861 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:23.912880 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:23.916119 (+ 3239us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":351,"threads_started":1}
W20260926 23:31:23.916641 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:23.906827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52475 (local address 127.0.97.62:34729)
0926 23:31:23.907011 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:23.907016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:23.907337 (+ 321us) server_negotiation.cc:407] Connection header received
0926 23:31:23.907507 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:23.907511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:23.907580 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:23.907702 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:23.908721 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:23.909569 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:23.911206 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:23.911822 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:23.911944 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:23.916369 (+ 4425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:52475: BlockingRecv error: recv got EOF from 127.0.97.3:52475 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 23:31:24 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:24 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:24.309178 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:24.298966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.2:58983)
0926 23:31:24.299619 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:24.299639 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:24.299833 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:24.300356 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:24.300361 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:24.300853 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:24.301038 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:24.301050 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:24.302147 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:24.302153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:24.303166 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:24.303179 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:24.303974 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:24.303980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:24.304158 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:24.305066 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:24.305093 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:24.308893 (+ 3800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":366,"threads_started":1}
W20260926 23:31:24.309588 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:24.299036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58983 (local address 127.0.97.62:34729)
0926 23:31:24.299340 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:24.299348 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:24.299759 (+ 411us) server_negotiation.cc:407] Connection header received
0926 23:31:24.300117 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:24.300121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:24.300193 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:24.300286 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:24.301199 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:24.302005 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:24.303318 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:24.303791 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:24.303912 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:24.309292 (+ 5380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58983: BlockingRecv error: recv got EOF from 127.0.97.2:58983 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Sep 26 23:31:24 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:24 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:24.783913 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:24.774504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.1:42827)
0926 23:31:24.775179 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:24.775203 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:24.775362 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:24.775784 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:24.775789 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:24.776226 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:24.776431 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:24.776442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:24.777716 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:24.777721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:24.778931 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:24.778947 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:24.779857 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:24.779863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:24.780036 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:24.780766 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:24.780786 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:24.783550 (+ 2764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":360,"threads_started":1}
W20260926 23:31:24.784101 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:24.774907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42827 (local address 127.0.97.62:34729)
0926 23:31:24.775114 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:24.775119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:24.775293 (+ 174us) server_negotiation.cc:407] Connection header received
0926 23:31:24.775501 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:24.775505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:24.775585 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:24.775706 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:24.776651 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:24.777526 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:24.779111 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:24.779690 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:24.779820 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:24.783821 (+ 4001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42827: BlockingRecv error: recv got EOF from 127.0.97.1:42827 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 23:31:24 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:24 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:24.927727 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:24.917904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.3:38337)
0926 23:31:24.918599 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:24.918620 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:24.918815 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:24.919209 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:24.919213 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:24.919712 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:24.919906 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:24.919916 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:24.921312 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:24.921319 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:24.922347 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:24.922368 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:24.923247 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:24.923254 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:24.923423 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:24.924094 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:24.924115 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:24.927443 (+ 3328us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":393,"threads_started":1}
W20260926 23:31:24.928144 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:24.918090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38337 (local address 127.0.97.62:34729)
0926 23:31:24.918304 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:24.918308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:24.918753 (+ 445us) server_negotiation.cc:407] Connection header received
0926 23:31:24.918943 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:24.918947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:24.919026 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:24.919170 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:24.920134 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:24.921047 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:24.922543 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:24.923057 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:24.923181 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:24.927827 (+ 4646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:38337: BlockingRecv error: recv got EOF from 127.0.97.3:38337 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 26 23:31:25 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:25 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:25.320602 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:25.310247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.2:39163)
0926 23:31:25.310886 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:25.310907 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:25.311076 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:25.311545 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:25.311549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:25.312057 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:25.312297 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:25.312309 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:25.313581 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:25.313587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:25.314724 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:25.314748 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:25.315597 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:25.315602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:25.315759 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:25.316527 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:25.316551 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:25.320249 (+ 3698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":279,"threads_started":1}
W20260926 23:31:25.321039 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:25.310405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39163 (local address 127.0.97.62:34729)
0926 23:31:25.310676 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:25.310716 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:25.311065 (+ 349us) server_negotiation.cc:407] Connection header received
0926 23:31:25.311235 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:25.311239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:25.311319 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:25.311427 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:25.312515 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:25.313371 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:25.314952 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:25.315401 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:25.315535 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:25.320739 (+ 5204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:39163: BlockingRecv error: recv got EOF from 127.0.97.2:39163 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Sep 26 23:31:25 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:25 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:25.795244 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:25.785329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.1:53029)
0926 23:31:25.785955 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:25.785979 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:25.786152 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:25.786576 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:25.786580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:25.787058 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:25.787279 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:25.787290 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:25.788602 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:25.788606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:25.789845 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:25.789860 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:25.790738 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:25.790745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:25.790937 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:25.791597 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:25.791617 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:25.794979 (+ 3362us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":308,"threads_started":1}
W20260926 23:31:25.795603 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:25.785526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53029 (local address 127.0.97.62:34729)
0926 23:31:25.785747 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:25.785751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:25.786093 (+ 342us) server_negotiation.cc:407] Connection header received
0926 23:31:25.786269 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:25.786272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:25.786385 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:25.786504 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:25.787444 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:25.788369 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:25.790009 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:25.790547 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:25.790719 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:25.795297 (+ 4578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53029: BlockingRecv error: recv got EOF from 127.0.97.1:53029 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 26 23:31:25 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:25 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:25.938231 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:25.928990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.3:48901)
0926 23:31:25.929578 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:25.929600 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:25.929746 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:25.930136 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:25.930140 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:25.930479 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:25.930750 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:25.930762 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:25.931925 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:25.931930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:25.932883 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:25.932897 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:25.933783 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:25.933788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:25.933939 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:25.934606 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:25.934625 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:25.937952 (+ 3327us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":339,"threads_started":1}
W20260926 23:31:25.938594 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:25.929031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48901 (local address 127.0.97.62:34729)
0926 23:31:25.929239 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:25.929246 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:25.929695 (+ 449us) server_negotiation.cc:407] Connection header received
0926 23:31:25.929877 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:25.929882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:25.929964 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:25.930067 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:25.930915 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:25.931751 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:25.933041 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:25.933535 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:25.933661 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:25.938283 (+ 4622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:48901: BlockingRecv error: recv got EOF from 127.0.97.3:48901 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 23:31:26 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:26 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:26.338521 9692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:26.322055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.2:51037)
0926 23:31:26.322677 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:26.322726 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:26.322903 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:26.323469 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:26.323473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:26.324160 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:26.324394 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:26.324407 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:26.325661 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:26.325665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:26.326879 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:26.326899 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:26.327860 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:26.327866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:26.328071 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:26.334222 (+ 6151us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:26.334252 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:26.338159 (+ 3907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":369,"threads_started":1}
W20260926 23:31:26.338784 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:26.322310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51037 (local address 127.0.97.62:34729)
0926 23:31:26.322527 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:26.322531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:26.322865 (+ 334us) server_negotiation.cc:407] Connection header received
0926 23:31:26.323139 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:26.323144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:26.323226 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:26.323356 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:26.324662 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:26.325468 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:26.327069 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:26.327669 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:26.327813 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:26.338501 (+ 10688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51037: BlockingRecv error: recv got EOF from 127.0.97.2:51037 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 26 23:31:26 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:26 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:26.807106 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:26.796595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.1:40653)
0926 23:31:26.797206 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:26.797232 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:26.797376 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:26.797889 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:26.797893 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:26.798335 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:26.798528 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:26.798539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:26.799890 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:26.799894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:26.801033 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:26.801052 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:26.802080 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:26.802090 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:26.802278 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:26.803079 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:26.803102 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:26.806647 (+ 3545us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":293,"threads_started":1}
W20260926 23:31:26.807252 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:26.796798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40653 (local address 127.0.97.62:34729)
0926 23:31:26.797009 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:26.797015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:26.797357 (+ 342us) server_negotiation.cc:407] Connection header received
0926 23:31:26.797532 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:26.797537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:26.797617 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:26.797716 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:26.798748 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:26.799697 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:26.801262 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:26.801823 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:26.801952 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:26.807005 (+ 5053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:40653: BlockingRecv error: recv got EOF from 127.0.97.1:40653 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 26 23:31:26 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:26 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:26.949398 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:26.939629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.3:49565)
0926 23:31:26.940188 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:26.940207 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:26.940387 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:26.940883 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:26.940888 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:26.941324 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:26.941489 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:26.941500 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:26.942870 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:26.942876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:26.944011 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:26.944027 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:26.944883 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:26.944888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:26.945061 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:26.945690 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:26.945715 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:26.949046 (+ 3331us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":284,"threads_started":1}
W20260926 23:31:26.949587 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:26.939728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49565 (local address 127.0.97.62:34729)
0926 23:31:26.940152 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:26.940159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:26.940365 (+ 206us) server_negotiation.cc:407] Connection header received
0926 23:31:26.940528 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:26.940535 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:26.940621 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:26.940739 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:26.941707 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:26.942629 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:26.944218 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:26.944714 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:26.944839 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:26.949348 (+ 4509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:49565: BlockingRecv error: recv got EOF from 127.0.97.3:49565 (error 108)
Metrics: {"server-negotiator.queue_time_us":159}
Sep 26 23:31:27 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:27 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:27.350997 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:27.339946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.2:56665)
0926 23:31:27.340723 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:27.340743 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:27.340899 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:27.341365 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:27.341369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:27.341838 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:27.342057 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:27.342069 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:27.343356 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:27.343360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:27.344403 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:27.344419 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:27.345301 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:27.345306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:27.345450 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:27.346160 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:27.346183 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:27.350579 (+ 4396us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":462,"threads_started":1}
W20260926 23:31:27.351137 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:27.340167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56665 (local address 127.0.97.62:34729)
0926 23:31:27.340366 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:27.340372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:27.340886 (+ 514us) server_negotiation.cc:407] Connection header received
0926 23:31:27.341009 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:27.341013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:27.341090 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:27.341280 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:27.342277 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:27.343167 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:27.344624 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:27.345126 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:27.345269 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:27.350903 (+ 5634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56665: BlockingRecv error: recv got EOF from 127.0.97.2:56665 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 23:31:27 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:27 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:27.819097 9696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:27.808529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.1:40571)
0926 23:31:27.809330 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:27.809355 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:27.809513 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:27.809981 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:27.809985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:27.810472 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:27.810666 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:27.810677 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:27.811820 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:27.811825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:27.812910 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:27.812926 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:27.813822 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:27.813828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:27.814077 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:27.814967 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:27.814994 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:27.818750 (+ 3756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":509,"threads_started":1}
W20260926 23:31:27.819446 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:27.808682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40571 (local address 127.0.97.62:34729)
0926 23:31:27.808906 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:27.808912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:27.809507 (+ 595us) server_negotiation.cc:407] Connection header received
0926 23:31:27.809621 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:27.809625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:27.809701 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:27.809891 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:27.810879 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:27.811622 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:27.813116 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:27.813598 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:27.813752 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:27.819017 (+ 5265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:40571: BlockingRecv error: recv got EOF from 127.0.97.1:40571 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 26 23:31:27 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:27 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:27.961185 9697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:27.950829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.3:41765)
0926 23:31:27.951354 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:27.951373 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:27.951509 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:27.951923 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:27.951927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:27.952335 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:27.952515 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:27.952531 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:27.953729 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:27.953734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:27.954755 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:27.954771 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:27.955615 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:27.955620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:27.955805 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:27.956759 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:27.956790 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:27.960789 (+ 3999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":272,"threads_started":1}
W20260926 23:31:27.961313 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:27.950911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41765 (local address 127.0.97.62:34729)
0926 23:31:27.951151 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:27.951156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:27.951468 (+ 312us) server_negotiation.cc:407] Connection header received
0926 23:31:27.951623 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:27.951628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:27.951698 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:27.951820 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:27.952690 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:27.953523 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:27.954922 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:27.955433 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:27.955539 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:27.961067 (+ 5528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41765: BlockingRecv error: recv got EOF from 127.0.97.3:41765 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Sep 26 23:31:28 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:28 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:28.362039 9698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:28.352444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.2:54829)
0926 23:31:28.353377 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:28.353396 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:28.353554 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:28.353893 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:28.353898 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:28.354363 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:28.354537 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:28.354546 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:28.355805 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:28.355811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:28.356836 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:28.356848 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:28.357585 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:28.357591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:28.357787 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:28.358486 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:28.358505 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:28.361761 (+ 3256us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":794,"thread_start_us":632,"threads_started":1}
W20260926 23:31:28.362428 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:28.352583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54829 (local address 127.0.97.62:34729)
0926 23:31:28.352846 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:28.352852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:28.353497 (+ 645us) server_negotiation.cc:407] Connection header received
0926 23:31:28.353673 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:28.353679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:28.353755 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:28.353851 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:28.354711 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:28.355656 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:28.356969 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:28.357429 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:28.357549 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:28.362144 (+ 4595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:54829: BlockingRecv error: recv got EOF from 127.0.97.2:54829 (error 108)
Metrics: {"server-negotiator.queue_time_us":135}
Sep 26 23:31:28 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:28 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:28.830646 9699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:28.820527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.1:43813)
0926 23:31:28.821499 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:28.821523 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:28.821725 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:28.822092 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:28.822097 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:28.822553 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:28.822801 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:28.822816 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:28.823880 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:28.823884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:28.824946 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:28.824963 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:28.826145 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:28.826152 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:28.826313 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:28.827205 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:28.827240 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:28.830345 (+ 3105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":803,"thread_start_us":473,"threads_started":1}
W20260926 23:31:28.831100 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:28.820706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43813 (local address 127.0.97.62:34729)
0926 23:31:28.820934 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:28.820939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:28.821658 (+ 719us) server_negotiation.cc:407] Connection header received
0926 23:31:28.821861 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:28.821867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:28.821944 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:28.822033 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:28.822958 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:28.823721 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:28.825170 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:28.825886 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:28.826040 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:28.830780 (+ 4740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:43813: BlockingRecv error: recv got EOF from 127.0.97.1:43813 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 23:31:28 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:28 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:28.972083 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:28.962404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.3:49507)
0926 23:31:28.963056 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:28.963078 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:28.963254 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:28.963616 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:28.963622 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:28.964078 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:28.964253 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:28.964263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:28.965327 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:28.965333 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:28.966313 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:28.966326 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:28.967134 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:28.967141 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:28.967338 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:28.968086 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:28.968108 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:28.971806 (+ 3698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":375,"threads_started":1}
W20260926 23:31:28.972277 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:28.962627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49507 (local address 127.0.97.62:34729)
0926 23:31:28.962902 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:28.962909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:28.963179 (+ 270us) server_negotiation.cc:407] Connection header received
0926 23:31:28.963391 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:28.963397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:28.963467 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:28.963555 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:28.964402 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:28.965185 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:28.966452 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:28.966955 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:28.967083 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:28.972039 (+ 4956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:49507: BlockingRecv error: recv got EOF from 127.0.97.3:49507 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 26 23:31:29 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:29 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:29.373623 9701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:29.363530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.2:48653)
0926 23:31:29.364295 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:29.364323 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:29.364477 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:29.364997 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:29.365001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:29.365413 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:29.365607 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:29.365617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:29.366897 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:29.366901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:29.367865 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:29.367878 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:29.368836 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:29.368843 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:29.369013 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:29.369845 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:29.369873 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:29.373296 (+ 3423us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":358,"threads_started":1}
W20260926 23:31:29.373858 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:29.363701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48653 (local address 127.0.97.62:34729)
0926 23:31:29.364024 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:29.364029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:29.364499 (+ 470us) server_negotiation.cc:407] Connection header received
0926 23:31:29.364588 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:29.364593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:29.364699 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:29.364963 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:29.365797 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:29.366734 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:29.368033 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:29.368583 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:29.368733 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:29.373554 (+ 4821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:48653: BlockingRecv error: recv got EOF from 127.0.97.2:48653 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Sep 26 23:31:29 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:29 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:29.842854 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:29.832143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.1:50693)
0926 23:31:29.832804 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:29.832838 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:29.833005 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:29.833593 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:29.833601 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:29.834151 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:29.834399 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:29.834411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:29.835648 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:29.835700 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:29.836683 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:29.836701 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:29.837929 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:29.837935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:29.838104 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:29.838877 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:29.838896 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:29.842436 (+ 3540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":349,"threads_started":1}
W20260926 23:31:29.843091 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:29.832313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50693 (local address 127.0.97.62:34729)
0926 23:31:29.832498 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:29.832504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:29.833028 (+ 524us) server_negotiation.cc:407] Connection header received
0926 23:31:29.833165 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:29.833172 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:29.833268 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:29.833381 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:29.834598 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:29.835438 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:29.836920 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:29.837629 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:29.837792 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:29.842826 (+ 5034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:50693: BlockingRecv error: recv got EOF from 127.0.97.1:50693 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 23:31:29 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:29 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:29.983752 9703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:29.973258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.3:46639)
0926 23:31:29.974084 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:29.974108 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:29.974299 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:29.974799 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:29.974804 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:29.975290 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:29.975526 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:29.975537 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:29.976666 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:29.976671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:29.977797 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:29.977817 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:29.978834 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:29.978841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:29.979042 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:29.979884 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:29.979910 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:29.983472 (+ 3562us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":382,"threads_started":1}
W20260926 23:31:29.984107 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:29.973695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46639 (local address 127.0.97.62:34729)
0926 23:31:29.973986 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:29.973991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:29.974266 (+ 275us) server_negotiation.cc:407] Connection header received
0926 23:31:29.974441 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:29.974446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:29.974531 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:29.974647 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:29.975690 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:29.976479 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:29.977997 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:29.978615 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:29.978798 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:29.983796 (+ 4998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:46639: BlockingRecv error: recv got EOF from 127.0.97.3:46639 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
Sep 26 23:31:30 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:30 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:30.387470 9704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:30.374943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.2:46025)
0926 23:31:30.375621 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:30.375647 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:30.375980 (+ 333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:30.376553 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:30.376557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:30.377214 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:30.377430 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:30.377462 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:30.379692 (+ 2230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:30.379699 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:30.380906 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:30.380932 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:30.382062 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:30.382074 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:30.382242 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:30.383080 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:30.383104 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:30.387163 (+ 4059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":354,"threads_started":1}
W20260926 23:31:30.387708 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:30.375400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46025 (local address 127.0.97.62:34729)
0926 23:31:30.375628 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:30.375633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:30.375772 (+ 139us) server_negotiation.cc:407] Connection header received
0926 23:31:30.376238 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:30.376245 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:30.376326 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:30.376434 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:30.377713 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:30.379410 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:30.381139 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:30.381780 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:30.381907 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:30.387465 (+ 5558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46025: BlockingRecv error: recv got EOF from 127.0.97.2:46025 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 26 23:31:30 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:30 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:30.854859 9705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:30.844039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.1:35997)
0926 23:31:30.844679 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:30.844704 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:30.844856 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:30.845466 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:30.845471 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:30.845923 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:30.846133 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:30.846143 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:30.847463 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:30.847468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:30.848588 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:30.848604 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:30.849499 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:30.849505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:30.849666 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:30.850467 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:30.850490 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:30.854306 (+ 3816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":348,"threads_started":1}
W20260926 23:31:30.854948 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:30.844083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35997 (local address 127.0.97.62:34729)
0926 23:31:30.844341 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:30.844347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:30.844814 (+ 467us) server_negotiation.cc:407] Connection header received
0926 23:31:30.844985 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:30.844989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:30.845060 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:30.845150 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:30.846328 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:30.847265 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:30.848792 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:30.849282 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:30.849417 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:30.854615 (+ 5198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:35997: BlockingRecv error: recv got EOF from 127.0.97.1:35997 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 26 23:31:30 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:30 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:30.994418 9706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:30.984835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.3:47389)
0926 23:31:30.985644 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:30.985664 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:30.985827 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:30.986285 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:30.986291 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:30.986836 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:30.987018 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:30.987029 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:30.988087 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:30.988091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:30.989094 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:30.989108 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:30.989896 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:30.989903 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:30.990064 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:30.990819 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:30.990840 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:30.994057 (+ 3217us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":551,"threads_started":1}
W20260926 23:31:30.994580 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:30.984937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47389 (local address 127.0.97.62:34729)
0926 23:31:30.985147 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:30.985153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:30.985759 (+ 606us) server_negotiation.cc:407] Connection header received
0926 23:31:30.986012 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:30.986016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:30.986096 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:30.986195 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:30.987166 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:30.987951 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:30.989238 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:30.989714 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:30.989824 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:30.994362 (+ 4538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47389: BlockingRecv error: recv got EOF from 127.0.97.3:47389 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 23:31:31 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:31 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:31.406904 9707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:31.388688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.2:38155)
0926 23:31:31.389541 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:31.389563 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:31.389758 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:31.390233 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:31.390237 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:31.390756 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:31.390960 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:31.390970 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:31.392066 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:31.392071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:31.393113 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:31.393127 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:31.393919 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:31.393925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:31.394098 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:31.403116 (+ 9018us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:31.403149 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:31.406472 (+ 3323us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":362,"threads_started":1}
W20260926 23:31:31.407094 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:31.388885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38155 (local address 127.0.97.62:34729)
0926 23:31:31.389181 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:31.389189 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:31.389698 (+ 509us) server_negotiation.cc:407] Connection header received
0926 23:31:31.389983 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:31.389990 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:31.390063 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:31.390164 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:31.391113 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:31.391926 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:31.393269 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:31.393739 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:31.393858 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:31.406829 (+ 12971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38155: BlockingRecv error: recv got EOF from 127.0.97.2:38155 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Sep 26 23:31:31 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:31 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:31.868594 9708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:31.856486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.1:39959)
0926 23:31:31.857143 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:31.857175 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:31.857453 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:31.858143 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:31.858147 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:31.858628 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:31.858866 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:31.858876 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:31.860081 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:31.860086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:31.861189 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:31.861204 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:31.862791 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:31.862798 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:31.863032 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:31.864358 (+ 1326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:31.864387 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:31.868258 (+ 3871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":371,"threads_started":1}
W20260926 23:31:31.868772 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:31.856563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39959 (local address 127.0.97.62:34729)
0926 23:31:31.856853 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:31.856858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:31.857424 (+ 566us) server_negotiation.cc:407] Connection header received
0926 23:31:31.857648 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:31.857652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:31.857758 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:31.858041 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:31.859061 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:31.859917 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:31.861478 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:31.862351 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:31.862734 (+ 383us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:31.868536 (+ 5802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39959: BlockingRecv error: recv got EOF from 127.0.97.1:39959 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Sep 26 23:31:32 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:32 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:32.006587 9709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:31.995819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.3:58383)
0926 23:31:31.996467 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:31.996486 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:31.996627 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:31.996985 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:31.996989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:31.997454 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:31.997688 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:31.997698 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:31.998918 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:31.998923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:32.000176 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:32.000194 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:32.001123 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:32.001129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:32.001286 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:32.002082 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:32.002101 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:32.006173 (+ 4072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":320,"threads_started":1}
W20260926 23:31:32.006754 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:31.995926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58383 (local address 127.0.97.62:34729)
0926 23:31:31.996159 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:31.996165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:31.996581 (+ 416us) server_negotiation.cc:407] Connection header received
0926 23:31:31.996741 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:31.996745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:31.996823 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:31.996923 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:31.997843 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:31.998741 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:32.000387 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:32.000921 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:32.001035 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:32.006467 (+ 5432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58383: BlockingRecv error: recv got EOF from 127.0.97.3:58383 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 26 23:31:32 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:32 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:32.423010 9710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:32.408197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.2:38187)
0926 23:31:32.408928 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:32.408948 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:32.409101 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:32.409546 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:32.409550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:32.410114 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:32.410307 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:32.410319 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:32.411555 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:32.411559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:32.412577 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:32.412591 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:32.413495 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:32.413500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:32.413654 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:32.418652 (+ 4998us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:32.418772 (+ 120us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:32.422718 (+ 3946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":419,"threads_started":1}
W20260926 23:31:32.423458 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:32.408347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38187 (local address 127.0.97.62:34729)
0926 23:31:32.408653 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:32.408658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:32.409075 (+ 417us) server_negotiation.cc:407] Connection header received
0926 23:31:32.409229 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:32.409233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:32.409320 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:32.409447 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:32.410522 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:32.411385 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:32.412732 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:32.413244 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:32.413533 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:32.423119 (+ 9586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38187: BlockingRecv error: recv got EOF from 127.0.97.2:38187 (error 108)
Metrics: {"server-negotiator.queue_time_us":150}
Sep 26 23:31:32 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:32 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:32.880882 9711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:32.870096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.1:43773)
0926 23:31:32.870997 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:32.871021 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:32.871176 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:32.871802 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:32.871809 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:32.872415 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:32.872727 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:32.872741 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:32.874051 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:32.874056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:32.875140 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:32.875156 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:32.876187 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:32.876194 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:32.876403 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:32.877151 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:32.877173 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:32.880613 (+ 3440us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":783,"thread_start_us":272,"threads_started":1}
W20260926 23:31:32.881335 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:32.870450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43773 (local address 127.0.97.62:34729)
0926 23:31:32.870748 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:32.870756 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:32.871106 (+ 350us) server_negotiation.cc:407] Connection header received
0926 23:31:32.871355 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:32.871360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:32.871559 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:32.871684 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:32.872997 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:32.873867 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:32.875308 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:32.875933 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:32.876097 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:32.880983 (+ 4886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:43773: BlockingRecv error: recv got EOF from 127.0.97.1:43773 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
Sep 26 23:31:33 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:33 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:33.017838 9712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:33.008145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.3:47599)
0926 23:31:33.008757 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:33.008779 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:33.008945 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:33.009429 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:33.009434 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:33.009834 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:33.010007 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:33.010015 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:33.011355 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:33.011369 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:33.012387 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:33.012402 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:33.013472 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:33.013477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:33.013645 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:33.014379 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:33.014399 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:33.017515 (+ 3116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":355,"threads_started":1}
W20260926 23:31:33.018227 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:33.008289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47599 (local address 127.0.97.62:34729)
0926 23:31:33.008585 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:33.008592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:33.008928 (+ 336us) server_negotiation.cc:407] Connection header received
0926 23:31:33.009097 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:33.009102 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:33.009210 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:33.009332 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:33.010232 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:33.011166 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:33.012607 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:33.013197 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:33.013404 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:33.017935 (+ 4531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47599: BlockingRecv error: recv got EOF from 127.0.97.3:47599 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 26 23:31:33 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:33 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:33.434527 9713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:33.424240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.2:33611)
0926 23:31:33.424967 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:33.424986 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:33.425154 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:33.425508 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:33.425513 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:33.426032 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:33.426236 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:33.426247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:33.427473 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:33.427478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:33.428435 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:33.428449 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:33.429172 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:33.429178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:33.429353 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:33.430158 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:33.430185 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:33.434127 (+ 3942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":487,"threads_started":1}
W20260926 23:31:33.434653 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:33.424542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33611 (local address 127.0.97.62:34729)
0926 23:31:33.424843 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:33.424850 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:33.425084 (+ 234us) server_negotiation.cc:407] Connection header received
0926 23:31:33.425292 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:33.425296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:33.425363 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:33.425465 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:33.426393 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:33.427326 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:33.428575 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:33.429012 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:33.429124 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:33.434397 (+ 5273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33611: BlockingRecv error: recv got EOF from 127.0.97.2:33611 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
Sep 26 23:31:33 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:33 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:33.892669 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:33.882304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.1:35763)
0926 23:31:33.883025 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:33.883055 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:33.883209 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:33.883557 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:33.883561 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:33.884012 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:33.884194 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:33.884206 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:33.885495 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:33.885500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:33.886492 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:33.886586 (+ 94us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:33.887468 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:33.887473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:33.887695 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:33.888376 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:33.888395 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:33.892289 (+ 3894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":382,"threads_started":1}
W20260926 23:31:33.892813 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:33.882400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35763 (local address 127.0.97.62:34729)
0926 23:31:33.882633 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:33.882639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:33.883145 (+ 506us) server_negotiation.cc:407] Connection header received
0926 23:31:33.883346 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:33.883351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:33.883420 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:33.883569 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:33.884368 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:33.885298 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:33.886847 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:33.887308 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:33.887429 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:33.892579 (+ 5150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:35763: BlockingRecv error: recv got EOF from 127.0.97.1:35763 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 23:31:34 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:34 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:34.029995 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:34.019210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.3:49191)
0926 23:31:34.019858 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:34.019877 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:34.020336 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:34.020858 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:34.020864 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:34.021439 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:34.021756 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:34.021774 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:34.023116 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:34.023122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:34.024053 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:34.024069 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:34.024877 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:34.024883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:34.025081 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:34.025893 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:34.025915 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:34.029672 (+ 3757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":366,"threads_started":1}
W20260926 23:31:34.030719 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:34.019367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49191 (local address 127.0.97.62:34729)
0926 23:31:34.019750 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:34.019757 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:34.020126 (+ 369us) server_negotiation.cc:407] Connection header received
0926 23:31:34.020594 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:34.020598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:34.020701 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:34.020802 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:34.021976 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:34.022953 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:34.024229 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:34.024704 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:34.024823 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:34.030219 (+ 5396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:49191: BlockingRecv error: recv got EOF from 127.0.97.3:49191 (error 108)
Metrics: {"server-negotiator.queue_time_us":145}
Sep 26 23:31:34 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:34 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:34.445550 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:34.435834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.2:49485)
0926 23:31:34.436501 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:34.436521 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:34.436750 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:34.437085 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:34.437089 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:34.437515 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:34.437693 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:34.437703 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:34.438714 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:34.438719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:34.439690 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:34.439705 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:34.440397 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:34.440403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:34.440584 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:34.441241 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:34.441259 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:34.445174 (+ 3915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":374,"threads_started":1}
W20260926 23:31:34.445715 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:34.435972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49485 (local address 127.0.97.62:34729)
0926 23:31:34.436213 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:34.436219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:34.436625 (+ 406us) server_negotiation.cc:407] Connection header received
0926 23:31:34.436881 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:34.436886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:34.436949 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:34.437030 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:34.437836 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:34.438541 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:34.439832 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:34.440247 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:34.440357 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:34.445457 (+ 5100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:49485: BlockingRecv error: recv got EOF from 127.0.97.2:49485 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 23:31:34 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:34 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:34.904489 9717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:34.893848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.1:54317)
0926 23:31:34.894582 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:34.894606 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:34.894804 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:34.895154 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:34.895159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:34.895618 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:34.895784 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:34.895793 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:34.897064 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:34.897069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:34.898162 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:34.898177 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:34.899185 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:34.899192 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:34.899356 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:34.900209 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:34.900229 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:34.904067 (+ 3838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":388,"threads_started":1}
W20260926 23:31:34.904721 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:34.893989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54317 (local address 127.0.97.62:34729)
0926 23:31:34.894279 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:34.894287 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:34.894719 (+ 432us) server_negotiation.cc:407] Connection header received
0926 23:31:34.894930 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:34.894934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:34.895005 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:34.895088 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:34.895944 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:34.896876 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:34.898350 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:34.898967 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:34.899099 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:34.904367 (+ 5268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54317: BlockingRecv error: recv got EOF from 127.0.97.1:54317 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Sep 26 23:31:35 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:35 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:35.041217 9718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:35.031402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.3:57725)
0926 23:31:35.032081 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:35.032099 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:35.032276 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:35.032723 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:35.032729 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:35.033377 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:35.033567 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:35.033578 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:35.034800 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:35.034806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:35.035861 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:35.035874 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:35.036583 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:35.036588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:35.036776 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:35.037454 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:35.037474 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:35.040674 (+ 3200us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":438,"threads_started":1}
W20260926 23:31:35.041326 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:35.031634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57725 (local address 127.0.97.62:34729)
0926 23:31:35.031863 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:35.031870 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:35.032236 (+ 366us) server_negotiation.cc:407] Connection header received
0926 23:31:35.032431 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:35.032436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:35.032522 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:35.032639 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:35.033727 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:35.034612 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:35.036005 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:35.036438 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:35.036545 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:35.041081 (+ 4536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57725: BlockingRecv error: recv got EOF from 127.0.97.3:57725 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 26 23:31:35 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:35 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:35.457464 9719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:35.446811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.2:55425)
0926 23:31:35.447497 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:35.447519 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:35.447681 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:35.448021 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:35.448024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:35.448459 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:35.448664 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:35.448673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:35.450014 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:35.450018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:35.451059 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:35.451074 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:35.452237 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:35.452243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:35.452389 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:35.453431 (+ 1042us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:35.453458 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:35.457118 (+ 3660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":389,"threads_started":1}
W20260926 23:31:35.457854 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:35.447052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55425 (local address 127.0.97.62:34729)
0926 23:31:35.447314 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:35.447321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:35.447622 (+ 301us) server_negotiation.cc:407] Connection header received
0926 23:31:35.447794 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:35.447799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:35.447874 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:35.447985 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:35.448847 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:35.449785 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:35.451267 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:35.451978 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:35.452103 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:35.457499 (+ 5396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55425: BlockingRecv error: recv got EOF from 127.0.97.2:55425 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 26 23:31:35 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:35 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:35.915760 9720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:35.906046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.1:34379)
0926 23:31:35.906581 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:35.906609 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:35.906795 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:35.907254 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:35.907257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:35.907634 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:35.907837 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:35.907846 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:35.909017 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:35.909021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:35.909964 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:35.909976 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:35.910771 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:35.910777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:35.910953 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:35.911670 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:35.911689 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:35.915393 (+ 3704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":266,"threads_started":1}
W20260926 23:31:35.916061 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:35.906239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34379 (local address 127.0.97.62:34729)
0926 23:31:35.906450 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:35.906455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:35.906709 (+ 254us) server_negotiation.cc:407] Connection header received
0926 23:31:35.906991 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:35.906996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:35.907078 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:35.907178 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:35.908048 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:35.908851 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:35.910128 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:35.910569 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:35.910738 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:35.915751 (+ 5013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:34379: BlockingRecv error: recv got EOF from 127.0.97.1:34379 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 26 23:31:36 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:36 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:36.052102 9721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:36.042563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.3:58437)
0926 23:31:36.043285 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:36.043305 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:36.043484 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:36.043844 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:36.043849 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:36.044364 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:36.044572 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:36.044582 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:36.045657 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:36.045662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:36.046650 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:36.046662 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:36.047473 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:36.047480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:36.047642 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:36.048424 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:36.048450 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:36.051772 (+ 3322us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":391,"threads_started":1}
W20260926 23:31:36.052438 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:36.042778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58437 (local address 127.0.97.62:34729)
0926 23:31:36.043181 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:36.043189 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:36.043408 (+ 219us) server_negotiation.cc:407] Connection header received
0926 23:31:36.043624 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:36.043630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:36.043704 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:36.043802 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:36.044723 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:36.045517 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:36.046822 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:36.047279 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:36.047391 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:36.052137 (+ 4746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58437: BlockingRecv error: recv got EOF from 127.0.97.3:58437 (error 108)
Metrics: {"server-negotiator.queue_time_us":219}
Sep 26 23:31:36 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:36 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:36.472573 9722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:36.458645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.2:52709)
0926 23:31:36.459362 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:36.459380 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:36.459534 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:36.459919 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:36.459923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:36.460421 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:36.460602 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:36.460611 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:36.461741 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:36.461746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:36.462728 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:36.462739 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:36.463466 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:36.463471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:36.463636 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:36.468482 (+ 4846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:36.468508 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:36.472271 (+ 3763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":408,"threads_started":1}
W20260926 23:31:36.472791 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:36.458769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52709 (local address 127.0.97.62:34729)
0926 23:31:36.458973 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:36.458979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:36.459472 (+ 493us) server_negotiation.cc:407] Connection header received
0926 23:31:36.459652 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:36.459658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:36.459727 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:36.459892 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:36.460749 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:36.461590 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:36.462860 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:36.463320 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:36.463431 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:36.472545 (+ 9114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:52709: BlockingRecv error: recv got EOF from 127.0.97.2:52709 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 23:31:36 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:36 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:36.927292 9723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:36.917286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.1:52189)
0926 23:31:36.917838 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:36.917863 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:36.918044 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:36.918567 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:36.918572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:36.919106 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:36.919374 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:36.919386 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:36.920772 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:36.920778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:36.921950 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:36.921969 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:36.922847 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:36.922857 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:36.923008 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:36.923733 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:36.923753 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:36.926955 (+ 3202us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":548,"threads_started":1}
W20260926 23:31:36.927596 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:36.917578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52189 (local address 127.0.97.62:34729)
0926 23:31:36.917824 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:36.917831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:36.918011 (+ 180us) server_negotiation.cc:407] Connection header received
0926 23:31:36.918192 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:36.918197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:36.918304 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:36.918469 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:36.919577 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:36.920491 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:36.922137 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:36.922620 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:36.922778 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:36.927309 (+ 4531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:52189: BlockingRecv error: recv got EOF from 127.0.97.1:52189 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 26 23:31:37 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:37 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:37.062829 9724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:37.053365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.3:56201)
0926 23:31:37.053930 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:37.053951 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:37.054089 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:37.054491 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:37.054494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:37.054919 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:37.055098 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:37.055107 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:37.056274 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:37.056279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:37.057326 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:37.057340 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:37.058259 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:37.058263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:37.058392 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:37.059124 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:37.059144 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:37.062427 (+ 3283us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":286,"threads_started":1}
W20260926 23:31:37.063004 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:37.053535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56201 (local address 127.0.97.62:34729)
0926 23:31:37.053817 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:37.053824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:37.054044 (+ 220us) server_negotiation.cc:407] Connection header received
0926 23:31:37.054269 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:37.054275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:37.054354 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:37.054454 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:37.055265 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:37.056112 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:37.057478 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:37.058111 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:37.058223 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:37.062738 (+ 4515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56201: BlockingRecv error: recv got EOF from 127.0.97.3:56201 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Sep 26 23:31:37 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 23:31:37 dist-test-slave-gwfx krb5kdc[8678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 23:31:37.483330 9725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:37.474042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34729 (local address 127.0.97.2:43243)
0926 23:31:37.474759 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:37.474781 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:37.474984 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:37.475372 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:37.475377 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:37.475850 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:37.476038 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:37.476047 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:37.477186 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:37.477191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:37.478192 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:37.478204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:37.478965 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:37.478971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:37.479126 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:37.479803 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:37.479821 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:37.483008 (+ 3187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:34729: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":353,"threads_started":1}
W20260926 23:31:37.483673 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:37.474281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43243 (local address 127.0.97.62:34729)
0926 23:31:37.474578 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:37.474585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:37.474910 (+ 325us) server_negotiation.cc:407] Connection header received
0926 23:31:37.475135 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:37.475139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:37.475229 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:37.475331 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:37.476193 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:37.477046 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:37.478321 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:37.478813 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:37.478930 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:37.483348 (+ 4418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43243: BlockingRecv error: recv got EOF from 127.0.97.2:43243 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
I20260926 23:31:37.878938 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 9267
I20260926 23:31:37.893923 9727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:37.880255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33732 (local address 127.0.97.1:32877)
0926 23:31:37.880996 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:37.881007 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:37.881054 (+ 47us) server_negotiation.cc:407] Connection header received
0926 23:31:37.881141 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:37.881150 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:37.881462 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:37.881674 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:37.883407 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:37.884675 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:37.886308 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:37.887154 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:37.887292 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:37.890179 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:37.890190 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:37.890212 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:37.890263 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:37.893320 (+ 3057us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 23:31:37.893647 (+ 327us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33732: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":356,"threads_started":1}
W20260926 23:31:37.894030 9727 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33732: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 23:31:37.894374 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 9267: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:32877: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 23:31:37.894625 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 9267
I20260926 23:31:37.911875 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 9418
I20260926 23:31:37.927949 9728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:37.913558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53792 (local address 127.0.97.2:40639)
0926 23:31:37.914314 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:37.914321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:37.914359 (+ 38us) server_negotiation.cc:407] Connection header received
0926 23:31:37.914429 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:37.914435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:37.914716 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:37.914924 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:37.916500 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:37.917740 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:37.919756 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:37.920489 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:37.920921 (+ 432us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:37.924199 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:37.924211 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:37.924230 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:37.924298 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:37.927310 (+ 3012us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 23:31:37.927664 (+ 354us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53792: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":391,"threads_started":1}
W20260926 23:31:37.928062 9728 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53792: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 23:31:37.928395 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 9418: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:40639: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 23:31:37.928578 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 9418
I20260926 23:31:37.945748 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 9549
I20260926 23:31:37.960942 9729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:37.947111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53502 (local address 127.0.97.3:45307)
0926 23:31:37.947866 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:37.947876 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:37.947914 (+ 38us) server_negotiation.cc:407] Connection header received
0926 23:31:37.947998 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:37.948007 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:37.948333 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:37.948543 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:37.950106 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:37.951530 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:37.953167 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:37.953909 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:37.954058 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:37.956843 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:37.956859 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:37.956883 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:37.956957 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:37.960297 (+ 3340us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 23:31:37.960657 (+ 360us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53502: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":306,"threads_started":1}
W20260926 23:31:37.961042 9729 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53502: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 23:31:37.961375 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 9549: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:45307: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 23:31:37.961572 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 9549
I20260926 23:31:37.978266 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 9198
I20260926 23:31:37.980019 9260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:31:38.101251 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 9198
2026-09-26T23:31:38Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (22061 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T23:31:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:31:38Z Disabled control of system clock
I20260926 23:31:38.189844 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35381
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35381
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 23:31:38.649461 9734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:31:38.649914 9734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:31:38.650060 9734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:31:38.658483 9734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:31:38.658654 9734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:31:38.658725 9734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:31:38.658782 9734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:31:38.658861 9734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:31:38.671060 9734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35381
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:31:38.673072 9734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:31:38.675797 9734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:38.694330 9740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:38.694481 9739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:38.694360 9742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:38.694908 9734 server_base.cc:1034] running on GCE node
I20260926 23:31:38.696369 9734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:38.698645 9734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:38.700084 9734 hybrid_clock.cc:648] HybridClock initialized: now 1790465498700004 us; error 58 us; skew 500 ppm
I20260926 23:31:38.700651 9734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:31:38.704100 9734 webserver.cc:460] Webserver started at http://127.0.97.62:36791/ using document root <none> and password file <none>
I20260926 23:31:38.705052 9734 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:38.705166 9734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:38.705538 9734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:31:38.709393 9734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "211ee3fe43c14367b8904041139b8d3b"
format_stamp: "Formatted at 2026-09-26 23:31:38 on dist-test-slave-gwfx"
server_key: "b64723baa858f41d439cb2a8732694b4"
server_key_iv: "db80a12c081ebcbfa83f0a8ba16d0463"
server_key_version: "encryptionkey@0"
I20260926 23:31:38.710392 9734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "211ee3fe43c14367b8904041139b8d3b"
format_stamp: "Formatted at 2026-09-26 23:31:38 on dist-test-slave-gwfx"
server_key: "b64723baa858f41d439cb2a8732694b4"
server_key_iv: "db80a12c081ebcbfa83f0a8ba16d0463"
server_key_version: "encryptionkey@0"
I20260926 23:31:38.717552 9734 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260926 23:31:38.723297 9748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:38.725641 9734 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260926 23:31:38.725932 9734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "211ee3fe43c14367b8904041139b8d3b"
format_stamp: "Formatted at 2026-09-26 23:31:38 on dist-test-slave-gwfx"
server_key: "b64723baa858f41d439cb2a8732694b4"
server_key_iv: "db80a12c081ebcbfa83f0a8ba16d0463"
server_key_version: "encryptionkey@0"
I20260926 23:31:38.726224 9734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:38.745576 9734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:38.746743 9734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:31:38.747066 9734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:38.768667 9734 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35381
I20260926 23:31:38.768714 9799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35381 every 8 connection(s)
I20260926 23:31:38.770421 9734 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:31:38.774726 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 9734
I20260926 23:31:38.774961 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:31:38.775013 9800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:31:38.775671 388 external_mini_cluster.cc:1428] Setting key 9c6d09908272de3769b69882590cbe9e
I20260926 23:31:38.789849 9800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 211ee3fe43c14367b8904041139b8d3b: Bootstrap starting.
I20260926 23:31:38.792689 9803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:38.778051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53178 (local address 127.0.97.62:35381)
0926 23:31:38.779320 (+ 1269us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:38.779334 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:38.779398 (+ 64us) server_negotiation.cc:407] Connection header received
0926 23:31:38.781337 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:38.781368 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:38.782064 (+ 696us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:38.782536 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:38.783464 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:38.785011 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:38.787104 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:38.787986 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:38.788171 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:38.789637 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:38.789646 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:38.789665 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:31:38.790428 (+ 763us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:38.790532 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:31:38.790562 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:38.790664 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:38.791013 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 23:31:38.791446 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":402,"threads_started":1}
I20260926 23:31:38.793601 9800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 211ee3fe43c14367b8904041139b8d3b: Neither blocks nor log segments found. Creating new log.
I20260926 23:31:38.794806 9800 log.cc:824] T 00000000000000000000000000000000 P 211ee3fe43c14367b8904041139b8d3b: Log is configured to *not* fsync() on all Append() calls
I20260926 23:31:38.797837 9800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 211ee3fe43c14367b8904041139b8d3b: No bootstrap required, opened a new log
I20260926 23:31:38.805804 9800 raft_consensus.cc:348] T 00000000000000000000000000000000 P 211ee3fe43c14367b8904041139b8d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211ee3fe43c14367b8904041139b8d3b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35381 } }
I20260926 23:31:38.806268 9800 raft_consensus.cc:374] T 00000000000000000000000000000000 P 211ee3fe43c14367b8904041139b8d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:31:38.806366 9800 raft_consensus.cc:729] T 00000000000000000000000000000000 P 211ee3fe43c14367b8904041139b8d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 211ee3fe43c14367b8904041139b8d3b, State: Initialized, Role: FOLLOWER
I20260926 23:31:38.807029 9800 consensus_queue.cc:260] T 00000000000000000000000000000000 P 211ee3fe43c14367b8904041139b8d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211ee3fe43c14367b8904041139b8d3b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35381 } }
I20260926 23:31:38.807272 9800 raft_consensus.cc:388] T 00000000000000000000000000000000 P 211ee3fe43c14367b8904041139b8d3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:31:38.807395 9800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 211ee3fe43c14367b8904041139b8d3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:31:38.807550 9800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 211ee3fe43c14367b8904041139b8d3b [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:31:38.810021 9800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 211ee3fe43c14367b8904041139b8d3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211ee3fe43c14367b8904041139b8d3b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35381 } }
I20260926 23:31:38.810752 9800 leader_election.cc:302] T 00000000000000000000000000000000 P 211ee3fe43c14367b8904041139b8d3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 211ee3fe43c14367b8904041139b8d3b; no voters:
I20260926 23:31:38.811421 9800 leader_election.cc:288] T 00000000000000000000000000000000 P 211ee3fe43c14367b8904041139b8d3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:31:38.811672 9805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 211ee3fe43c14367b8904041139b8d3b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:31:38.812498 9805 raft_consensus.cc:686] T 00000000000000000000000000000000 P 211ee3fe43c14367b8904041139b8d3b [term 1 LEADER]: Becoming Leader. State: Replica: 211ee3fe43c14367b8904041139b8d3b, State: Running, Role: LEADER
I20260926 23:31:38.813333 9805 consensus_queue.cc:237] T 00000000000000000000000000000000 P 211ee3fe43c14367b8904041139b8d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211ee3fe43c14367b8904041139b8d3b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35381 } }
I20260926 23:31:38.814183 9800 sys_catalog.cc:564] T 00000000000000000000000000000000 P 211ee3fe43c14367b8904041139b8d3b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:31:38.817137 9806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 211ee3fe43c14367b8904041139b8d3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "211ee3fe43c14367b8904041139b8d3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211ee3fe43c14367b8904041139b8d3b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35381 } } }
I20260926 23:31:38.817783 9806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 211ee3fe43c14367b8904041139b8d3b [sys.catalog]: This master's current role is: LEADER
I20260926 23:31:38.817051 9807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 211ee3fe43c14367b8904041139b8d3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 211ee3fe43c14367b8904041139b8d3b. Latest consensus state: current_term: 1 leader_uuid: "211ee3fe43c14367b8904041139b8d3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211ee3fe43c14367b8904041139b8d3b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35381 } } }
I20260926 23:31:38.818389 9807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 211ee3fe43c14367b8904041139b8d3b [sys.catalog]: This master's current role is: LEADER
I20260926 23:31:38.821048 9813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:31:38.827924 9813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:31:38.840411 9813 catalog_manager.cc:1347] Generated new cluster ID: 3be1ffcc53a54ad2ae4ed417e0c1f323
I20260926 23:31:38.840567 9813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:31:38.860662 9813 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:31:38.862357 9813 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:31:38.884306 9813 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 211ee3fe43c14367b8904041139b8d3b: Generated new TSK 0
I20260926 23:31:38.885507 9813 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:31:38.897068 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35381
--builtin_ntp_servers=127.0.97.20:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 23:31:39.372311 9824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:31:39.372762 9824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:31:39.372866 9824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:31:39.381608 9824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:31:39.381786 9824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:31:39.381922 9824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:31:39.392629 9824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:31:39.394490 9824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:31:39.396914 9824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:39.414373 9829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:39.414306 9830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:39.414753 9832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:39.415850 9824 server_base.cc:1034] running on GCE node
I20260926 23:31:39.417016 9824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:39.418360 9824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:39.419906 9824 hybrid_clock.cc:648] HybridClock initialized: now 1790465499419696 us; error 125 us; skew 500 ppm
I20260926 23:31:39.420388 9824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:31:39.424453 9824 webserver.cc:460] Webserver started at http://127.0.97.1:40171/ using document root <none> and password file <none>
I20260926 23:31:39.425334 9824 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:39.425457 9824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:39.425814 9824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:31:39.429440 9824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/data/instance:
uuid: "93274ee692044e8382379c6ddf9ef072"
format_stamp: "Formatted at 2026-09-26 23:31:39 on dist-test-slave-gwfx"
server_key: "c3c814438c64f2a8549c9a02c5aa7966"
server_key_iv: "ff72becb5d49ebc429997d0dee1325da"
server_key_version: "encryptionkey@0"
I20260926 23:31:39.430317 9824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/wal/instance:
uuid: "93274ee692044e8382379c6ddf9ef072"
format_stamp: "Formatted at 2026-09-26 23:31:39 on dist-test-slave-gwfx"
server_key: "c3c814438c64f2a8549c9a02c5aa7966"
server_key_iv: "ff72becb5d49ebc429997d0dee1325da"
server_key_version: "encryptionkey@0"
I20260926 23:31:39.436986 9824 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 23:31:39.441823 9838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:39.444105 9824 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260926 23:31:39.444343 9824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "93274ee692044e8382379c6ddf9ef072"
format_stamp: "Formatted at 2026-09-26 23:31:39 on dist-test-slave-gwfx"
server_key: "c3c814438c64f2a8549c9a02c5aa7966"
server_key_iv: "ff72becb5d49ebc429997d0dee1325da"
server_key_version: "encryptionkey@0"
I20260926 23:31:39.444598 9824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:39.465524 9824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:39.466499 9824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:31:39.466809 9824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:39.468103 9824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:31:39.471006 9824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:31:39.471136 9824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:39.471233 9824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:31:39.471313 9824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:39.507727 9824 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36665
I20260926 23:31:39.507812 9950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36665 every 8 connection(s)
I20260926 23:31:39.509155 9824 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:31:39.516815 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 9824
I20260926 23:31:39.517001 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:31:39.517645 388 external_mini_cluster.cc:1428] Setting key e9e23e69a64ed8827eb6b028ef80534c
I20260926 23:31:39.520921 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35381
--builtin_ntp_servers=127.0.97.20:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 23:31:39.525916 9953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:39.512461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34699 (local address 127.0.97.62:35381)
0926 23:31:39.512987 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:39.512992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:39.514475 (+ 1483us) server_negotiation.cc:407] Connection header received
0926 23:31:39.516986 (+ 2511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:39.516993 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:39.517102 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:39.517218 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:39.518944 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:39.519842 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:39.522202 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:39.522867 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:39.523035 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:39.524941 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:39.524948 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:39.524953 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:31:39.524982 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:39.525038 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:31:39.525055 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:39.525171 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:39.525434 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 23:31:39.525550 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":269,"threads_started":1}
I20260926 23:31:39.527396 9954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:39.512915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35381 (local address 127.0.97.1:34699)
0926 23:31:39.514199 (+ 1284us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:39.514272 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:39.516575 (+ 2303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:39.517644 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:39.517655 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:39.517798 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:39.518738 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:39.518767 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:39.520196 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:39.520202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:39.521953 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:39.521978 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:39.523220 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:39.523229 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:39.523508 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:39.524500 (+ 992us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:31:39.524503 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:39.524560 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:31:39.524677 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:39.525190 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:39.525201 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:39.525210 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 23:31:39.525409 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 23:31:39.525925 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":333,"threads_started":1}
I20260926 23:31:39.529367 9951 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35381
I20260926 23:31:39.529728 9951 heartbeater.cc:464] Registering TS with master...
I20260926 23:31:39.530750 9951 heartbeater.cc:511] Master 127.0.97.62:35381 requested a full tablet report, sending...
I20260926 23:31:39.533280 9765 ts_manager.cc:194] Registered new tserver with Master: 93274ee692044e8382379c6ddf9ef072 (127.0.97.1:36665)
I20260926 23:31:39.535091 9765 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:34699
W20260926 23:31:39.937740 9955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:31:39.938159 9955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:31:39.938279 9955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:31:39.946590 9955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:31:39.946777 9955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:31:39.946949 9955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:31:39.957331 9955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:31:39.959134 9955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:31:39.961450 9955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:39.979709 9963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:39.979856 9961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:39.979700 9960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:39.985139 9955 server_base.cc:1034] running on GCE node
I20260926 23:31:39.986173 9955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:39.987576 9955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:39.988883 9955 hybrid_clock.cc:648] HybridClock initialized: now 1790465499988804 us; error 80 us; skew 500 ppm
I20260926 23:31:39.989282 9955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:31:39.992542 9955 webserver.cc:460] Webserver started at http://127.0.97.2:34833/ using document root <none> and password file <none>
I20260926 23:31:39.993446 9955 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:39.993557 9955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:39.993885 9955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:31:39.997542 9955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/data/instance:
uuid: "a9976780f2264b048ad02d66d62b6d40"
format_stamp: "Formatted at 2026-09-26 23:31:39 on dist-test-slave-gwfx"
server_key: "89073961c40161525e9f5e9c773d329b"
server_key_iv: "87a6795720c64466b4a23b77c9688484"
server_key_version: "encryptionkey@0"
I20260926 23:31:39.998582 9955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a9976780f2264b048ad02d66d62b6d40"
format_stamp: "Formatted at 2026-09-26 23:31:39 on dist-test-slave-gwfx"
server_key: "89073961c40161525e9f5e9c773d329b"
server_key_iv: "87a6795720c64466b4a23b77c9688484"
server_key_version: "encryptionkey@0"
I20260926 23:31:40.006309 9955 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s
I20260926 23:31:40.012588 9969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:40.015084 9955 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.001s
I20260926 23:31:40.015367 9955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "a9976780f2264b048ad02d66d62b6d40"
format_stamp: "Formatted at 2026-09-26 23:31:39 on dist-test-slave-gwfx"
server_key: "89073961c40161525e9f5e9c773d329b"
server_key_iv: "87a6795720c64466b4a23b77c9688484"
server_key_version: "encryptionkey@0"
I20260926 23:31:40.015664 9955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:40.037120 9955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:40.038780 9955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:31:40.039239 9955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:40.040831 9955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:31:40.043200 9955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:31:40.043337 9955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:40.043452 9955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:31:40.043541 9955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:40.085355 9955 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43115
I20260926 23:31:40.085362 10081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43115 every 8 connection(s)
I20260926 23:31:40.087176 9955 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:31:40.090536 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 9955
I20260926 23:31:40.090858 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:31:40.091382 388 external_mini_cluster.cc:1428] Setting key a32d134bee2b4b7874b574b65d1718b1
I20260926 23:31:40.094951 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35381
--builtin_ntp_servers=127.0.97.20:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 23:31:40.103443 10084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:40.090398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54567 (local address 127.0.97.62:35381)
0926 23:31:40.091024 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:40.091031 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:40.092479 (+ 1448us) server_negotiation.cc:407] Connection header received
0926 23:31:40.094616 (+ 2137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:40.094623 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:40.094762 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:40.094909 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:40.096359 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:40.097466 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:40.099550 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:40.100148 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:40.100313 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:40.102261 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:40.102268 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:40.102274 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:31:40.102314 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:40.102380 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:31:40.102398 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:40.102504 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:40.102879 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 23:31:40.103011 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":305,"threads_started":1}
I20260926 23:31:40.104691 10085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:40.091092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35381 (local address 127.0.97.2:54567)
0926 23:31:40.092274 (+ 1182us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:40.092331 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:40.094281 (+ 1950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:40.095276 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:40.095286 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:40.095397 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:40.096125 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:40.096146 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:40.097689 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:40.097694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:40.099248 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:40.099266 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:40.100466 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:40.100472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:40.100855 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:40.101794 (+ 939us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:31:40.101798 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:40.101862 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:31:40.101960 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:40.102550 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:40.102561 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:40.102568 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 23:31:40.102853 (+ 285us) client_negotiation.cc:240] Negotiation successful
0926 23:31:40.103258 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":356,"threads_started":1}
I20260926 23:31:40.106479 10082 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35381
I20260926 23:31:40.106995 10082 heartbeater.cc:464] Registering TS with master...
I20260926 23:31:40.108031 10082 heartbeater.cc:511] Master 127.0.97.62:35381 requested a full tablet report, sending...
I20260926 23:31:40.110001 9765 ts_manager.cc:194] Registered new tserver with Master: a9976780f2264b048ad02d66d62b6d40 (127.0.97.2:43115)
I20260926 23:31:40.110982 9765 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:54567
W20260926 23:31:40.487125 10086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:31:40.487454 10086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:31:40.487552 10086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:31:40.495332 10086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:31:40.495476 10086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:31:40.495641 10086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:31:40.509434 10086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:31:40.511492 10086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:31:40.514031 10086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:40.530915 10091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:40.530977 10094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:40.530963 10092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:40.532073 10086 server_base.cc:1034] running on GCE node
I20260926 23:31:40.532943 10086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:40.534216 10086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:40.535509 10086 hybrid_clock.cc:648] HybridClock initialized: now 1790465500535447 us; error 57 us; skew 500 ppm
I20260926 23:31:40.536039 10086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:31:40.538908 9951 heartbeater.cc:503] Master 127.0.97.62:35381 was elected leader, sending a full tablet report...
I20260926 23:31:40.539117 10086 webserver.cc:460] Webserver started at http://127.0.97.3:39981/ using document root <none> and password file <none>
I20260926 23:31:40.540194 10086 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:40.540341 10086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:40.540798 10086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:31:40.544373 10086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/data/instance:
uuid: "65959b9245fd4ad8b9a3a2d8e3a87b1e"
format_stamp: "Formatted at 2026-09-26 23:31:40 on dist-test-slave-gwfx"
server_key: "e5ccb8e49edddc7b112ca2affb5635cc"
server_key_iv: "62634dd39cbfe2edc3e7587527f4b344"
server_key_version: "encryptionkey@0"
I20260926 23:31:40.545300 10086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/wal/instance:
uuid: "65959b9245fd4ad8b9a3a2d8e3a87b1e"
format_stamp: "Formatted at 2026-09-26 23:31:40 on dist-test-slave-gwfx"
server_key: "e5ccb8e49edddc7b112ca2affb5635cc"
server_key_iv: "62634dd39cbfe2edc3e7587527f4b344"
server_key_version: "encryptionkey@0"
I20260926 23:31:40.552241 10086 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260926 23:31:40.557014 10100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:40.559026 10086 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 23:31:40.559260 10086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "65959b9245fd4ad8b9a3a2d8e3a87b1e"
format_stamp: "Formatted at 2026-09-26 23:31:40 on dist-test-slave-gwfx"
server_key: "e5ccb8e49edddc7b112ca2affb5635cc"
server_key_iv: "62634dd39cbfe2edc3e7587527f4b344"
server_key_version: "encryptionkey@0"
I20260926 23:31:40.559531 10086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:40.576864 10086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:40.577996 10086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:31:40.578305 10086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:40.579545 10086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:31:40.581496 10086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:31:40.581606 10086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:40.581727 10086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:31:40.581806 10086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:40.622464 10086 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42089
I20260926 23:31:40.622527 10212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42089 every 8 connection(s)
I20260926 23:31:40.624274 10086 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:31:40.626602 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 10086
I20260926 23:31:40.626811 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:31:40.627430 388 external_mini_cluster.cc:1428] Setting key cfe692ceb4f7f6513b068885d17c1fe6
I20260926 23:31:40.640491 10215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:40.627683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37061 (local address 127.0.97.62:35381)
0926 23:31:40.628231 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:40.628236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:40.630095 (+ 1859us) server_negotiation.cc:407] Connection header received
0926 23:31:40.632632 (+ 2537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:40.632637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:40.632759 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:40.632861 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:40.634313 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:40.635199 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:40.637005 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:40.637471 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:40.637590 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:40.639441 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:40.639448 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:40.639451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:31:40.639478 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:40.639525 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:31:40.639539 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:40.639622 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:40.639893 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 23:31:40.640023 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":290,"threads_started":1}
I20260926 23:31:40.641947 10216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:40.628450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35381 (local address 127.0.97.3:37061)
0926 23:31:40.629923 (+ 1473us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:40.629978 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:40.632239 (+ 2261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:40.633191 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:40.633203 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:40.633331 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:40.634114 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:40.634135 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:40.635361 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:40.635365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:40.636833 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:40.636848 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:40.637729 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:40.637736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:40.638015 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:40.639013 (+ 998us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:31:40.639016 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:40.639073 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:31:40.639197 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:40.639637 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:40.639648 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:40.639657 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 23:31:40.639874 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 23:31:40.640442 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":390,"threads_started":1}
I20260926 23:31:40.644244 10213 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35381
I20260926 23:31:40.644781 10213 heartbeater.cc:464] Registering TS with master...
I20260926 23:31:40.646216 10213 heartbeater.cc:511] Master 127.0.97.62:35381 requested a full tablet report, sending...
I20260926 23:31:40.648273 9764 ts_manager.cc:194] Registered new tserver with Master: 65959b9245fd4ad8b9a3a2d8e3a87b1e (127.0.97.3:42089)
I20260926 23:31:40.649286 9764 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:37061
I20260926 23:31:40.659073 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 23:31:40.667539 10215 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:40.665277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53194 (local address 127.0.97.62:35381)
0926 23:31:40.665584 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:40.665594 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:40.666032 (+ 438us) server_negotiation.cc:407] Connection header received
0926 23:31:40.666290 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:40.666298 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:40.666394 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:40.666532 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:40.667265 (+ 733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53194: BlockingRecv error: recv got EOF from 127.0.0.1:53194 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 23:31:40.668640 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 9824
I20260926 23:31:40.679628 10223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:40.669871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48280 (local address 127.0.97.1:36665)
0926 23:31:40.670512 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:40.670521 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:40.670567 (+ 46us) server_negotiation.cc:407] Connection header received
0926 23:31:40.670658 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:40.670666 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:40.671033 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:40.671233 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:40.671874 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:40.673054 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:40.674640 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:40.676037 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:40.676218 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:40.677614 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:40.677626 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:40.677647 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:31:40.678508 (+ 861us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:40.678579 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:31:40.678606 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:40.678845 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:40.679067 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 23:31:40.679241 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":302,"threads_started":1}
I20260926 23:31:40.681710 9946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:31:40.773734 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 9824
I20260926 23:31:40.790957 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 9955
I20260926 23:31:40.800549 10225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:40.792054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47086 (local address 127.0.97.2:43115)
0926 23:31:40.792741 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:40.792751 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:40.792785 (+ 34us) server_negotiation.cc:407] Connection header received
0926 23:31:40.792859 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:40.792866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:40.793168 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:40.793326 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:40.793798 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:40.795033 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:40.796490 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:40.797229 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:40.797350 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:40.798623 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:40.798633 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:40.798653 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:31:40.799538 (+ 885us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:40.799600 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:31:40.799627 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:40.799747 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:40.799974 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 23:31:40.800118 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":332,"threads_started":1}
I20260926 23:31:40.802245 10077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:31:40.891661 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 9955
I20260926 23:31:40.907245 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 10086
I20260926 23:31:40.916697 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:40.908368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44358 (local address 127.0.97.3:42089)
0926 23:31:40.908978 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:40.908986 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:40.909025 (+ 39us) server_negotiation.cc:407] Connection header received
0926 23:31:40.909098 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:40.909106 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:40.909386 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:40.909532 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:40.910056 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:40.911279 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:40.912686 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:40.913423 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:40.913542 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:40.914932 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:40.914949 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:40.914974 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:31:40.915722 (+ 748us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:40.915787 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:31:40.915815 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:40.915951 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:40.916138 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 23:31:40.916280 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":268,"threads_started":1}
I20260926 23:31:40.918344 10208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:31:41.008917 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 10086
I20260926 23:31:41.024758 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 9734
I20260926 23:31:41.026638 9795 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:31:41.143425 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 9734
2026-09-26T23:31:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (3038 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T23:31:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:31:41Z Disabled control of system clock
I20260926 23:31:41.230262 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46031
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 23:31:41.681499 10233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:31:41.681891 10233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:31:41.682017 10233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:31:41.690323 10233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:31:41.690490 10233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:31:41.690534 10233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:31:41.690622 10233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:31:41.690716 10233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:31:41.701122 10233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46031
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:31:41.702833 10233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:31:41.705024 10233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:41.718879 10239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:41.718838 10238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:41.718824 10241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:41.720163 10233 server_base.cc:1034] running on GCE node
I20260926 23:31:41.721222 10233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:41.723233 10233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:41.724498 10233 hybrid_clock.cc:648] HybridClock initialized: now 1790465501724440 us; error 63 us; skew 500 ppm
I20260926 23:31:41.724918 10233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:31:41.732002 10233 webserver.cc:460] Webserver started at http://127.0.97.62:39259/ using document root <none> and password file <none>
I20260926 23:31:41.732975 10233 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:41.733111 10233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:41.733503 10233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:31:41.737311 10233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "e3c7ee0bfdd7431d8fa9be6b6f7d4d9e"
format_stamp: "Formatted at 2026-09-26 23:31:41 on dist-test-slave-gwfx"
server_key: "60ce1d0146c82d1908ce683cee197c31"
server_key_iv: "b33adf4ecde121c9c4b541f2afcaa6ba"
server_key_version: "encryptionkey@0"
I20260926 23:31:41.738286 10233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "e3c7ee0bfdd7431d8fa9be6b6f7d4d9e"
format_stamp: "Formatted at 2026-09-26 23:31:41 on dist-test-slave-gwfx"
server_key: "60ce1d0146c82d1908ce683cee197c31"
server_key_iv: "b33adf4ecde121c9c4b541f2afcaa6ba"
server_key_version: "encryptionkey@0"
I20260926 23:31:41.745039 10233 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260926 23:31:41.749714 10247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:41.751762 10233 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 23:31:41.751997 10233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "e3c7ee0bfdd7431d8fa9be6b6f7d4d9e"
format_stamp: "Formatted at 2026-09-26 23:31:41 on dist-test-slave-gwfx"
server_key: "60ce1d0146c82d1908ce683cee197c31"
server_key_iv: "b33adf4ecde121c9c4b541f2afcaa6ba"
server_key_version: "encryptionkey@0"
I20260926 23:31:41.752269 10233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:41.759233 10233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:41.760229 10233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:31:41.760546 10233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:41.781445 10233 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46031
I20260926 23:31:41.781482 10298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46031 every 8 connection(s)
I20260926 23:31:41.783079 10233 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:31:41.787467 10299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:31:41.793504 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 10233
I20260926 23:31:41.793754 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:31:41.794443 388 external_mini_cluster.cc:1428] Setting key 4ae4372b6ce2073322e44216c433561b
I20260926 23:31:41.799731 10299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3c7ee0bfdd7431d8fa9be6b6f7d4d9e: Bootstrap starting.
I20260926 23:31:41.803907 10299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3c7ee0bfdd7431d8fa9be6b6f7d4d9e: Neither blocks nor log segments found. Creating new log.
I20260926 23:31:41.805159 10299 log.cc:824] T 00000000000000000000000000000000 P e3c7ee0bfdd7431d8fa9be6b6f7d4d9e: Log is configured to *not* fsync() on all Append() calls
I20260926 23:31:41.805202 10302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:41.797214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46818 (local address 127.0.97.62:46031)
0926 23:31:41.798474 (+ 1260us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:41.798489 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:41.798544 (+ 55us) server_negotiation.cc:407] Connection header received
0926 23:31:41.800679 (+ 2135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:41.800711 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:41.801341 (+ 630us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:41.801822 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:41.803156 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:41.803163 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:41.803178 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:31:41.803194 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:41.803303 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:31:41.803324 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:41.803487 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:41.803655 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 23:31:41.803989 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":396,"threads_started":1}
I20260926 23:31:41.808992 10299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3c7ee0bfdd7431d8fa9be6b6f7d4d9e: No bootstrap required, opened a new log
I20260926 23:31:41.816576 10299 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3c7ee0bfdd7431d8fa9be6b6f7d4d9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c7ee0bfdd7431d8fa9be6b6f7d4d9e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46031 } }
I20260926 23:31:41.817024 10299 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3c7ee0bfdd7431d8fa9be6b6f7d4d9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:31:41.817121 10299 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3c7ee0bfdd7431d8fa9be6b6f7d4d9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c7ee0bfdd7431d8fa9be6b6f7d4d9e, State: Initialized, Role: FOLLOWER
I20260926 23:31:41.817759 10299 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3c7ee0bfdd7431d8fa9be6b6f7d4d9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c7ee0bfdd7431d8fa9be6b6f7d4d9e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46031 } }
I20260926 23:31:41.818014 10299 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3c7ee0bfdd7431d8fa9be6b6f7d4d9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:31:41.818149 10299 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3c7ee0bfdd7431d8fa9be6b6f7d4d9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:31:41.818313 10299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3c7ee0bfdd7431d8fa9be6b6f7d4d9e [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:31:41.820829 10299 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3c7ee0bfdd7431d8fa9be6b6f7d4d9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c7ee0bfdd7431d8fa9be6b6f7d4d9e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46031 } }
I20260926 23:31:41.821419 10299 leader_election.cc:302] T 00000000000000000000000000000000 P e3c7ee0bfdd7431d8fa9be6b6f7d4d9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3c7ee0bfdd7431d8fa9be6b6f7d4d9e; no voters:
I20260926 23:31:41.822131 10299 leader_election.cc:288] T 00000000000000000000000000000000 P e3c7ee0bfdd7431d8fa9be6b6f7d4d9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:31:41.822371 10304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3c7ee0bfdd7431d8fa9be6b6f7d4d9e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:31:41.823305 10304 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3c7ee0bfdd7431d8fa9be6b6f7d4d9e [term 1 LEADER]: Becoming Leader. State: Replica: e3c7ee0bfdd7431d8fa9be6b6f7d4d9e, State: Running, Role: LEADER
I20260926 23:31:41.823989 10304 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3c7ee0bfdd7431d8fa9be6b6f7d4d9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c7ee0bfdd7431d8fa9be6b6f7d4d9e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46031 } }
I20260926 23:31:41.824504 10299 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3c7ee0bfdd7431d8fa9be6b6f7d4d9e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:31:41.828279 10306 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3c7ee0bfdd7431d8fa9be6b6f7d4d9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3c7ee0bfdd7431d8fa9be6b6f7d4d9e. Latest consensus state: current_term: 1 leader_uuid: "e3c7ee0bfdd7431d8fa9be6b6f7d4d9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c7ee0bfdd7431d8fa9be6b6f7d4d9e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46031 } } }
I20260926 23:31:41.828377 10305 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3c7ee0bfdd7431d8fa9be6b6f7d4d9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3c7ee0bfdd7431d8fa9be6b6f7d4d9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c7ee0bfdd7431d8fa9be6b6f7d4d9e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46031 } } }
I20260926 23:31:41.829047 10305 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3c7ee0bfdd7431d8fa9be6b6f7d4d9e [sys.catalog]: This master's current role is: LEADER
I20260926 23:31:41.829078 10306 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3c7ee0bfdd7431d8fa9be6b6f7d4d9e [sys.catalog]: This master's current role is: LEADER
I20260926 23:31:41.832029 10310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:31:41.838747 10310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:31:41.849785 10310 catalog_manager.cc:1347] Generated new cluster ID: e01285013fba4ff8b95b23ffb2c6ca4a
I20260926 23:31:41.849944 10310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:31:41.876150 10310 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:31:41.876751 10310 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:31:41.884073 10310 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3c7ee0bfdd7431d8fa9be6b6f7d4d9e: Generated new TSK 0
I20260926 23:31:41.885380 10310 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:31:41.899783 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46031
--builtin_ntp_servers=127.0.97.20:41647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 23:31:42.297556 10323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:31:42.297955 10323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:31:42.298098 10323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:31:42.306054 10323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:31:42.306205 10323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:31:42.306368 10323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:31:42.316723 10323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:31:42.318480 10323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:31:42.320845 10323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:42.338747 10328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:42.338811 10329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:42.338886 10331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:42.339759 10323 server_base.cc:1034] running on GCE node
I20260926 23:31:42.340466 10323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:42.341696 10323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:42.343034 10323 hybrid_clock.cc:648] HybridClock initialized: now 1790465502342982 us; error 55 us; skew 500 ppm
I20260926 23:31:42.343427 10323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:31:42.346222 10323 webserver.cc:460] Webserver started at http://127.0.97.1:41351/ using document root <none> and password file <none>
I20260926 23:31:42.347131 10323 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:42.347244 10323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:42.347587 10323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:31:42.350880 10323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/data/instance:
uuid: "3da99f5831ed404d8125d970938530d2"
format_stamp: "Formatted at 2026-09-26 23:31:42 on dist-test-slave-gwfx"
server_key: "a2214c85e9def398d688bc618b0c4893"
server_key_iv: "7d0dff781f40c3c6e44d3b6b2148944a"
server_key_version: "encryptionkey@0"
I20260926 23:31:42.351744 10323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3da99f5831ed404d8125d970938530d2"
format_stamp: "Formatted at 2026-09-26 23:31:42 on dist-test-slave-gwfx"
server_key: "a2214c85e9def398d688bc618b0c4893"
server_key_iv: "7d0dff781f40c3c6e44d3b6b2148944a"
server_key_version: "encryptionkey@0"
I20260926 23:31:42.358201 10323 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260926 23:31:42.362823 10337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:42.364852 10323 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:31:42.365079 10323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "3da99f5831ed404d8125d970938530d2"
format_stamp: "Formatted at 2026-09-26 23:31:42 on dist-test-slave-gwfx"
server_key: "a2214c85e9def398d688bc618b0c4893"
server_key_iv: "7d0dff781f40c3c6e44d3b6b2148944a"
server_key_version: "encryptionkey@0"
I20260926 23:31:42.365334 10323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:42.371636 10323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:42.372526 10323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:31:42.372784 10323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:42.373878 10323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:31:42.375610 10323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:31:42.375708 10323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:42.375818 10323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:31:42.375885 10323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:42.412256 10323 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37419
I20260926 23:31:42.412286 10449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37419 every 8 connection(s)
I20260926 23:31:42.413757 10323 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:31:42.419813 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 10323
I20260926 23:31:42.420037 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:31:42.420883 388 external_mini_cluster.cc:1428] Setting key 880b66afc3f4d9b2fca2964ba12662b9
I20260926 23:31:42.424739 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46031
--builtin_ntp_servers=127.0.97.20:41647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 23:31:42.425057 10452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:42.417337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55745 (local address 127.0.97.62:46031)
0926 23:31:42.417927 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:42.417933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:42.419379 (+ 1446us) server_negotiation.cc:407] Connection header received
0926 23:31:42.421662 (+ 2283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:42.421667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:42.421737 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:42.421889 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:42.423938 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:42.423945 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:42.423950 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:31:42.423976 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:42.424043 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:31:42.424063 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:42.424220 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:42.424564 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 23:31:42.424653 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":289,"threads_started":1}
I20260926 23:31:42.426240 10453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:42.417796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46031 (local address 127.0.97.1:55745)
0926 23:31:42.419124 (+ 1328us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:42.419186 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:42.421182 (+ 1996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:42.422284 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:42.422296 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:42.422442 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:42.423389 (+ 947us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:31:42.423440 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:42.423512 (+ 72us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:31:42.423653 (+ 141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:42.424217 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:42.424227 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:42.424235 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 23:31:42.424656 (+ 421us) client_negotiation.cc:240] Negotiation successful
0926 23:31:42.425044 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":350,"threads_started":1}
I20260926 23:31:42.428260 10450 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46031
I20260926 23:31:42.428745 10450 heartbeater.cc:464] Registering TS with master...
I20260926 23:31:42.429628 10450 heartbeater.cc:511] Master 127.0.97.62:46031 requested a full tablet report, sending...
I20260926 23:31:42.432758 10264 ts_manager.cc:194] Registered new tserver with Master: 3da99f5831ed404d8125d970938530d2 (127.0.97.1:37419)
W20260926 23:31:42.820817 10454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:31:42.821201 10454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:31:42.821303 10454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:31:42.829460 10454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:31:42.829598 10454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:31:42.829730 10454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:31:42.839828 10454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:31:42.841517 10454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:31:42.843739 10454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:42.859916 10459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:42.860000 10462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:42.859939 10460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:42.864657 10454 server_base.cc:1034] running on GCE node
I20260926 23:31:42.865649 10454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:42.866976 10454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:42.868310 10454 hybrid_clock.cc:648] HybridClock initialized: now 1790465502868256 us; error 45 us; skew 500 ppm
I20260926 23:31:42.868777 10454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:31:42.871876 10454 webserver.cc:460] Webserver started at http://127.0.97.2:44827/ using document root <none> and password file <none>
I20260926 23:31:42.872756 10454 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:42.872869 10454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:42.873236 10454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:31:42.877580 10454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/data/instance:
uuid: "2618da398c6a4a4d807c5f35e1194f81"
format_stamp: "Formatted at 2026-09-26 23:31:42 on dist-test-slave-gwfx"
server_key: "6b003d478fea5f00aa26898dedd92c37"
server_key_iv: "274fb68cb1e160069d23f723f2ad03eb"
server_key_version: "encryptionkey@0"
I20260926 23:31:42.878664 10454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2618da398c6a4a4d807c5f35e1194f81"
format_stamp: "Formatted at 2026-09-26 23:31:42 on dist-test-slave-gwfx"
server_key: "6b003d478fea5f00aa26898dedd92c37"
server_key_iv: "274fb68cb1e160069d23f723f2ad03eb"
server_key_version: "encryptionkey@0"
I20260926 23:31:42.886067 10454 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260926 23:31:42.890753 10468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:42.892620 10454 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:31:42.892843 10454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "2618da398c6a4a4d807c5f35e1194f81"
format_stamp: "Formatted at 2026-09-26 23:31:42 on dist-test-slave-gwfx"
server_key: "6b003d478fea5f00aa26898dedd92c37"
server_key_iv: "274fb68cb1e160069d23f723f2ad03eb"
server_key_version: "encryptionkey@0"
I20260926 23:31:42.893106 10454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:42.901018 10454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:42.902240 10454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:31:42.902565 10454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:42.903851 10454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:31:42.905752 10454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:31:42.905866 10454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:42.905999 10454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:31:42.906070 10454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:42.946852 10454 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43145
I20260926 23:31:42.946862 10580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43145 every 8 connection(s)
I20260926 23:31:42.948629 10454 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:31:42.955782 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 10454
I20260926 23:31:42.956037 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:31:42.956655 388 external_mini_cluster.cc:1428] Setting key 412a176da5c0752a800ca3a7c7f3061d
I20260926 23:31:42.960604 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46031
--builtin_ntp_servers=127.0.97.20:41647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 23:31:42.961305 10583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:42.952152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58787 (local address 127.0.97.62:46031)
0926 23:31:42.952806 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:42.952812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:42.954554 (+ 1742us) server_negotiation.cc:407] Connection header received
0926 23:31:42.957535 (+ 2981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:42.957541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:42.957630 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:42.957823 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:42.960183 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:42.960189 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:42.960194 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:31:42.960220 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:42.960283 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:31:42.960303 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:42.960503 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:42.960906 (+ 403us) server_negotiation.cc:299] Negotiation successful
0926 23:31:42.960980 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":314,"threads_started":1}
I20260926 23:31:42.962754 10584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:42.952746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46031 (local address 127.0.97.2:58787)
0926 23:31:42.954287 (+ 1541us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:42.954351 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:42.956983 (+ 2632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:42.958621 (+ 1638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:42.958638 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:42.958885 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:42.959750 (+ 865us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:31:42.959753 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:42.959808 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:31:42.959938 (+ 130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:42.960498 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:42.960508 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:42.960517 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 23:31:42.960949 (+ 432us) client_negotiation.cc:240] Negotiation successful
0926 23:31:42.961293 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"thread_start_us":449,"threads_started":1}
I20260926 23:31:42.965134 10581 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46031
I20260926 23:31:42.965834 10581 heartbeater.cc:464] Registering TS with master...
I20260926 23:31:42.966825 10581 heartbeater.cc:511] Master 127.0.97.62:46031 requested a full tablet report, sending...
I20260926 23:31:42.969226 10264 ts_manager.cc:194] Registered new tserver with Master: 2618da398c6a4a4d807c5f35e1194f81 (127.0.97.2:43145)
W20260926 23:31:43.313840 10585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:31:43.314325 10585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:31:43.314554 10585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:31:43.325752 10585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:31:43.325919 10585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:31:43.326079 10585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:31:43.337134 10585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:31:43.338546 10585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:31:43.340613 10585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:43.357697 10590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:43.357589 10591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:43.357869 10593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:43.359113 10585 server_base.cc:1034] running on GCE node
I20260926 23:31:43.359853 10585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:43.361166 10585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:43.362458 10585 hybrid_clock.cc:648] HybridClock initialized: now 1790465503362367 us; error 74 us; skew 500 ppm
I20260926 23:31:43.362903 10585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:31:43.366096 10585 webserver.cc:460] Webserver started at http://127.0.97.3:40765/ using document root <none> and password file <none>
I20260926 23:31:43.366976 10585 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:43.367091 10585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:43.367460 10585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:31:43.370977 10585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/data/instance:
uuid: "2b542e0f3f6241fe81440e43c60ec012"
format_stamp: "Formatted at 2026-09-26 23:31:43 on dist-test-slave-gwfx"
server_key: "93117e476ab7495d4d3a6360d1721c78"
server_key_iv: "8c74c9c60bf1b24a1e901cf667c51141"
server_key_version: "encryptionkey@0"
I20260926 23:31:43.371851 10585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2b542e0f3f6241fe81440e43c60ec012"
format_stamp: "Formatted at 2026-09-26 23:31:43 on dist-test-slave-gwfx"
server_key: "93117e476ab7495d4d3a6360d1721c78"
server_key_iv: "8c74c9c60bf1b24a1e901cf667c51141"
server_key_version: "encryptionkey@0"
I20260926 23:31:43.378175 10585 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 23:31:43.383153 10599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:43.385192 10585 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260926 23:31:43.385421 10585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "2b542e0f3f6241fe81440e43c60ec012"
format_stamp: "Formatted at 2026-09-26 23:31:43 on dist-test-slave-gwfx"
server_key: "93117e476ab7495d4d3a6360d1721c78"
server_key_iv: "8c74c9c60bf1b24a1e901cf667c51141"
server_key_version: "encryptionkey@0"
I20260926 23:31:43.385692 10585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:43.392619 10585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:43.393573 10585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:31:43.393855 10585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:43.395128 10585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:31:43.397817 10585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:31:43.397969 10585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:43.398090 10585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:31:43.398175 10585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:43.435390 10585 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41981
I20260926 23:31:43.435513 10711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41981 every 8 connection(s)
I20260926 23:31:43.436810 10585 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:31:43.437148 10450 heartbeater.cc:503] Master 127.0.97.62:46031 was elected leader, sending a full tablet report...
I20260926 23:31:43.439021 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 10585
I20260926 23:31:43.439201 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790465400694755-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:31:43.439754 388 external_mini_cluster.cc:1428] Setting key b93b546d409d63776710494afb583652
I20260926 23:31:43.448599 10583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:43.441248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50619 (local address 127.0.97.62:46031)
0926 23:31:43.441445 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:43.441450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:43.443759 (+ 2309us) server_negotiation.cc:407] Connection header received
0926 23:31:43.445737 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:43.445742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:43.445903 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:43.446020 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:43.447803 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:43.447810 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:43.447814 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:31:43.447834 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:43.447872 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:31:43.447885 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:43.448006 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:43.448208 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 23:31:43.448292 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 23:31:43.449949 10714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:43.442070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46031 (local address 127.0.97.3:50619)
0926 23:31:43.443588 (+ 1518us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:43.443644 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:43.445408 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:43.446378 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:43.446390 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:43.446530 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:43.447390 (+ 860us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:31:43.447392 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:43.447447 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:31:43.447574 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:43.447979 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:43.447989 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:43.447997 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 23:31:43.448312 (+ 315us) client_negotiation.cc:240] Negotiation successful
0926 23:31:43.448719 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":375,"threads_started":1}
I20260926 23:31:43.452087 10712 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46031
I20260926 23:31:43.452500 10712 heartbeater.cc:464] Registering TS with master...
I20260926 23:31:43.453285 10712 heartbeater.cc:511] Master 127.0.97.62:46031 requested a full tablet report, sending...
I20260926 23:31:43.455329 10263 ts_manager.cc:194] Registered new tserver with Master: 2b542e0f3f6241fe81440e43c60ec012 (127.0.97.3:41981)
I20260926 23:31:43.456787 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 23:31:43.465494 10583 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:31:43.463228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46834 (local address 127.0.97.62:46031)
0926 23:31:43.463538 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:43.463547 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:43.464084 (+ 537us) server_negotiation.cc:407] Connection header received
0926 23:31:43.464292 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:43.464297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:43.464367 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:43.464509 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:43.465145 (+ 636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46834: BlockingRecv error: recv got EOF from 127.0.0.1:46834 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
I20260926 23:31:43.466091 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 10323
I20260926 23:31:43.471880 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:43.468747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39670 (local address 127.0.97.1:37419)
0926 23:31:43.469374 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:43.469384 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:43.469425 (+ 41us) server_negotiation.cc:407] Connection header received
0926 23:31:43.469505 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:43.469512 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:43.469751 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:43.469911 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:43.471124 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:43.471132 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:43.471145 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:31:43.471160 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:43.471206 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:31:43.471221 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:43.471383 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:43.471466 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 23:31:43.471555 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":247,"threads_started":1}
I20260926 23:31:43.473337 10445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:31:43.568014 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 10323
I20260926 23:31:43.585410 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 10454
I20260926 23:31:43.589869 10723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:43.586657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45180 (local address 127.0.97.2:43145)
0926 23:31:43.587362 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:43.587371 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:43.587408 (+ 37us) server_negotiation.cc:407] Connection header received
0926 23:31:43.587527 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:43.587534 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:43.587830 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:43.587989 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:43.589017 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:43.589025 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:43.589039 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:31:43.589057 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:43.589103 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:31:43.589119 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:43.589264 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:43.589367 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 23:31:43.589530 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":301,"threads_started":1}
I20260926 23:31:43.591606 10576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:31:43.678056 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 10454
I20260926 23:31:43.693444 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 10585
I20260926 23:31:43.698161 10725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:43.694733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56472 (local address 127.0.97.3:41981)
0926 23:31:43.695514 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:43.695522 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:43.695554 (+ 32us) server_negotiation.cc:407] Connection header received
0926 23:31:43.695669 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:43.695676 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:43.695913 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:43.696094 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:43.697266 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:43.697272 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:43.697285 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:31:43.697302 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:43.697340 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:31:43.697354 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:43.697525 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:43.697607 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 23:31:43.697716 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":403,"threads_started":1}
I20260926 23:31:43.699939 10707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:31:43.791508 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 10585
I20260926 23:31:43.806529 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 10233
I20260926 23:31:43.808013 10294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:31:43.922103 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 10233
2026-09-26T23:31:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2775 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:31:44 dist-test-slave-gwfx krb5kdc[10730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:31:44 dist-test-slave-gwfx krb5kdc[10730](info): set up 2 sockets
Sep 26 23:31:44 dist-test-slave-gwfx krb5kdc[10730](info): commencing operation
krb5kdc: starting...
W20260926 23:31:44.975720 388 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:31:45 dist-test-slave-gwfx krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:31:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:31:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:31:45.258450 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42271
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42271
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:31:45.658195 10743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:31:45.658620 10743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:31:45.658787 10743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:31:45.667870 10743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:31:45.668056 10743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:31:45.668144 10743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:31:45.668228 10743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:31:45.668303 10743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:31:45.680038 10743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:31:45.681881 10743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:31:45.683560 10743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:45.698477 10748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:45.698619 10751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:45.698880 10749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:45.700161 10743 server_base.cc:1034] running on GCE node
I20260926 23:31:45.701198 10743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:45.703004 10743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:45.704291 10743 hybrid_clock.cc:648] HybridClock initialized: now 1790465505704204 us; error 71 us; skew 500 ppm
I20260926 23:31:45.704730 10743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:31:45 dist-test-slave-gwfx krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:31:45.708870 10743 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:31:45.711262 10743 webserver.cc:460] Webserver started at http://127.0.97.62:45071/ using document root <none> and password file <none>
I20260926 23:31:45.712175 10743 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:45.712313 10743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:45.712636 10743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:31:45.716593 10743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "6a8d39262e9644fd8ceb583a3cb0b0c9"
format_stamp: "Formatted at 2026-09-26 23:31:45 on dist-test-slave-gwfx"
server_key: "6b687c77c27381d65b49ca7472d0362f"
server_key_iv: "bf7ea062276710ef04623d523faf9ae4"
server_key_version: "encryptionkey@0"
I20260926 23:31:45.717736 10743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "6a8d39262e9644fd8ceb583a3cb0b0c9"
format_stamp: "Formatted at 2026-09-26 23:31:45 on dist-test-slave-gwfx"
server_key: "6b687c77c27381d65b49ca7472d0362f"
server_key_iv: "bf7ea062276710ef04623d523faf9ae4"
server_key_version: "encryptionkey@0"
I20260926 23:31:45.725157 10743 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260926 23:31:45.730113 10758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:45.732451 10743 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 23:31:45.732710 10743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "6a8d39262e9644fd8ceb583a3cb0b0c9"
format_stamp: "Formatted at 2026-09-26 23:31:45 on dist-test-slave-gwfx"
server_key: "6b687c77c27381d65b49ca7472d0362f"
server_key_iv: "bf7ea062276710ef04623d523faf9ae4"
server_key_version: "encryptionkey@0"
I20260926 23:31:45.732941 10743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:45.783571 10743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:45.789078 10743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:31:45.789376 10743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:45.810413 10743 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42271
I20260926 23:31:45.810446 10809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42271 every 8 connection(s)
I20260926 23:31:45.812117 10743 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:31:45.817236 10810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:31:45.819744 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 10743
I20260926 23:31:45.819998 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:31:45.820762 388 external_mini_cluster.cc:1428] Setting key 4142565de859abfc7163e05e58fa1c05
I20260926 23:31:45.831423 10810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a8d39262e9644fd8ceb583a3cb0b0c9: Bootstrap starting.
I20260926 23:31:45.836045 10810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a8d39262e9644fd8ceb583a3cb0b0c9: Neither blocks nor log segments found. Creating new log.
I20260926 23:31:45.837643 10810 log.cc:824] T 00000000000000000000000000000000 P 6a8d39262e9644fd8ceb583a3cb0b0c9: Log is configured to *not* fsync() on all Append() calls
Sep 26 23:31:45 dist-test-slave-gwfx krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:31:45.842024 10810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a8d39262e9644fd8ceb583a3cb0b0c9: No bootstrap required, opened a new log
I20260926 23:31:45.849809 10813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:45.823435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57394 (local address 127.0.97.62:42271)
0926 23:31:45.824435 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:45.824451 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:45.824506 (+ 55us) server_negotiation.cc:407] Connection header received
0926 23:31:45.826504 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:45.826535 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:45.827226 (+ 691us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:45.827716 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:45.829739 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:45.831483 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:45.833768 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:45.834772 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:45.834995 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:45.840390 (+ 5395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:45.840401 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:45.840423 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:45.840479 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:45.844754 (+ 4275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:45.845748 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:45.845757 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:45.845766 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:45.845969 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:45.846365 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:45.846368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:45.846371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:45.847105 (+ 734us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:45.847468 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:45.847929 (+ 461us) server_negotiation.cc:299] Negotiation successful
0926 23:31:45.848452 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":457,"threads_started":1}
I20260926 23:31:45.851362 10810 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a8d39262e9644fd8ceb583a3cb0b0c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a8d39262e9644fd8ceb583a3cb0b0c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42271 } }
I20260926 23:31:45.851796 10810 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a8d39262e9644fd8ceb583a3cb0b0c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:31:45.851899 10810 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a8d39262e9644fd8ceb583a3cb0b0c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a8d39262e9644fd8ceb583a3cb0b0c9, State: Initialized, Role: FOLLOWER
I20260926 23:31:45.852562 10810 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a8d39262e9644fd8ceb583a3cb0b0c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a8d39262e9644fd8ceb583a3cb0b0c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42271 } }
I20260926 23:31:45.852819 10810 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a8d39262e9644fd8ceb583a3cb0b0c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:31:45.852964 10810 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a8d39262e9644fd8ceb583a3cb0b0c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:31:45.853181 10810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a8d39262e9644fd8ceb583a3cb0b0c9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:31:45.855666 10810 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a8d39262e9644fd8ceb583a3cb0b0c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a8d39262e9644fd8ceb583a3cb0b0c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42271 } }
I20260926 23:31:45.856271 10810 leader_election.cc:302] T 00000000000000000000000000000000 P 6a8d39262e9644fd8ceb583a3cb0b0c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a8d39262e9644fd8ceb583a3cb0b0c9; no voters:
I20260926 23:31:45.857000 10810 leader_election.cc:288] T 00000000000000000000000000000000 P 6a8d39262e9644fd8ceb583a3cb0b0c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:31:45.857173 10815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a8d39262e9644fd8ceb583a3cb0b0c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:31:45.858315 10815 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a8d39262e9644fd8ceb583a3cb0b0c9 [term 1 LEADER]: Becoming Leader. State: Replica: 6a8d39262e9644fd8ceb583a3cb0b0c9, State: Running, Role: LEADER
I20260926 23:31:45.859030 10815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a8d39262e9644fd8ceb583a3cb0b0c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a8d39262e9644fd8ceb583a3cb0b0c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42271 } }
I20260926 23:31:45.859584 10810 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a8d39262e9644fd8ceb583a3cb0b0c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:31:45.861925 10816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a8d39262e9644fd8ceb583a3cb0b0c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a8d39262e9644fd8ceb583a3cb0b0c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a8d39262e9644fd8ceb583a3cb0b0c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42271 } } }
I20260926 23:31:45.862301 10816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a8d39262e9644fd8ceb583a3cb0b0c9 [sys.catalog]: This master's current role is: LEADER
I20260926 23:31:45.863022 10817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a8d39262e9644fd8ceb583a3cb0b0c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a8d39262e9644fd8ceb583a3cb0b0c9. Latest consensus state: current_term: 1 leader_uuid: "6a8d39262e9644fd8ceb583a3cb0b0c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a8d39262e9644fd8ceb583a3cb0b0c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42271 } } }
I20260926 23:31:45.863503 10817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a8d39262e9644fd8ceb583a3cb0b0c9 [sys.catalog]: This master's current role is: LEADER
I20260926 23:31:45.864734 10820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:31:45.871628 10820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:31:45.882947 10820 catalog_manager.cc:1347] Generated new cluster ID: 55ff1b4e7ccc40f5a24e625259b10eb9
I20260926 23:31:45.883102 10820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:31:45.925037 10820 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:31:45.927245 10820 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:31:45.938146 10820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a8d39262e9644fd8ceb583a3cb0b0c9: Generated new TSK 0
I20260926 23:31:45.939625 10820 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:31:46.011991 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42271
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:31:46.417294 10836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:31:46.417678 10836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:31:46.417793 10836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:31:46.425765 10836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:31:46.425926 10836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:31:46.426057 10836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:31:46.437115 10836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:31:46.438906 10836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:31:46.440583 10836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:46.456703 10842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:46.456619 10841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:46.456604 10844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:46.463057 10836 server_base.cc:1034] running on GCE node
I20260926 23:31:46.464056 10836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:46.465461 10836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:46.466789 10836 hybrid_clock.cc:648] HybridClock initialized: now 1790465506466705 us; error 77 us; skew 500 ppm
I20260926 23:31:46.467195 10836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:31:46 dist-test-slave-gwfx krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:31:46.471383 10836 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:31:46.473488 10836 webserver.cc:460] Webserver started at http://127.0.97.1:33785/ using document root <none> and password file <none>
I20260926 23:31:46.474436 10836 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:46.474566 10836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:46.474957 10836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:31:46.479041 10836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-0/data/instance:
uuid: "f80dd6f69aa843088dc5b7a0cdac5c48"
format_stamp: "Formatted at 2026-09-26 23:31:46 on dist-test-slave-gwfx"
server_key: "2449708274b4b34408a82e953a6747ed"
server_key_iv: "5d7291d3d25d2bfd00ad0d8a29387b95"
server_key_version: "encryptionkey@0"
I20260926 23:31:46.480108 10836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f80dd6f69aa843088dc5b7a0cdac5c48"
format_stamp: "Formatted at 2026-09-26 23:31:46 on dist-test-slave-gwfx"
server_key: "2449708274b4b34408a82e953a6747ed"
server_key_iv: "5d7291d3d25d2bfd00ad0d8a29387b95"
server_key_version: "encryptionkey@0"
I20260926 23:31:46.487247 10836 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260926 23:31:46.492122 10851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:46.494424 10836 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 23:31:46.494649 10836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "f80dd6f69aa843088dc5b7a0cdac5c48"
format_stamp: "Formatted at 2026-09-26 23:31:46 on dist-test-slave-gwfx"
server_key: "2449708274b4b34408a82e953a6747ed"
server_key_iv: "5d7291d3d25d2bfd00ad0d8a29387b95"
server_key_version: "encryptionkey@0"
I20260926 23:31:46.494982 10836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:46.513856 10836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:46.518762 10836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:31:46.519109 10836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:46.520252 10836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:31:46.521938 10836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:31:46.522015 10836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:46.522142 10836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:31:46.522236 10836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:46.558938 10836 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44679
I20260926 23:31:46.558952 10963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44679 every 8 connection(s)
I20260926 23:31:46.560489 10836 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:31:46.562024 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 10836
I20260926 23:31:46.562289 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:31:46.563216 388 external_mini_cluster.cc:1428] Setting key 0e635aa85e9e996e228204bf104d6dc7
Sep 26 23:31:46 dist-test-slave-gwfx krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:31:46.583534 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:46.563735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39273 (local address 127.0.97.62:42271)
0926 23:31:46.564225 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:46.564230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:46.565663 (+ 1433us) server_negotiation.cc:407] Connection header received
0926 23:31:46.567841 (+ 2178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:46.567847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:46.567942 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:46.568048 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:46.569862 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:46.570716 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:46.572426 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:46.572863 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:46.572977 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:46.577161 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:46.577167 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:46.577171 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:46.577212 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:46.579927 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:46.580810 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:46.580816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:46.580820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:46.580962 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:46.581452 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:46.581456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:46.581458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:46.581774 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:46.581916 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:46.582844 (+ 928us) server_negotiation.cc:299] Negotiation successful
0926 23:31:46.583121 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":236,"threads_started":1}
I20260926 23:31:46.584836 10967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:46.564519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42271 (local address 127.0.97.1:39273)
0926 23:31:46.565507 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:46.565565 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:46.567534 (+ 1969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:46.568357 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:46.568369 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:46.568993 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:46.569689 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:46.569709 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:46.570884 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:46.570888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:46.572271 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:46.572284 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:46.573108 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:46.573126 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:46.573381 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:46.574105 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:46.574139 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:46.576867 (+ 2728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:46.580107 (+ 3240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:46.580127 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:46.580157 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:46.580678 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:46.581075 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:46.581082 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:46.581086 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:46.581327 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:46.581904 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:46.581915 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:46.582376 (+ 461us) client_negotiation.cc:769] Sending connection context
0926 23:31:46.582674 (+ 298us) client_negotiation.cc:240] Negotiation successful
0926 23:31:46.583395 (+ 721us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":383,"threads_started":1}
I20260926 23:31:46.586843 10964 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42271
I20260926 23:31:46.587260 10964 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 23:31:46.588382 10964 heartbeater.cc:511] Master 127.0.97.62:42271 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 23:31:46.591138 10775 ts_manager.cc:194] Registered new tserver with Master: f80dd6f69aa843088dc5b7a0cdac5c48 (127.0.97.1:44679)
I20260926 23:31:46.592849 10775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39273
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:31:46.622455 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42271
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:31:47.020473 10970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:31:47.020864 10970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:31:47.020992 10970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:31:47.029248 10970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:31:47.029424 10970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:31:47.029588 10970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:31:47.040771 10970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:31:47.042531 10970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:31:47.044092 10970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:47.059373 10978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:47.059294 10976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:47.059386 10975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:47.064700 10970 server_base.cc:1034] running on GCE node
I20260926 23:31:47.065594 10970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:47.066977 10970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:47.068218 10970 hybrid_clock.cc:648] HybridClock initialized: now 1790465507068168 us; error 70 us; skew 500 ppm
I20260926 23:31:47.068601 10970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:31:47 dist-test-slave-gwfx krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:31:47.077939 10970 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:31:47.079984 10970 webserver.cc:460] Webserver started at http://127.0.97.2:38405/ using document root <none> and password file <none>
I20260926 23:31:47.080924 10970 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:47.081040 10970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:47.081355 10970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:31:47.084877 10970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-1/data/instance:
uuid: "a80321ad8ed14f42980851d9390e8e9a"
format_stamp: "Formatted at 2026-09-26 23:31:47 on dist-test-slave-gwfx"
server_key: "a709165e352ad98cb7dc1cd63bb6ea64"
server_key_iv: "eeeb6b837536d6305b1ae1681ffd411b"
server_key_version: "encryptionkey@0"
I20260926 23:31:47.085829 10970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a80321ad8ed14f42980851d9390e8e9a"
format_stamp: "Formatted at 2026-09-26 23:31:47 on dist-test-slave-gwfx"
server_key: "a709165e352ad98cb7dc1cd63bb6ea64"
server_key_iv: "eeeb6b837536d6305b1ae1681ffd411b"
server_key_version: "encryptionkey@0"
I20260926 23:31:47.092264 10970 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260926 23:31:47.096840 10985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:47.098771 10970 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260926 23:31:47.098992 10970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "a80321ad8ed14f42980851d9390e8e9a"
format_stamp: "Formatted at 2026-09-26 23:31:47 on dist-test-slave-gwfx"
server_key: "a709165e352ad98cb7dc1cd63bb6ea64"
server_key_iv: "eeeb6b837536d6305b1ae1681ffd411b"
server_key_version: "encryptionkey@0"
I20260926 23:31:47.099303 10970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:47.123456 10970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:47.128723 10970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:31:47.129069 10970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:47.130299 10970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:31:47.132021 10970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:31:47.132105 10970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:47.132241 10970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:31:47.132293 10970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:47.169046 10970 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38003
I20260926 23:31:47.169107 11097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38003 every 8 connection(s)
I20260926 23:31:47.170552 10970 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:31:47.172894 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 10970
I20260926 23:31:47.173092 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:31:47.173564 388 external_mini_cluster.cc:1428] Setting key 8d233c741f00f3a69df636fc119cc04e
Sep 26 23:31:47 dist-test-slave-gwfx krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:31:47.196431 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:47.173685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35515 (local address 127.0.97.62:42271)
0926 23:31:47.174228 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:47.174236 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:47.175640 (+ 1404us) server_negotiation.cc:407] Connection header received
0926 23:31:47.178392 (+ 2752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:47.178397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:47.178481 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:47.178586 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:47.181080 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:47.181925 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:47.184642 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:47.185133 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:47.185250 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:47.189722 (+ 4472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:47.189729 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:47.189733 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:47.189775 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:47.192723 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:47.193673 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:47.193678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:47.193681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:47.193836 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:47.194326 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:47.194330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:47.194332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:47.194644 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:47.194836 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:47.195719 (+ 883us) server_negotiation.cc:299] Negotiation successful
0926 23:31:47.196006 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":247,"threads_started":1}
I20260926 23:31:47.197733 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:47.174371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42271 (local address 127.0.97.2:35515)
0926 23:31:47.175415 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:47.175489 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:47.177998 (+ 2509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:47.178969 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:47.178986 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:47.179855 (+ 869us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:47.180825 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:47.180856 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:47.182122 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:47.182127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:47.184442 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:47.184461 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:47.185426 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:47.185433 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:47.185756 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:47.186646 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:47.186679 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:47.189388 (+ 2709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:47.192935 (+ 3547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:47.192944 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:47.192984 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:47.193524 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:47.193961 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:47.193967 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:47.193971 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:47.194201 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:47.194795 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:47.194813 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:47.195282 (+ 469us) client_negotiation.cc:769] Sending connection context
0926 23:31:47.195590 (+ 308us) client_negotiation.cc:240] Negotiation successful
0926 23:31:47.196293 (+ 703us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":437,"threads_started":1}
I20260926 23:31:47.199837 11098 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42271
I20260926 23:31:47.200255 11098 heartbeater.cc:464] Registering TS with master...
I20260926 23:31:47.201197 11098 heartbeater.cc:511] Master 127.0.97.62:42271 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 23:31:47.203104 10775 ts_manager.cc:194] Registered new tserver with Master: a80321ad8ed14f42980851d9390e8e9a (127.0.97.2:38003)
I20260926 23:31:47.204097 10775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:31:47.236195 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42271
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:31:47.596578 10964 heartbeater.cc:503] Master 127.0.97.62:42271 was elected leader, sending a full tablet report...
W20260926 23:31:47.635879 11104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:31:47.636260 11104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:31:47.636394 11104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:31:47.644547 11104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:31:47.644706 11104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:31:47.644873 11104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:31:47.656030 11104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:31:47.657754 11104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:31:47.659759 11104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:31:47.677073 11112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:47.677181 11109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:31:47.677151 11110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:31:47.678164 11104 server_base.cc:1034] running on GCE node
I20260926 23:31:47.679162 11104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:31:47.680516 11104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:31:47.681844 11104 hybrid_clock.cc:648] HybridClock initialized: now 1790465507681700 us; error 85 us; skew 500 ppm
I20260926 23:31:47.682278 11104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:31:47 dist-test-slave-gwfx krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:31:47.686870 11104 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:31:47.688876 11104 webserver.cc:460] Webserver started at http://127.0.97.3:35283/ using document root <none> and password file <none>
I20260926 23:31:47.689868 11104 fs_manager.cc:359] Metadata directory not provided
I20260926 23:31:47.690006 11104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:31:47.690384 11104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:31:47.694291 11104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-2/data/instance:
uuid: "26dd3ef6a8914933923af9bccb3caf02"
format_stamp: "Formatted at 2026-09-26 23:31:47 on dist-test-slave-gwfx"
server_key: "0cbe6359f3d7295f12f545d5614a9a28"
server_key_iv: "39f1d65ec3b83f029ab988c03579fd89"
server_key_version: "encryptionkey@0"
I20260926 23:31:47.695344 11104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-2/wal/instance:
uuid: "26dd3ef6a8914933923af9bccb3caf02"
format_stamp: "Formatted at 2026-09-26 23:31:47 on dist-test-slave-gwfx"
server_key: "0cbe6359f3d7295f12f545d5614a9a28"
server_key_iv: "39f1d65ec3b83f029ab988c03579fd89"
server_key_version: "encryptionkey@0"
I20260926 23:31:47.702102 11104 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 23:31:47.706979 11119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:47.708959 11104 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260926 23:31:47.709177 11104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "26dd3ef6a8914933923af9bccb3caf02"
format_stamp: "Formatted at 2026-09-26 23:31:47 on dist-test-slave-gwfx"
server_key: "0cbe6359f3d7295f12f545d5614a9a28"
server_key_iv: "39f1d65ec3b83f029ab988c03579fd89"
server_key_version: "encryptionkey@0"
I20260926 23:31:47.709414 11104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:31:47.736584 11104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:31:47.741884 11104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:31:47.742228 11104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:31:47.743454 11104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:31:47.745046 11104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:31:47.745128 11104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:47.745255 11104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:31:47.745311 11104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:31:47.779755 11104 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43849
I20260926 23:31:47.779814 11231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43849 every 8 connection(s)
I20260926 23:31:47.781275 11104 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:31:47.786365 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 11104
I20260926 23:31:47.786561 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:31:47.787045 388 external_mini_cluster.cc:1428] Setting key 26944973d9fd037538df6fff4b60b002
Sep 26 23:31:47 dist-test-slave-gwfx krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:31:47.804489 11234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:47.784384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50775 (local address 127.0.97.62:42271)
0926 23:31:47.784980 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:47.784985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:47.785913 (+ 928us) server_negotiation.cc:407] Connection header received
0926 23:31:47.788083 (+ 2170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:47.788090 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:47.788181 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:47.788280 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:47.789967 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:47.790836 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:47.792526 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:47.792955 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:47.793065 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:47.797347 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:47.797353 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:47.797356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:47.797394 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:47.800229 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:47.801403 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:47.801409 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:47.801413 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:47.801591 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:47.802265 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:47.802272 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:47.802275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:47.802722 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:47.802898 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:47.803854 (+ 956us) server_negotiation.cc:299] Negotiation successful
0926 23:31:47.804085 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":278,"threads_started":1}
I20260926 23:31:47.805838 11235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:47.784931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42271 (local address 127.0.97.3:50775)
0926 23:31:47.785731 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:47.785783 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:47.787805 (+ 2022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:47.788543 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:47.788552 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:47.789138 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:31:47.789803 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:47.789821 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:47.790985 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:47.790989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:47.792386 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:47.792399 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:47.793187 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:47.793194 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:47.793435 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:47.794169 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:31:47.794193 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:31:47.797065 (+ 2872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:31:47.800480 (+ 3415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:47.800493 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:47.800516 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:47.801259 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:47.801787 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:31:47.801793 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:31:47.801796 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:31:47.802122 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:31:47.802872 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:31:47.802879 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:31:47.803434 (+ 555us) client_negotiation.cc:769] Sending connection context
0926 23:31:47.803765 (+ 331us) client_negotiation.cc:240] Negotiation successful
0926 23:31:47.804349 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":304,"threads_started":1}
I20260926 23:31:47.807749 11232 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42271
I20260926 23:31:47.808112 11232 heartbeater.cc:464] Registering TS with master...
I20260926 23:31:47.808986 11232 heartbeater.cc:511] Master 127.0.97.62:42271 requested a full tablet report, sending...
I20260926 23:31:47.810879 10775 ts_manager.cc:194] Registered new tserver with Master: 26dd3ef6a8914933923af9bccb3caf02 (127.0.97.3:43849)
I20260926 23:31:47.811846 10775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50775
I20260926 23:31:47.816299 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:31:47.836839 11234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:47.821925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44374 (local address 127.0.97.62:42271)
0926 23:31:47.822183 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:47.822191 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:47.822654 (+ 463us) server_negotiation.cc:407] Connection header received
0926 23:31:47.823258 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:47.823264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:47.823363 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:47.823462 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:47.825199 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:47.826082 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:47.827551 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:47.828003 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:47.828113 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:47.830938 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:47.830943 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:47.830947 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:47.830987 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:47.833757 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:47.834540 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:47.834545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:47.834547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:47.834708 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:47.835128 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:47.835132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:47.835135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:47.835552 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:47.835845 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:47.836190 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 23:31:47.836413 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 23:31:47.841790 10775 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:31:47.849370 10775 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:31:47.876579 11248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:47.865695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38210 (local address 127.0.97.2:38003)
0926 23:31:47.866417 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:47.866453 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:47.867293 (+ 840us) server_negotiation.cc:407] Connection header received
0926 23:31:47.867631 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:47.867642 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:47.867913 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:47.868073 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:31:47.869057 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:47.870518 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:47.873213 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:47.874842 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:47.874959 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:47.875217 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:47.875966 (+ 749us) server_negotiation.cc:299] Negotiation successful
0926 23:31:47.876187 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":294,"threads_started":1}
I20260926 23:31:47.877533 11247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:47.865125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38003 (local address 127.0.0.1:38210)
0926 23:31:47.867149 (+ 2024us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:47.867176 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:47.867410 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:47.868441 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:47.868448 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:47.868480 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:31:47.868877 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:47.868886 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:47.870651 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:47.870657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:47.873017 (+ 2360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:47.873032 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:47.875366 (+ 2334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:47.875373 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:47.875666 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:47.875728 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 23:31:47.876928 (+ 1200us) client_negotiation.cc:240] Negotiation successful
0926 23:31:47.877185 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1907,"mutex_wait_us":21,"thread_start_us":1760,"threads_started":1}
I20260926 23:31:47.879884 11249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:47.866667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53598 (local address 127.0.97.3:43849)
0926 23:31:47.867646 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:47.867654 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:47.871354 (+ 3700us) server_negotiation.cc:407] Connection header received
0926 23:31:47.871671 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:47.871680 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:47.871968 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:47.872191 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:31:47.873112 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:47.874726 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:47.877270 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:47.878804 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:47.878924 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:47.879189 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:47.879363 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 23:31:47.879534 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":795,"thread_start_us":609,"threads_started":1}
I20260926 23:31:47.880604 11250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:47.867042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43849 (local address 127.0.0.1:53598)
0926 23:31:47.871235 (+ 4193us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:47.871254 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:47.871465 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:47.872365 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:47.872370 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:47.872400 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:31:47.872884 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:47.872896 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:47.874900 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:47.874906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:47.877078 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:47.877091 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:47.878986 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:47.878992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:47.879157 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:47.879217 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 23:31:47.879314 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 23:31:47.879463 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4039,"mutex_wait_us":4,"thread_start_us":3880,"threads_started":1}
I20260926 23:31:47.882218 11033 tablet_service.cc:1467] Processing CreateTablet for tablet cce06dff25f5491091bc84490b86a47d (DEFAULT_TABLE table=test-table [id=a1ec14f86b9c449cb7aa8d4f74eb89e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:31:47.883365 11245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:47.864427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44679 (local address 127.0.0.1:60064)
0926 23:31:47.865338 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:47.865396 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:47.865609 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:47.874084 (+ 8475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:47.874089 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:47.874114 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:31:47.874532 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:47.874542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:47.876353 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:47.876359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:47.878369 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:47.878383 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:47.882487 (+ 4104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:47.882494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:47.882726 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:47.882761 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 23:31:47.882866 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 23:31:47.883023 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":428,"threads_started":1}
I20260926 23:31:47.883474 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:47.864740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60064 (local address 127.0.97.1:44679)
0926 23:31:47.873573 (+ 8833us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:47.873580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:47.873605 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:31:47.873685 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:47.873691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:47.873890 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:47.874073 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:31:47.874745 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:47.876220 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:47.878550 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:47.882261 (+ 3711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:47.882409 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:47.882677 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:47.882927 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 23:31:47.883118 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8662,"thread_start_us":8482,"threads_started":1}
I20260926 23:31:47.884044 11033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cce06dff25f5491091bc84490b86a47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:31:47.884827 11167 tablet_service.cc:1467] Processing CreateTablet for tablet cce06dff25f5491091bc84490b86a47d (DEFAULT_TABLE table=test-table [id=a1ec14f86b9c449cb7aa8d4f74eb89e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:31:47.886359 11167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cce06dff25f5491091bc84490b86a47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:31:47.887557 10899 tablet_service.cc:1467] Processing CreateTablet for tablet cce06dff25f5491091bc84490b86a47d (DEFAULT_TABLE table=test-table [id=a1ec14f86b9c449cb7aa8d4f74eb89e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:31:47.888954 10899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cce06dff25f5491091bc84490b86a47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:31:47.903239 11251 tablet_bootstrap.cc:492] T cce06dff25f5491091bc84490b86a47d P f80dd6f69aa843088dc5b7a0cdac5c48: Bootstrap starting.
I20260926 23:31:47.906515 11253 tablet_bootstrap.cc:492] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a: Bootstrap starting.
I20260926 23:31:47.907764 11251 tablet_bootstrap.cc:654] T cce06dff25f5491091bc84490b86a47d P f80dd6f69aa843088dc5b7a0cdac5c48: Neither blocks nor log segments found. Creating new log.
I20260926 23:31:47.908978 11252 tablet_bootstrap.cc:492] T cce06dff25f5491091bc84490b86a47d P 26dd3ef6a8914933923af9bccb3caf02: Bootstrap starting.
I20260926 23:31:47.910184 11251 log.cc:824] T cce06dff25f5491091bc84490b86a47d P f80dd6f69aa843088dc5b7a0cdac5c48: Log is configured to *not* fsync() on all Append() calls
I20260926 23:31:47.912652 11253 tablet_bootstrap.cc:654] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a: Neither blocks nor log segments found. Creating new log.
I20260926 23:31:47.912887 11252 tablet_bootstrap.cc:654] T cce06dff25f5491091bc84490b86a47d P 26dd3ef6a8914933923af9bccb3caf02: Neither blocks nor log segments found. Creating new log.
I20260926 23:31:47.914165 11252 log.cc:824] T cce06dff25f5491091bc84490b86a47d P 26dd3ef6a8914933923af9bccb3caf02: Log is configured to *not* fsync() on all Append() calls
I20260926 23:31:47.914232 11253 log.cc:824] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a: Log is configured to *not* fsync() on all Append() calls
I20260926 23:31:47.914848 11251 tablet_bootstrap.cc:492] T cce06dff25f5491091bc84490b86a47d P f80dd6f69aa843088dc5b7a0cdac5c48: No bootstrap required, opened a new log
I20260926 23:31:47.915277 11251 ts_tablet_manager.cc:1397] T cce06dff25f5491091bc84490b86a47d P f80dd6f69aa843088dc5b7a0cdac5c48: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.002s
I20260926 23:31:47.917929 11253 tablet_bootstrap.cc:492] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a: No bootstrap required, opened a new log
I20260926 23:31:47.917951 11252 tablet_bootstrap.cc:492] T cce06dff25f5491091bc84490b86a47d P 26dd3ef6a8914933923af9bccb3caf02: No bootstrap required, opened a new log
I20260926 23:31:47.918275 11253 ts_tablet_manager.cc:1397] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260926 23:31:47.918283 11252 ts_tablet_manager.cc:1397] T cce06dff25f5491091bc84490b86a47d P 26dd3ef6a8914933923af9bccb3caf02: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260926 23:31:47.926668 11253 raft_consensus.cc:348] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f80dd6f69aa843088dc5b7a0cdac5c48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44679 } } peers { permanent_uuid: "26dd3ef6a8914933923af9bccb3caf02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43849 } } peers { permanent_uuid: "a80321ad8ed14f42980851d9390e8e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38003 } }
I20260926 23:31:47.926717 11251 raft_consensus.cc:348] T cce06dff25f5491091bc84490b86a47d P f80dd6f69aa843088dc5b7a0cdac5c48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f80dd6f69aa843088dc5b7a0cdac5c48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44679 } } peers { permanent_uuid: "26dd3ef6a8914933923af9bccb3caf02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43849 } } peers { permanent_uuid: "a80321ad8ed14f42980851d9390e8e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38003 } }
I20260926 23:31:47.927229 11253 raft_consensus.cc:374] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:31:47.927228 11251 raft_consensus.cc:374] T cce06dff25f5491091bc84490b86a47d P f80dd6f69aa843088dc5b7a0cdac5c48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:31:47.927328 11253 raft_consensus.cc:729] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80321ad8ed14f42980851d9390e8e9a, State: Initialized, Role: FOLLOWER
I20260926 23:31:47.927328 11251 raft_consensus.cc:729] T cce06dff25f5491091bc84490b86a47d P f80dd6f69aa843088dc5b7a0cdac5c48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f80dd6f69aa843088dc5b7a0cdac5c48, State: Initialized, Role: FOLLOWER
I20260926 23:31:47.928054 11253 consensus_queue.cc:260] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f80dd6f69aa843088dc5b7a0cdac5c48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44679 } } peers { permanent_uuid: "26dd3ef6a8914933923af9bccb3caf02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43849 } } peers { permanent_uuid: "a80321ad8ed14f42980851d9390e8e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38003 } }
I20260926 23:31:47.928066 11251 consensus_queue.cc:260] T cce06dff25f5491091bc84490b86a47d P f80dd6f69aa843088dc5b7a0cdac5c48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f80dd6f69aa843088dc5b7a0cdac5c48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44679 } } peers { permanent_uuid: "26dd3ef6a8914933923af9bccb3caf02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43849 } } peers { permanent_uuid: "a80321ad8ed14f42980851d9390e8e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38003 } }
I20260926 23:31:47.929369 11098 heartbeater.cc:503] Master 127.0.97.62:42271 was elected leader, sending a full tablet report...
I20260926 23:31:47.929284 11252 raft_consensus.cc:348] T cce06dff25f5491091bc84490b86a47d P 26dd3ef6a8914933923af9bccb3caf02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f80dd6f69aa843088dc5b7a0cdac5c48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44679 } } peers { permanent_uuid: "26dd3ef6a8914933923af9bccb3caf02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43849 } } peers { permanent_uuid: "a80321ad8ed14f42980851d9390e8e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38003 } }
I20260926 23:31:47.929822 11252 raft_consensus.cc:374] T cce06dff25f5491091bc84490b86a47d P 26dd3ef6a8914933923af9bccb3caf02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:31:47.929919 11252 raft_consensus.cc:729] T cce06dff25f5491091bc84490b86a47d P 26dd3ef6a8914933923af9bccb3caf02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26dd3ef6a8914933923af9bccb3caf02, State: Initialized, Role: FOLLOWER
I20260926 23:31:47.930506 11253 ts_tablet_manager.cc:1428] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260926 23:31:47.930611 11252 consensus_queue.cc:260] T cce06dff25f5491091bc84490b86a47d P 26dd3ef6a8914933923af9bccb3caf02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f80dd6f69aa843088dc5b7a0cdac5c48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44679 } } peers { permanent_uuid: "26dd3ef6a8914933923af9bccb3caf02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43849 } } peers { permanent_uuid: "a80321ad8ed14f42980851d9390e8e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38003 } }
I20260926 23:31:47.933898 11251 ts_tablet_manager.cc:1428] T cce06dff25f5491091bc84490b86a47d P f80dd6f69aa843088dc5b7a0cdac5c48: Time spent starting tablet: real 0.018s user 0.010s sys 0.005s
I20260926 23:31:47.935351 11232 heartbeater.cc:503] Master 127.0.97.62:42271 was elected leader, sending a full tablet report...
I20260926 23:31:47.936142 11252 ts_tablet_manager.cc:1428] T cce06dff25f5491091bc84490b86a47d P 26dd3ef6a8914933923af9bccb3caf02: Time spent starting tablet: real 0.018s user 0.018s sys 0.000s
I20260926 23:31:48.000602 11257 raft_consensus.cc:482] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:31:48.000980 11257 raft_consensus.cc:504] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f80dd6f69aa843088dc5b7a0cdac5c48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44679 } } peers { permanent_uuid: "26dd3ef6a8914933923af9bccb3caf02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43849 } } peers { permanent_uuid: "a80321ad8ed14f42980851d9390e8e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38003 } }
I20260926 23:31:48.003269 11257 leader_election.cc:288] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f80dd6f69aa843088dc5b7a0cdac5c48 (127.0.97.1:44679), 26dd3ef6a8914933923af9bccb3caf02 (127.0.97.3:43849)
I20260926 23:31:48.010938 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:48.003951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35399 (local address 127.0.97.1:44679)
0926 23:31:48.004253 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:48.004261 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:48.004628 (+ 367us) server_negotiation.cc:407] Connection header received
0926 23:31:48.005047 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:48.005055 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:48.005145 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:48.005217 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:31:48.006093 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:48.006923 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:48.008965 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:48.009855 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:48.009997 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:48.010122 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:48.010483 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 23:31:48.010634 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 23:31:48.010975 11249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:48.004958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56335 (local address 127.0.97.3:43849)
0926 23:31:48.005166 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:48.005173 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:48.005449 (+ 276us) server_negotiation.cc:407] Connection header received
0926 23:31:48.005587 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:48.005593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:48.005677 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:48.005785 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:31:48.006422 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:48.007258 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:48.009130 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:48.009989 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:48.010112 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:48.010234 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:48.010542 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 23:31:48.010679 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 23:31:48.011128 11260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:48.003679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44679 (local address 127.0.97.2:35399)
0926 23:31:48.004454 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:48.004481 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:48.004761 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:48.005426 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:48.005431 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:48.005502 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:31:48.005939 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:48.005954 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:48.007053 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:48.007059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:48.008774 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:48.008797 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:48.010116 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:48.010122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:48.010309 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:48.010340 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 23:31:48.010430 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 23:31:48.010596 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":25,"spinlock_wait_cycles":45440,"thread_start_us":404,"threads_started":1}
I20260926 23:31:48.011581 11261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:48.004617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43849 (local address 127.0.97.2:56335)
0926 23:31:48.005306 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0926 23:31:48.005333 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 23:31:48.005472 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:31:48.005835 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:31:48.005839 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:31:48.005866 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:31:48.006222 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:48.006230 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:48.007412 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:48.007416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:48.008956 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:31:48.008969 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:31:48.010214 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:31:48.010221 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:31:48.010384 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:48.010407 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 23:31:48.010485 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 23:31:48.010618 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":19,"thread_start_us":359,"threads_started":1}
I20260926 23:31:48.012408 10919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cce06dff25f5491091bc84490b86a47d" candidate_uuid: "a80321ad8ed14f42980851d9390e8e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f80dd6f69aa843088dc5b7a0cdac5c48" is_pre_election: true
I20260926 23:31:48.012624 11187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cce06dff25f5491091bc84490b86a47d" candidate_uuid: "a80321ad8ed14f42980851d9390e8e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26dd3ef6a8914933923af9bccb3caf02" is_pre_election: true
I20260926 23:31:48.013067 10919 raft_consensus.cc:2454] T cce06dff25f5491091bc84490b86a47d P f80dd6f69aa843088dc5b7a0cdac5c48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80321ad8ed14f42980851d9390e8e9a in term 0.
I20260926 23:31:48.013230 11187 raft_consensus.cc:2454] T cce06dff25f5491091bc84490b86a47d P 26dd3ef6a8914933923af9bccb3caf02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80321ad8ed14f42980851d9390e8e9a in term 0.
I20260926 23:31:48.013806 10986 leader_election.cc:302] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80321ad8ed14f42980851d9390e8e9a, f80dd6f69aa843088dc5b7a0cdac5c48; no voters:
I20260926 23:31:48.014259 11257 raft_consensus.cc:2781] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:31:48.014448 11257 raft_consensus.cc:482] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:31:48.014557 11257 raft_consensus.cc:3037] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:31:48.017095 11257 raft_consensus.cc:504] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f80dd6f69aa843088dc5b7a0cdac5c48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44679 } } peers { permanent_uuid: "26dd3ef6a8914933923af9bccb3caf02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43849 } } peers { permanent_uuid: "a80321ad8ed14f42980851d9390e8e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38003 } }
I20260926 23:31:48.017818 11257 leader_election.cc:288] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a [CANDIDATE]: Term 1 election: Requested vote from peers f80dd6f69aa843088dc5b7a0cdac5c48 (127.0.97.1:44679), 26dd3ef6a8914933923af9bccb3caf02 (127.0.97.3:43849)
I20260926 23:31:48.018452 10919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cce06dff25f5491091bc84490b86a47d" candidate_uuid: "a80321ad8ed14f42980851d9390e8e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f80dd6f69aa843088dc5b7a0cdac5c48"
I20260926 23:31:48.018532 11187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cce06dff25f5491091bc84490b86a47d" candidate_uuid: "a80321ad8ed14f42980851d9390e8e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26dd3ef6a8914933923af9bccb3caf02"
I20260926 23:31:48.018708 10919 raft_consensus.cc:3037] T cce06dff25f5491091bc84490b86a47d P f80dd6f69aa843088dc5b7a0cdac5c48 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:31:48.018846 11187 raft_consensus.cc:3037] T cce06dff25f5491091bc84490b86a47d P 26dd3ef6a8914933923af9bccb3caf02 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:31:48.021375 10919 raft_consensus.cc:2454] T cce06dff25f5491091bc84490b86a47d P f80dd6f69aa843088dc5b7a0cdac5c48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80321ad8ed14f42980851d9390e8e9a in term 1.
I20260926 23:31:48.021377 11187 raft_consensus.cc:2454] T cce06dff25f5491091bc84490b86a47d P 26dd3ef6a8914933923af9bccb3caf02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80321ad8ed14f42980851d9390e8e9a in term 1.
I20260926 23:31:48.021927 10986 leader_election.cc:302] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26dd3ef6a8914933923af9bccb3caf02, a80321ad8ed14f42980851d9390e8e9a; no voters:
I20260926 23:31:48.022300 11257 raft_consensus.cc:2781] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:31:48.022852 11257 raft_consensus.cc:686] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a [term 1 LEADER]: Becoming Leader. State: Replica: a80321ad8ed14f42980851d9390e8e9a, State: Running, Role: LEADER
I20260926 23:31:48.023522 11257 consensus_queue.cc:237] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f80dd6f69aa843088dc5b7a0cdac5c48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44679 } } peers { permanent_uuid: "26dd3ef6a8914933923af9bccb3caf02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43849 } } peers { permanent_uuid: "a80321ad8ed14f42980851d9390e8e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38003 } }
I20260926 23:31:48.029368 10774 catalog_manager.cc:5495] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a reported cstate change: term changed from 0 to 1, leader changed from <none> to a80321ad8ed14f42980851d9390e8e9a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "a80321ad8ed14f42980851d9390e8e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f80dd6f69aa843088dc5b7a0cdac5c48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26dd3ef6a8914933923af9bccb3caf02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80321ad8ed14f42980851d9390e8e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38003 } health_report { overall_health: HEALTHY } } }
W20260926 23:31:48.033188 11233 tablet.cc:2369] T cce06dff25f5491091bc84490b86a47d P 26dd3ef6a8914933923af9bccb3caf02: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:31:48.063594 10965 tablet.cc:2369] T cce06dff25f5491091bc84490b86a47d P f80dd6f69aa843088dc5b7a0cdac5c48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:31:48.080401 11248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:48.073460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38226 (local address 127.0.97.2:38003)
0926 23:31:48.073776 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:48.073782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:48.073803 (+ 21us) server_negotiation.cc:407] Connection header received
0926 23:31:48.073921 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:48.073925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:48.074026 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:48.074189 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:31:48.074632 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:48.075537 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:48.077608 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:48.078054 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:48.078168 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:48.078784 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:31:48.079549 (+ 765us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:31:48.079698 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:48.079967 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 23:31:48.080067 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 23:31:48.091759 10919 raft_consensus.cc:1261] T cce06dff25f5491091bc84490b86a47d P f80dd6f69aa843088dc5b7a0cdac5c48 [term 1 FOLLOWER]: Refusing update from remote peer a80321ad8ed14f42980851d9390e8e9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:31:48.091759 11187 raft_consensus.cc:1261] T cce06dff25f5491091bc84490b86a47d P 26dd3ef6a8914933923af9bccb3caf02 [term 1 FOLLOWER]: Refusing update from remote peer a80321ad8ed14f42980851d9390e8e9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:31:48.092990 11262 consensus_queue.cc:1035] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f80dd6f69aa843088dc5b7a0cdac5c48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:31:48.093461 11257 consensus_queue.cc:1035] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26dd3ef6a8914933923af9bccb3caf02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:31:48.130330 10773 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:31:48.130817 10773 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:31:48.136039 10773 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a8d39262e9644fd8ceb583a3cb0b0c9: Sending DeleteTablet for 3 replicas of tablet cce06dff25f5491091bc84490b86a47d
I20260926 23:31:48.137827 11167 tablet_service.cc:1514] Processing DeleteTablet for tablet cce06dff25f5491091bc84490b86a47d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:31:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53598
I20260926 23:31:48.137828 10899 tablet_service.cc:1514] Processing DeleteTablet for tablet cce06dff25f5491091bc84490b86a47d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:31:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60064
I20260926 23:31:48.138290 11033 tablet_service.cc:1514] Processing DeleteTablet for tablet cce06dff25f5491091bc84490b86a47d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:31:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38210
I20260926 23:31:48.143770 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 10836
I20260926 23:31:48.148422 11277 tablet_replica.cc:330] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a: stopping tablet replica
I20260926 23:31:48.149348 11277 raft_consensus.cc:2229] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:31:48.150660 11277 raft_consensus.cc:2258] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:31:48.152971 11276 tablet_replica.cc:330] T cce06dff25f5491091bc84490b86a47d P 26dd3ef6a8914933923af9bccb3caf02: stopping tablet replica
I20260926 23:31:48.153729 11276 raft_consensus.cc:2229] T cce06dff25f5491091bc84490b86a47d P 26dd3ef6a8914933923af9bccb3caf02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:31:48.154551 11276 raft_consensus.cc:2258] T cce06dff25f5491091bc84490b86a47d P 26dd3ef6a8914933923af9bccb3caf02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:31:48.157083 11276 ts_tablet_manager.cc:1905] T cce06dff25f5491091bc84490b86a47d P 26dd3ef6a8914933923af9bccb3caf02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:31:48.157094 11277 ts_tablet_manager.cc:1905] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:31:48.163093 11275 tablet_replica.cc:330] T cce06dff25f5491091bc84490b86a47d P f80dd6f69aa843088dc5b7a0cdac5c48: stopping tablet replica
I20260926 23:31:48.166002 11276 ts_tablet_manager.cc:1918] T cce06dff25f5491091bc84490b86a47d P 26dd3ef6a8914933923af9bccb3caf02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:31:48.166405 11276 log.cc:1196] T cce06dff25f5491091bc84490b86a47d P 26dd3ef6a8914933923af9bccb3caf02: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-2/wal/wals/cce06dff25f5491091bc84490b86a47d
I20260926 23:31:48.166429 11275 raft_consensus.cc:2229] T cce06dff25f5491091bc84490b86a47d P f80dd6f69aa843088dc5b7a0cdac5c48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:31:48.167274 11276 ts_tablet_manager.cc:1939] T cce06dff25f5491091bc84490b86a47d P 26dd3ef6a8914933923af9bccb3caf02: Deleting consensus metadata
I20260926 23:31:48.167480 11275 raft_consensus.cc:2258] T cce06dff25f5491091bc84490b86a47d P f80dd6f69aa843088dc5b7a0cdac5c48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:31:48.170106 10759 catalog_manager.cc:4841] TS 26dd3ef6a8914933923af9bccb3caf02 (127.0.97.3:43849): tablet cce06dff25f5491091bc84490b86a47d (table test-table [id=a1ec14f86b9c449cb7aa8d4f74eb89e6]) successfully deleted
I20260926 23:31:48.170591 11277 ts_tablet_manager.cc:1918] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:31:48.170902 11277 log.cc:1196] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-1/wal/wals/cce06dff25f5491091bc84490b86a47d
I20260926 23:31:48.171552 11277 ts_tablet_manager.cc:1939] T cce06dff25f5491091bc84490b86a47d P a80321ad8ed14f42980851d9390e8e9a: Deleting consensus metadata
Sep 26 23:31:48 dist-test-slave-gwfx krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:31:48.173462 11275 ts_tablet_manager.cc:1905] T cce06dff25f5491091bc84490b86a47d P f80dd6f69aa843088dc5b7a0cdac5c48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:31:48.173686 10760 catalog_manager.cc:4841] TS a80321ad8ed14f42980851d9390e8e9a (127.0.97.2:38003): tablet cce06dff25f5491091bc84490b86a47d (table test-table [id=a1ec14f86b9c449cb7aa8d4f74eb89e6]) successfully deleted
I20260926 23:31:48.181792 11275 ts_tablet_manager.cc:1918] T cce06dff25f5491091bc84490b86a47d P f80dd6f69aa843088dc5b7a0cdac5c48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:31:48.182071 11275 log.cc:1196] T cce06dff25f5491091bc84490b86a47d P f80dd6f69aa843088dc5b7a0cdac5c48: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790465400694755-388-0/minicluster-data/ts-0/wal/wals/cce06dff25f5491091bc84490b86a47d
I20260926 23:31:48.182296 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:48.145535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60072 (local address 127.0.97.1:44679)
0926 23:31:48.145798 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:48.145807 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:48.151982 (+ 6175us) server_negotiation.cc:407] Connection header received
0926 23:31:48.152244 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:48.152252 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:48.152419 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:48.152541 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:48.158991 (+ 6450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:48.160301 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:48.164113 (+ 3812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:48.164898 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:48.165057 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:48.173400 (+ 8343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:48.173414 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:48.173434 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:48.173496 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:48.177881 (+ 4385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:48.179118 (+ 1237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:48.179131 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:48.179135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:48.179349 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:48.179818 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:48.179824 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:48.179827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:48.180402 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:48.180742 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:48.181407 (+ 665us) server_negotiation.cc:299] Negotiation successful
0926 23:31:48.181665 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 23:31:48.182675 11275 ts_tablet_manager.cc:1939] T cce06dff25f5491091bc84490b86a47d P f80dd6f69aa843088dc5b7a0cdac5c48: Deleting consensus metadata
I20260926 23:31:48.184214 10959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:31:48.185043 10759 catalog_manager.cc:4841] TS f80dd6f69aa843088dc5b7a0cdac5c48 (127.0.97.1:44679): tablet cce06dff25f5491091bc84490b86a47d (table test-table [id=a1ec14f86b9c449cb7aa8d4f74eb89e6]) successfully deleted
I20260926 23:31:48.283083 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 10836
I20260926 23:31:48.301124 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 10970
Sep 26 23:31:48 dist-test-slave-gwfx krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:31:48.319624 11248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:48.301996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38232 (local address 127.0.97.2:38003)
0926 23:31:48.302218 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:48.302234 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:48.302258 (+ 24us) server_negotiation.cc:407] Connection header received
0926 23:31:48.302389 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:48.302395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:48.302481 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:48.302589 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:48.303988 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:48.304964 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:48.306363 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:48.306947 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:48.307092 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:48.311688 (+ 4596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:48.311711 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:48.311738 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:48.311796 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:48.315809 (+ 4013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:48.316916 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:48.316928 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:48.316931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:48.317162 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:48.317687 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:48.317692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:48.317695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:48.318183 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:48.318622 (+ 439us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:48.318898 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 23:31:48.319203 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 23:31:48.320457 11093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:31:48.447952 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 10970
I20260926 23:31:48.466470 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 11104
Sep 26 23:31:48 dist-test-slave-gwfx krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:31:48.483731 11249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:31:48.467474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53604 (local address 127.0.97.3:43849)
0926 23:31:48.467729 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 23:31:48.467736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:31:48.467759 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:31:48.467931 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:31:48.467936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:31:48.468024 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:31:48.468116 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:31:48.469618 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:48.470440 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:48.471925 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:31:48.472354 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:31:48.472477 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:31:48.477301 (+ 4824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:31:48.477323 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:31:48.477345 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:31:48.477393 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:31:48.480466 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:48.481355 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:48.481365 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:48.481368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:48.481544 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:31:48.481934 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:31:48.481938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:31:48.481940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:31:48.482383 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:31:48.482744 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0926 23:31:48.483068 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 23:31:48.483347 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 23:31:48.484701 11227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:31:48.588837 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 11104
I20260926 23:31:48.606841 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 10743
I20260926 23:31:48.608278 10805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:31:48.729861 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 10743
2026-09-26T23:31:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4809 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:31:48 dist-test-slave-gwfx krb5kdc[11285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:31:48 dist-test-slave-gwfx krb5kdc[11285](info): set up 2 sockets
Sep 26 23:31:48 dist-test-slave-gwfx krb5kdc[11285](info): commencing operation
krb5kdc: starting...
W20260926 23:31:49.779804 388 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.003s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:31:49 dist-test-slave-gwfx krb5kdc[11285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:31:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:31:49Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260926 23:31:50.183576 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasklIDZWX/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/postgres -l logfile start
2026-09-26 23:31:53.085 UTC [11315] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 23:31:53.085 UTC [11315] LOG: listening on IPv4 address "127.0.97.20", port 35777
2026-09-26 23:31:53.089 UTC [11315] LOG: listening on Unix socket "/tmp/.s.PGSQL.35777"
2026-09-26 23:31:53.114 UTC [11318] LOG: database system was shut down at 2026-09-26 23:31:51 UTC
2026-09-26 23:31:53.119 UTC [11315] LOG: database system is ready to accept connections
I20260926 23:31:54.024753 388 mini_postgres.cc:96] Postgres bound to 35777
2026-09-26 23:31:54.042 UTC [11326] FATAL: database "slave" does not exist
127.0.97.20:35777 - accepting connections
I20260926 23:31:54.043505 388 mini_ranger.cc:161] Starting Ranger...
I20260926 23:31:54.073288 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 23:31:54.686408 388 mini_ranger.cc:87] Created ranger Postgres database
I20260926 23:31:54.686568 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ranger-admin
2026-09-26 23:31:54,991 [I] DB FLAVOR :POSTGRES
2026-09-26 23:31:54,992 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 23:31:54,992 [I] Checking connection..
2026-09-26 23:31:54,992 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 23:31:55,289 [I] Checking connection passed.
2026-09-26 23:31:55,289 [I] --------- Verifying version history table ---------
2026-09-26 23:31:55,289 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 23:31:55,595 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 23:31:55,596 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 23:31:55,879 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 23:31:55,879 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 23:31:55,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35777/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 23:31:56.180 UTC [11390] WARNING: there is no transaction in progress
2026-09-26 23:31:56,188 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 23:31:56,188 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 23:31:56,495 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 23:31:56,496 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 23:31:56,496 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 23:31:56,759 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 23:31:57,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gwfx.c.gcp-upstream.internal','N') ;"
2026-09-26 23:31:57,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 23:31:57,617 [I] Table x_portal_user does not exist in database ranger
2026-09-26 23:31:57,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 23:31:57,886 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 23:31:57,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 23:31:58,157 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 23:31:58,435 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 23:31:58,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35777/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 23:31:59.624 UTC [11498] WARNING: there is no transaction in progress
2026-09-26 23:31:59.644 UTC [11498] WARNING: there is no transaction in progress
2026-09-26 23:31:59.666 UTC [11498] WARNING: there is no transaction in progress
2026-09-26 23:31:59.688 UTC [11498] WARNING: there is no transaction in progress
2026-09-26 23:31:59.707 UTC [11498] WARNING: there is no transaction in progress
2026-09-26 23:31:59.728 UTC [11498] WARNING: there is no transaction in progress
2026-09-26 23:31:59.809 UTC [11498] WARNING: there is no transaction in progress
2026-09-26 23:31:59.819 UTC [11498] WARNING: there is no transaction in progress
2026-09-26 23:31:59.838 UTC [11498] WARNING: there is no transaction in progress
2026-09-26 23:31:59.852 UTC [11498] WARNING: there is no transaction in progress
2026-09-26 23:31:59.866 UTC [11498] WARNING: there is no transaction in progress
2026-09-26 23:31:59.882 UTC [11498] WARNING: there is no transaction in progress
2026-09-26 23:31:59.895 UTC [11498] WARNING: there is no transaction in progress
2026-09-26 23:31:59.903 UTC [11498] WARNING: there is no transaction in progress
2026-09-26 23:31:59.914 UTC [11498] WARNING: there is no transaction in progress
2026-09-26 23:31:59.923 UTC [11498] WARNING: there is no transaction in progress
2026-09-26 23:32:00,807 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 23:32:00,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 23:32:01,095 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 23:32:01,096 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 23:32:01,390 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 23:32:01,390 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 23:32:01,677 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 23:32:01,677 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 23:32:01,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 23:32:02,252 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gwfx.c.gcp-upstream.internal';"
2026-09-26 23:32:02,534 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 23:32:02,535 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 23:32:02,840 [I] DB_PATCHES have already been applied
I20260926 23:32:02.848122 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ranger-admin:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasklIDZWX/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 23:32:02.848243 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260926 23:32:02.867918 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:41099
Sep 26, 2026 11:32:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 11:32:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 11:32:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 11:32:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 11:32:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 11:32:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 11:32:03 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 23:32:05.069870 388 mini_ranger.cc:160] Time spent starting Ranger: real 11.026s user 0.001s sys 0.066s
Sep 26, 2026 11:32:28 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 11:32:28 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 11:32:28 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 11:32:28 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 11:32:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 11:32:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 11:32:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 11:32:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 11:32:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 11:32:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 11:32:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 11:32:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 11:32:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 11:32:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 11:32:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 11:32:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 11:32:28 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 11:32:29 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 23:32:30.043051 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 23:32:30 dist-test-slave-gwfx krb5kdc[11285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465550, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 23:32:30.245715 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 23:32:30.276863 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 23:32:31.275256 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 23:32:31.275457 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ranger-kms
I20260926 23:32:31.275638 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:35777
2026-09-26 23:32:31,728 [I] DB FLAVOR :POSTGRES
2026-09-26 23:32:31,729 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 23:32:31,729 [I] Checking connection
2026-09-26 23:32:31,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35777/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 23:32:31,999 [I] connection success
2026-09-26 23:32:32,000 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 23:32:32,000 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 23:32:32,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35777/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 23:32:32,268 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 23:32:32,269 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 23:32:32,269 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 23:32:32,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35777/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 23:32:32,567 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 23:32:33.138731 388 mini_ranger_kms.cc:310] Created kms service
I20260926 23:32:33.391851 388 mini_ranger_kms.cc:326] Created kudu user
I20260926 23:32:33.442036 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 23:32:33 dist-test-slave-gwfx krb5kdc[11285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465550, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 23:32:33.877568 388 mini_ranger_kms.cc:384] Added ranger policy
I20260926 23:32:33.877929 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ranger-kms:/tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 23:32:33.877992 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260926 23:32:33.910890 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11680
I20260926 23:32:33.911173 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:48301
Sep 26, 2026 11:32:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 11:32:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 11:32:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 11:32:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 11:32:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 11:32:35 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 23:32:36.029263 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.784s user 0.129s sys 0.363s
I20260926 23:32:36.029546 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 23:32:36.029774 388 mini_ranger_kms.cc:401] 127.0.97.20:48301/kms/v1/keys
Sep 26 23:32:36 dist-test-slave-gwfx krb5kdc[11285](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790465556, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 23:32:36 dist-test-slave-gwfx krb5kdc[11285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465556, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 23:32:37 dist-test-slave-gwfx krb5kdc[11285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465556, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 23:32:39 dist-test-slave-gwfx krb5kdc[11285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:32:39.826660 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41035
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41035
--ranger_config_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:48301/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:32:40.266453 11730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:32:40.266902 11730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:32:40.267052 11730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:32:40.275979 11730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:32:40.276136 11730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:32:40.276217 11730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:32:40.276292 11730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:32:40.276371 11730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:32:40.288211 11730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:48301/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41035
--ranger_config_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:32:40.290534 11730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:32:40.292747 11730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:32:40.309434 11735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:32:40.309578 11736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:32:40.310491 11730 server_base.cc:1034] running on GCE node
W20260926 23:32:40.311002 11738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:32:40.312011 11730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:32:40.313787 11730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:32:40.315165 11730 hybrid_clock.cc:648] HybridClock initialized: now 1790465560315093 us; error 58 us; skew 500 ppm
I20260926 23:32:40.315626 11730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:32:40 dist-test-slave-gwfx krb5kdc[11285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:32:40.325500 11730 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:32:40.327607 11730 webserver.cc:460] Webserver started at http://127.0.97.62:37565/ using document root <none> and password file <none>
I20260926 23:32:40.329000 11730 fs_manager.cc:359] Metadata directory not provided
I20260926 23:32:40.329164 11730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:32:40.329559 11730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 23:32:40 dist-test-slave-gwfx krb5kdc[11285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 23:32:40.393175 11730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "88d8ec2d4d154c9486b10db65dad994e"
format_stamp: "Formatted at 2026-09-26 23:32:40 on dist-test-slave-gwfx"
server_key: "71268fb840f47581f80c7ea7992c8ae1"
server_key_iv: "dcae7d96b78fd322ff528efca5a9fc97"
server_key_version: "kuduclusterkey@0"
I20260926 23:32:40.394063 11730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "88d8ec2d4d154c9486b10db65dad994e"
format_stamp: "Formatted at 2026-09-26 23:32:40 on dist-test-slave-gwfx"
server_key: "71268fb840f47581f80c7ea7992c8ae1"
server_key_iv: "dcae7d96b78fd322ff528efca5a9fc97"
server_key_version: "kuduclusterkey@0"
I20260926 23:32:40.400370 11730 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260926 23:32:40.420737 11745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:32:40.422878 11730 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:32:40.423113 11730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "88d8ec2d4d154c9486b10db65dad994e"
format_stamp: "Formatted at 2026-09-26 23:32:40 on dist-test-slave-gwfx"
server_key: "71268fb840f47581f80c7ea7992c8ae1"
server_key_iv: "dcae7d96b78fd322ff528efca5a9fc97"
server_key_version: "kuduclusterkey@0"
I20260926 23:32:40.423367 11730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:32:40.450285 11730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:32:40.455518 11730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:32:40.455888 11730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:32:40.478570 11730 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41035
I20260926 23:32:40.478612 11796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41035 every 8 connection(s)
I20260926 23:32:40.480400 11730 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:32:40.483669 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 11730
I20260926 23:32:40.483917 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:32:40.484694 388 external_mini_cluster.cc:1428] Setting key 5b0ca5926ade5fabd226548db306a0cb
I20260926 23:32:40.485494 11797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 23:32:40 dist-test-slave-gwfx krb5kdc[11285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:32:40.503567 11797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88d8ec2d4d154c9486b10db65dad994e: Bootstrap starting.
I20260926 23:32:40.507766 11797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88d8ec2d4d154c9486b10db65dad994e: Neither blocks nor log segments found. Creating new log.
I20260926 23:32:40.509181 11797 log.cc:824] T 00000000000000000000000000000000 P 88d8ec2d4d154c9486b10db65dad994e: Log is configured to *not* fsync() on all Append() calls
I20260926 23:32:40.512809 11797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88d8ec2d4d154c9486b10db65dad994e: No bootstrap required, opened a new log
I20260926 23:32:40.514595 11800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:32:40.487326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35968 (local address 127.0.97.62:41035)
0926 23:32:40.488309 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0926 23:32:40.488327 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 23:32:40.488383 (+ 56us) server_negotiation.cc:407] Connection header received
0926 23:32:40.490648 (+ 2265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:32:40.490730 (+ 82us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:32:40.491337 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:32:40.491860 (+ 523us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:32:40.493635 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:32:40.496441 (+ 2806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:32:40.498170 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:32:40.499426 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:32:40.499687 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:32:40.504861 (+ 5174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:32:40.504875 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:32:40.504897 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:32:40.504961 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:32:40.508806 (+ 3845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:32:40.509800 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:32:40.509811 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:32:40.509821 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:32:40.510068 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:32:40.510465 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:32:40.510470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:32:40.510472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:32:40.511982 (+ 1510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:32:40.512338 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0926 23:32:40.512807 (+ 469us) server_negotiation.cc:299] Negotiation successful
0926 23:32:40.513312 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":446,"threads_started":1}
I20260926 23:32:40.521492 11797 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88d8ec2d4d154c9486b10db65dad994e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d8ec2d4d154c9486b10db65dad994e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41035 } }
I20260926 23:32:40.521914 11797 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88d8ec2d4d154c9486b10db65dad994e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:32:40.522006 11797 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88d8ec2d4d154c9486b10db65dad994e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88d8ec2d4d154c9486b10db65dad994e, State: Initialized, Role: FOLLOWER
I20260926 23:32:40.522672 11797 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88d8ec2d4d154c9486b10db65dad994e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d8ec2d4d154c9486b10db65dad994e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41035 } }
I20260926 23:32:40.522984 11797 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88d8ec2d4d154c9486b10db65dad994e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:32:40.523114 11797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88d8ec2d4d154c9486b10db65dad994e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:32:40.523295 11797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88d8ec2d4d154c9486b10db65dad994e [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:32:40.525967 11797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88d8ec2d4d154c9486b10db65dad994e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d8ec2d4d154c9486b10db65dad994e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41035 } }
I20260926 23:32:40.526562 11797 leader_election.cc:302] T 00000000000000000000000000000000 P 88d8ec2d4d154c9486b10db65dad994e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88d8ec2d4d154c9486b10db65dad994e; no voters:
I20260926 23:32:40.527347 11797 leader_election.cc:288] T 00000000000000000000000000000000 P 88d8ec2d4d154c9486b10db65dad994e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:32:40.527580 11802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88d8ec2d4d154c9486b10db65dad994e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:32:40.528982 11802 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88d8ec2d4d154c9486b10db65dad994e [term 1 LEADER]: Becoming Leader. State: Replica: 88d8ec2d4d154c9486b10db65dad994e, State: Running, Role: LEADER
I20260926 23:32:40.529695 11802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88d8ec2d4d154c9486b10db65dad994e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d8ec2d4d154c9486b10db65dad994e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41035 } }
I20260926 23:32:40.530516 11797 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88d8ec2d4d154c9486b10db65dad994e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:32:40.532902 11803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88d8ec2d4d154c9486b10db65dad994e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88d8ec2d4d154c9486b10db65dad994e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d8ec2d4d154c9486b10db65dad994e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41035 } } }
I20260926 23:32:40.533424 11803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88d8ec2d4d154c9486b10db65dad994e [sys.catalog]: This master's current role is: LEADER
I20260926 23:32:40.532902 11804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88d8ec2d4d154c9486b10db65dad994e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88d8ec2d4d154c9486b10db65dad994e. Latest consensus state: current_term: 1 leader_uuid: "88d8ec2d4d154c9486b10db65dad994e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d8ec2d4d154c9486b10db65dad994e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41035 } } }
I20260926 23:32:40.534982 11804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88d8ec2d4d154c9486b10db65dad994e [sys.catalog]: This master's current role is: LEADER
I20260926 23:32:40.537582 11807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:32:40.545840 11807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:32:40.546852 11797 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 23:32:40.565443 11807 catalog_manager.cc:1347] Generated new cluster ID: 6fe3eddd44664275ac2c4b1665768ad3
I20260926 23:32:40.565672 11807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:32:40.593171 11807 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:32:40.595538 11807 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:32:40.606788 11807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88d8ec2d4d154c9486b10db65dad994e: Generated new TSK 0
I20260926 23:32:40.608217 11807 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:32:41.443370 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:48301/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41035
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:32:41 dist-test-slave-gwfx krb5kdc[11285](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790465561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 23:32:42.009199 11842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:32:42.010012 11842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:32:42.010197 11842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:32:42.023221 11842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:32:42.023433 11842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:32:42.023656 11842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:32:42.042135 11842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:48301/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:32:42.044497 11842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:32:42.046818 11842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:32:42.067575 11853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:32:42.067584 11852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:32:42.069352 11855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:32:42.069669 11842 server_base.cc:1034] running on GCE node
I20260926 23:32:42.070528 11842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:32:42.072160 11842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:32:42.073473 11842 hybrid_clock.cc:648] HybridClock initialized: now 1790465562073338 us; error 57 us; skew 500 ppm
I20260926 23:32:42.073894 11842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:32:42 dist-test-slave-gwfx krb5kdc[11285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:32:42.078001 11842 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:32:42.080237 11842 webserver.cc:460] Webserver started at http://127.0.97.1:33087/ using document root <none> and password file <none>
I20260926 23:32:42.081420 11842 fs_manager.cc:359] Metadata directory not provided
I20260926 23:32:42.081727 11842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:32:42.082139 11842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 23:32:42 dist-test-slave-gwfx krb5kdc[11285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 23:32:42.106411 11842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-0/data/instance:
uuid: "7fd0f8cf09b145c594af033b7f5b7c91"
format_stamp: "Formatted at 2026-09-26 23:32:42 on dist-test-slave-gwfx"
server_key: "418e21a7a616d3762d072c303a60c77b"
server_key_iv: "5605a12163be246ea60b07665bae2982"
server_key_version: "kuduclusterkey@0"
I20260926 23:32:42.107903 11842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7fd0f8cf09b145c594af033b7f5b7c91"
format_stamp: "Formatted at 2026-09-26 23:32:42 on dist-test-slave-gwfx"
server_key: "418e21a7a616d3762d072c303a60c77b"
server_key_iv: "5605a12163be246ea60b07665bae2982"
server_key_version: "kuduclusterkey@0"
I20260926 23:32:42.117005 11842 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s
I20260926 23:32:42.153522 11862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:32:42.156453 11842 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.003s sys 0.001s
I20260926 23:32:42.156774 11842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "7fd0f8cf09b145c594af033b7f5b7c91"
format_stamp: "Formatted at 2026-09-26 23:32:42 on dist-test-slave-gwfx"
server_key: "418e21a7a616d3762d072c303a60c77b"
server_key_iv: "5605a12163be246ea60b07665bae2982"
server_key_version: "kuduclusterkey@0"
I20260926 23:32:42.157147 11842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:32:42.190128 11842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:32:42.196480 11842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:32:42.196875 11842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:32:42.198292 11842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:32:42.200528 11842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:32:42.200631 11842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:32:42.200786 11842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:32:42.200845 11842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:32:42.284545 11842 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41495
I20260926 23:32:42.284545 11974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41495 every 8 connection(s)
I20260926 23:32:42.286250 11842 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:32:42.297072 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 11842
I20260926 23:32:42.297343 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:32:42.298242 388 external_mini_cluster.cc:1428] Setting key 6ba40b8d8c3cf95c072d061a104aed51
Sep 26 23:32:42 dist-test-slave-gwfx krb5kdc[11285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:32:42.337293 11977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:32:42.293978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44111 (local address 127.0.97.62:41035)
0926 23:32:42.303211 (+ 9233us) server_negotiation.cc:206] Beginning negotiation
0926 23:32:42.303220 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:32:42.304681 (+ 1461us) server_negotiation.cc:407] Connection header received
0926 23:32:42.307107 (+ 2426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:32:42.307115 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:32:42.307224 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:32:42.307323 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:32:42.310077 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:32:42.311375 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:32:42.315228 (+ 3853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:32:42.315950 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:32:42.316087 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:32:42.326173 (+ 10086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:32:42.326184 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:32:42.326191 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:32:42.326257 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:32:42.330078 (+ 3821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:32:42.333671 (+ 3593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:32:42.333679 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:32:42.333684 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:32:42.333922 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:32:42.334660 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:32:42.334665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:32:42.334669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:32:42.335121 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:32:42.335271 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 23:32:42.336524 (+ 1253us) server_negotiation.cc:299] Negotiation successful
0926 23:32:42.336842 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9036,"thread_start_us":24730,"threads_started":1}
I20260926 23:32:42.340178 11978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:32:42.294810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41035 (local address 127.0.97.1:44111)
0926 23:32:42.303945 (+ 9135us) negotiation.cc:107] Waiting for socket to connect
0926 23:32:42.304017 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0926 23:32:42.306662 (+ 2645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:32:42.308051 (+ 1389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:32:42.308079 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:32:42.308806 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:32:42.309831 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:32:42.309862 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:32:42.311607 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:32:42.311613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:32:42.314938 (+ 3325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:32:42.314967 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:32:42.316557 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:32:42.316565 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:32:42.316971 (+ 406us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:32:42.319147 (+ 2176us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:32:42.319189 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:32:42.323165 (+ 3976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:32:42.330389 (+ 7224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:32:42.330401 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:32:42.330429 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:32:42.331232 (+ 803us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:32:42.334144 (+ 2912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:32:42.334149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:32:42.334154 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:32:42.334514 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:32:42.335385 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:32:42.335397 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:32:42.336005 (+ 608us) client_negotiation.cc:769] Sending connection context
0926 23:32:42.337580 (+ 1575us) client_negotiation.cc:240] Negotiation successful
0926 23:32:42.338348 (+ 768us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8755,"thread_start_us":8458,"threads_started":1}
I20260926 23:32:42.342864 11975 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41035
I20260926 23:32:42.343297 11975 heartbeater.cc:464] Registering TS with master...
I20260926 23:32:42.344408 11975 heartbeater.cc:511] Master 127.0.97.62:41035 requested a full tablet report, sending...
I20260926 23:32:42.347707 11762 ts_manager.cc:194] Registered new tserver with Master: 7fd0f8cf09b145c594af033b7f5b7c91 (127.0.97.1:41495)
I20260926 23:32:42.349931 11762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44111
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Sep 26 23:32:42 dist-test-slave-gwfx krb5kdc[11285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:32:42.400280 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:48301/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41035
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:32:42 dist-test-slave-gwfx krb5kdc[11285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260926 23:32:42.992231 11981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:32:42.992852 11981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:32:42.993069 11981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:32:43.006626 11981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:32:43.006860 11981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:32:43.007083 11981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:32:43.024941 11981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:48301/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:32:43.027297 11981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:32:43.029713 11981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:32:43.053577 11991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:32:43.053615 11997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:32:43.053584 11993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:32:43.056717 11981 server_base.cc:1034] running on GCE node
I20260926 23:32:43.057570 11981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:32:43.058799 11981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:32:43.060048 11981 hybrid_clock.cc:648] HybridClock initialized: now 1790465563059992 us; error 44 us; skew 500 ppm
I20260926 23:32:43.061172 11981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:32:43 dist-test-slave-gwfx krb5kdc[11285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:32:43.064970 11981 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:32:43.066821 11981 webserver.cc:460] Webserver started at http://127.0.97.2:33551/ using document root <none> and password file <none>
I20260926 23:32:43.067727 11981 fs_manager.cc:359] Metadata directory not provided
I20260926 23:32:43.067855 11981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:32:43.068198 11981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 23:32:43 dist-test-slave-gwfx krb5kdc[11285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 23:32:43.087725 11981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-1/data/instance:
uuid: "a7331bb1c40a496aa96d0f9cd2a650d5"
format_stamp: "Formatted at 2026-09-26 23:32:43 on dist-test-slave-gwfx"
server_key: "42299439d5756c97e22d58bf4b67320e"
server_key_iv: "66ccaa400a9e267aea07fb799c015d46"
server_key_version: "kuduclusterkey@0"
I20260926 23:32:43.088917 11981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a7331bb1c40a496aa96d0f9cd2a650d5"
format_stamp: "Formatted at 2026-09-26 23:32:43 on dist-test-slave-gwfx"
server_key: "42299439d5756c97e22d58bf4b67320e"
server_key_iv: "66ccaa400a9e267aea07fb799c015d46"
server_key_version: "kuduclusterkey@0"
I20260926 23:32:43.095803 11981 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 23:32:43.114473 12005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:32:43.116575 11981 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 23:32:43.116791 11981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "a7331bb1c40a496aa96d0f9cd2a650d5"
format_stamp: "Formatted at 2026-09-26 23:32:43 on dist-test-slave-gwfx"
server_key: "42299439d5756c97e22d58bf4b67320e"
server_key_iv: "66ccaa400a9e267aea07fb799c015d46"
server_key_version: "kuduclusterkey@0"
I20260926 23:32:43.117053 11981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:32:43.140211 11981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:32:43.145272 11981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:32:43.145607 11981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:32:43.146822 11981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:32:43.148617 11981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:32:43.148693 11981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:32:43.148815 11981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:32:43.148864 11981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:32:43.191013 11981 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35841
I20260926 23:32:43.191031 12117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35841 every 8 connection(s)
I20260926 23:32:43.192517 11981 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:32:43.198518 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 11981
I20260926 23:32:43.198750 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:32:43.199299 388 external_mini_cluster.cc:1428] Setting key 6803be13ff5f46bdc8077295614d1824
Sep 26 23:32:43 dist-test-slave-gwfx krb5kdc[11285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:32:43.217718 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:32:43.195572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45561 (local address 127.0.97.62:41035)
0926 23:32:43.196125 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0926 23:32:43.196133 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:32:43.197327 (+ 1194us) server_negotiation.cc:407] Connection header received
0926 23:32:43.200311 (+ 2984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:32:43.200318 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:32:43.200433 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:32:43.200558 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:32:43.202558 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:32:43.203402 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:32:43.205280 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:32:43.205729 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:32:43.205842 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:32:43.210117 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:32:43.210123 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:32:43.210127 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:32:43.210168 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:32:43.213008 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:32:43.214343 (+ 1335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:32:43.214350 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:32:43.214354 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:32:43.214545 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:32:43.215248 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:32:43.215254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:32:43.215257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:32:43.215665 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:32:43.215822 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 23:32:43.216990 (+ 1168us) server_negotiation.cc:299] Negotiation successful
0926 23:32:43.217291 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":257,"threads_started":1}
I20260926 23:32:43.219449 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:32:43.196188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41035 (local address 127.0.97.2:45561)
0926 23:32:43.197110 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0926 23:32:43.197183 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0926 23:32:43.199575 (+ 2392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:32:43.200901 (+ 1326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:32:43.200914 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:32:43.201592 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:32:43.202363 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:32:43.202383 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:32:43.203555 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:32:43.203559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:32:43.205130 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:32:43.205145 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:32:43.205972 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:32:43.205979 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:32:43.206245 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:32:43.207034 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:32:43.207061 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:32:43.209796 (+ 2735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:32:43.213242 (+ 3446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:32:43.213256 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:32:43.213285 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:32:43.214149 (+ 864us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:32:43.214722 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:32:43.214728 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:32:43.214731 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:32:43.215106 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:32:43.215822 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:32:43.215833 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:32:43.216445 (+ 612us) client_negotiation.cc:769] Sending connection context
0926 23:32:43.216840 (+ 395us) client_negotiation.cc:240] Negotiation successful
0926 23:32:43.217557 (+ 717us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":430,"threads_started":1}
I20260926 23:32:43.222620 12118 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41035
I20260926 23:32:43.223034 12118 heartbeater.cc:464] Registering TS with master...
I20260926 23:32:43.224133 12118 heartbeater.cc:511] Master 127.0.97.62:41035 requested a full tablet report, sending...
I20260926 23:32:43.226496 11762 ts_manager.cc:194] Registered new tserver with Master: a7331bb1c40a496aa96d0f9cd2a650d5 (127.0.97.2:35841)
I20260926 23:32:43.227814 11762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45561
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:32:43.268488 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:48301/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41035
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:32:43.355399 11975 heartbeater.cc:503] Master 127.0.97.62:41035 was elected leader, sending a full tablet report...
W20260926 23:32:43.700043 12124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:32:43.700423 12124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:32:43.700554 12124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:32:43.708536 12124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:32:43.708704 12124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:32:43.708868 12124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:32:43.720813 12124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:48301/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:32:43.722527 12124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:32:43.724134 12124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:32:43.739555 12130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:32:43.740578 12129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:32:43.741097 12132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:32:43.741333 12124 server_base.cc:1034] running on GCE node
I20260926 23:32:43.742045 12124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:32:43.743204 12124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:32:43.744433 12124 hybrid_clock.cc:648] HybridClock initialized: now 1790465563744357 us; error 61 us; skew 500 ppm
I20260926 23:32:43.744812 12124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:32:43 dist-test-slave-gwfx krb5kdc[11285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:32:43.748622 12124 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:32:43.750423 12124 webserver.cc:460] Webserver started at http://127.0.97.3:42233/ using document root <none> and password file <none>
I20260926 23:32:43.751330 12124 fs_manager.cc:359] Metadata directory not provided
I20260926 23:32:43.751452 12124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:32:43.751814 12124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 23:32:43 dist-test-slave-gwfx krb5kdc[11285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 23:32:43.772135 12124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-2/data/instance:
uuid: "cd5dfbfc00f140739fbdbe6ac2730112"
format_stamp: "Formatted at 2026-09-26 23:32:43 on dist-test-slave-gwfx"
server_key: "339b5dfdc12dbee6603e707b55587dec"
server_key_iv: "1916fdd4c648ab5ffe47a266eb0b4910"
server_key_version: "kuduclusterkey@0"
I20260926 23:32:43.773190 12124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-2/wal/instance:
uuid: "cd5dfbfc00f140739fbdbe6ac2730112"
format_stamp: "Formatted at 2026-09-26 23:32:43 on dist-test-slave-gwfx"
server_key: "339b5dfdc12dbee6603e707b55587dec"
server_key_iv: "1916fdd4c648ab5ffe47a266eb0b4910"
server_key_version: "kuduclusterkey@0"
I20260926 23:32:43.779948 12124 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260926 23:32:43.801119 12139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:32:43.803339 12124 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 23:32:43.803571 12124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "cd5dfbfc00f140739fbdbe6ac2730112"
format_stamp: "Formatted at 2026-09-26 23:32:43 on dist-test-slave-gwfx"
server_key: "339b5dfdc12dbee6603e707b55587dec"
server_key_iv: "1916fdd4c648ab5ffe47a266eb0b4910"
server_key_version: "kuduclusterkey@0"
I20260926 23:32:43.803887 12124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:32:43.831950 12124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:32:43.837287 12124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:32:43.838307 12124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:32:43.839540 12124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:32:43.841259 12124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:32:43.841347 12124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:32:43.841459 12124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:32:43.841514 12124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:32:43.882135 12124 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35151
I20260926 23:32:43.882206 12251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35151 every 8 connection(s)
I20260926 23:32:43.883751 12124 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:32:43.884382 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 12124
I20260926 23:32:43.884593 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:32:43.885180 388 external_mini_cluster.cc:1428] Setting key 19b177d7eb0794cc4a145a517f7257c6
Sep 26 23:32:43 dist-test-slave-gwfx krb5kdc[11285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:32:43.908318 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:32:43.887791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54945 (local address 127.0.97.62:41035)
0926 23:32:43.888387 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0926 23:32:43.888392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:32:43.889480 (+ 1088us) server_negotiation.cc:407] Connection header received
0926 23:32:43.892493 (+ 3013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:32:43.892498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:32:43.892612 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:32:43.892727 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:32:43.894716 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:32:43.895575 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:32:43.897481 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:32:43.897967 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:32:43.898077 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:32:43.902160 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:32:43.902167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:32:43.902171 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:32:43.902215 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:32:43.904942 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:32:43.905793 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:32:43.905798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:32:43.905801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:32:43.905962 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:32:43.906401 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:32:43.906406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:32:43.906408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:32:43.906850 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:32:43.906988 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 23:32:43.907763 (+ 775us) server_negotiation.cc:299] Negotiation successful
0926 23:32:43.908019 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":269,"threads_started":1}
I20260926 23:32:43.909502 12255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:32:43.888271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41035 (local address 127.0.97.3:54945)
0926 23:32:43.889299 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0926 23:32:43.889360 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 23:32:43.892108 (+ 2748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:32:43.893058 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:32:43.893071 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:32:43.893650 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:32:43.894436 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:32:43.894459 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:32:43.895783 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:32:43.895788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:32:43.897304 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:32:43.897320 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:32:43.898246 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:32:43.898253 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:32:43.898524 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:32:43.899388 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:32:43.899424 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:32:43.901897 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:32:43.905107 (+ 3210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:32:43.905116 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:32:43.905137 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:32:43.905657 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:32:43.906055 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:32:43.906058 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:32:43.906060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:32:43.906280 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:32:43.906960 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:32:43.906970 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:32:43.907373 (+ 403us) client_negotiation.cc:769] Sending connection context
0926 23:32:43.907666 (+ 293us) client_negotiation.cc:240] Negotiation successful
0926 23:32:43.908139 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":420,"threads_started":1}
I20260926 23:32:43.911258 12252 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41035
I20260926 23:32:43.911661 12252 heartbeater.cc:464] Registering TS with master...
I20260926 23:32:43.912482 12252 heartbeater.cc:511] Master 127.0.97.62:41035 requested a full tablet report, sending...
I20260926 23:32:43.914248 11762 ts_manager.cc:194] Registered new tserver with Master: cd5dfbfc00f140739fbdbe6ac2730112 (127.0.97.3:35151)
I20260926 23:32:43.915500 11762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54945
I20260926 23:32:43.915582 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:32:43.935696 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:32:43.920507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35982 (local address 127.0.97.62:41035)
0926 23:32:43.920726 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 23:32:43.920735 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:32:43.921147 (+ 412us) server_negotiation.cc:407] Connection header received
0926 23:32:43.921303 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:32:43.921307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:32:43.921389 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:32:43.921503 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:32:43.923838 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:32:43.924676 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:32:43.926146 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:32:43.926611 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:32:43.926781 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:32:43.929351 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:32:43.929357 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:32:43.929360 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:32:43.929405 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:32:43.932400 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:32:43.933312 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:32:43.933318 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:32:43.933321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:32:43.933522 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:32:43.934067 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:32:43.934072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:32:43.934074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:32:43.934498 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:32:43.934803 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 23:32:43.935155 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 23:32:43.935360 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 23:32:43.940707 11762 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:32:43.949380 11762 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:32:44.000761 12266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:32:43.965101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35841 (local address 127.0.0.1:33852)
0926 23:32:43.990583 (+ 25482us) negotiation.cc:107] Waiting for socket to connect
0926 23:32:43.990604 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:32:43.990759 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:32:43.992176 (+ 1417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:32:43.992187 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:32:43.992223 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:32:43.992775 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:32:43.992797 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:32:43.994880 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:32:43.994886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:32:43.997490 (+ 2604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:32:43.997506 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:32:43.999430 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:32:43.999437 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:32:43.999658 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:32:43.999713 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 23:32:43.999870 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 23:32:44.000165 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25340,"mutex_wait_us":41,"thread_start_us":26841,"threads_started":1}
I20260926 23:32:44.001478 12268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:32:43.965239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33182 (local address 127.0.97.3:35151)
0926 23:32:43.992787 (+ 27548us) server_negotiation.cc:206] Beginning negotiation
0926 23:32:43.992796 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:32:43.992838 (+ 42us) server_negotiation.cc:407] Connection header received
0926 23:32:43.992947 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:32:43.992954 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:32:43.993225 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:32:43.993431 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:32:43.994359 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:32:43.995660 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:32:43.998048 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:32:43.999289 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:32:43.999430 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:32:43.999673 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 23:32:44.000903 (+ 1230us) server_negotiation.cc:299] Negotiation successful
0926 23:32:44.001117 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27345,"thread_start_us":27126,"threads_started":1}
I20260926 23:32:44.000782 12267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:32:43.964763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35151 (local address 127.0.0.1:33182)
0926 23:32:43.989469 (+ 24706us) negotiation.cc:107] Waiting for socket to connect
0926 23:32:43.989550 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0926 23:32:43.989873 (+ 323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:32:43.993719 (+ 3846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:32:43.993725 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:32:43.993750 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:32:43.994148 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:32:43.994166 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:32:43.995837 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:32:43.995842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:32:43.997869 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:32:43.997883 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:32:43.999822 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:32:43.999829 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:32:44.000039 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:32:44.000071 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 23:32:44.000153 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 23:32:44.000281 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24302,"mutex_wait_us":2,"thread_start_us":28081,"threads_started":1}
I20260926 23:32:44.002111 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:32:43.966077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33852 (local address 127.0.97.2:35841)
0926 23:32:43.988281 (+ 22204us) server_negotiation.cc:206] Beginning negotiation
0926 23:32:43.988297 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 23:32:43.990924 (+ 2627us) server_negotiation.cc:407] Connection header received
0926 23:32:43.991088 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:32:43.991100 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:32:43.991510 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:32:43.991740 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:32:43.993071 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:32:43.994745 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:32:43.997684 (+ 2939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:32:43.999206 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:32:43.999346 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:32:43.999600 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0926 23:32:44.001528 (+ 1928us) server_negotiation.cc:299] Negotiation successful
0926 23:32:44.001734 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21896,"thread_start_us":22248,"threads_started":1}
I20260926 23:32:44.002591 12265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:32:43.965460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41495 (local address 127.0.0.1:58178)
0926 23:32:43.993138 (+ 27678us) negotiation.cc:107] Waiting for socket to connect
0926 23:32:43.993156 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:32:43.993274 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:32:43.993976 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:32:43.993981 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:32:43.994012 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:32:43.994398 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:32:43.994407 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:32:43.996446 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:32:43.996451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:32:43.998399 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:32:43.998413 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:32:44.001650 (+ 3237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:32:44.001656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:32:44.001952 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:32:44.001985 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 23:32:44.002074 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 23:32:44.002243 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27559,"thread_start_us":23441,"threads_started":1}
I20260926 23:32:44.002774 12270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:32:43.966503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58178 (local address 127.0.97.1:41495)
0926 23:32:43.991601 (+ 25098us) server_negotiation.cc:206] Beginning negotiation
0926 23:32:43.991613 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:32:43.993413 (+ 1800us) server_negotiation.cc:407] Connection header received
0926 23:32:43.993536 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:32:43.993543 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:32:43.993757 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:32:43.993968 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:32:43.994582 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:32:43.996256 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:32:43.998580 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:32:44.001097 (+ 2517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:32:44.001226 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:32:44.001495 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0926 23:32:44.002161 (+ 666us) server_negotiation.cc:299] Negotiation successful
0926 23:32:44.002371 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24853,"thread_start_us":25104,"threads_started":1}
I20260926 23:32:44.007107 11910 tablet_service.cc:1467] Processing CreateTablet for tablet 26069034e69849a1a100b52b97929906 (DEFAULT_TABLE table=test-table [id=74bbc73d63c745649a097f5a84ec9960]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:32:44.007019 12187 tablet_service.cc:1467] Processing CreateTablet for tablet 26069034e69849a1a100b52b97929906 (DEFAULT_TABLE table=test-table [id=74bbc73d63c745649a097f5a84ec9960]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:32:44.007019 12053 tablet_service.cc:1467] Processing CreateTablet for tablet 26069034e69849a1a100b52b97929906 (DEFAULT_TABLE table=test-table [id=74bbc73d63c745649a097f5a84ec9960]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:32:44.008773 11910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26069034e69849a1a100b52b97929906. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:32:44.008788 12053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26069034e69849a1a100b52b97929906. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:32:44.008791 12187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26069034e69849a1a100b52b97929906. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:32:44.023910 12271 tablet_bootstrap.cc:492] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91: Bootstrap starting.
I20260926 23:32:44.029474 12271 tablet_bootstrap.cc:654] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91: Neither blocks nor log segments found. Creating new log.
I20260926 23:32:44.030864 12271 log.cc:824] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91: Log is configured to *not* fsync() on all Append() calls
I20260926 23:32:44.031746 12273 tablet_bootstrap.cc:492] T 26069034e69849a1a100b52b97929906 P cd5dfbfc00f140739fbdbe6ac2730112: Bootstrap starting.
I20260926 23:32:44.031746 12272 tablet_bootstrap.cc:492] T 26069034e69849a1a100b52b97929906 P a7331bb1c40a496aa96d0f9cd2a650d5: Bootstrap starting.
I20260926 23:32:44.035583 12271 tablet_bootstrap.cc:492] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91: No bootstrap required, opened a new log
I20260926 23:32:44.035893 12271 ts_tablet_manager.cc:1397] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260926 23:32:44.036720 12273 tablet_bootstrap.cc:654] T 26069034e69849a1a100b52b97929906 P cd5dfbfc00f140739fbdbe6ac2730112: Neither blocks nor log segments found. Creating new log.
I20260926 23:32:44.036718 12272 tablet_bootstrap.cc:654] T 26069034e69849a1a100b52b97929906 P a7331bb1c40a496aa96d0f9cd2a650d5: Neither blocks nor log segments found. Creating new log.
I20260926 23:32:44.038130 12272 log.cc:824] T 26069034e69849a1a100b52b97929906 P a7331bb1c40a496aa96d0f9cd2a650d5: Log is configured to *not* fsync() on all Append() calls
I20260926 23:32:44.038130 12273 log.cc:824] T 26069034e69849a1a100b52b97929906 P cd5dfbfc00f140739fbdbe6ac2730112: Log is configured to *not* fsync() on all Append() calls
I20260926 23:32:44.042974 12273 tablet_bootstrap.cc:492] T 26069034e69849a1a100b52b97929906 P cd5dfbfc00f140739fbdbe6ac2730112: No bootstrap required, opened a new log
I20260926 23:32:44.043295 12273 ts_tablet_manager.cc:1397] T 26069034e69849a1a100b52b97929906 P cd5dfbfc00f140739fbdbe6ac2730112: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260926 23:32:44.046115 12272 tablet_bootstrap.cc:492] T 26069034e69849a1a100b52b97929906 P a7331bb1c40a496aa96d0f9cd2a650d5: No bootstrap required, opened a new log
I20260926 23:32:44.046422 12272 ts_tablet_manager.cc:1397] T 26069034e69849a1a100b52b97929906 P a7331bb1c40a496aa96d0f9cd2a650d5: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20260926 23:32:44.046327 12271 raft_consensus.cc:348] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd5dfbfc00f140739fbdbe6ac2730112" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35151 } } peers { permanent_uuid: "a7331bb1c40a496aa96d0f9cd2a650d5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35841 } } peers { permanent_uuid: "7fd0f8cf09b145c594af033b7f5b7c91" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41495 } }
I20260926 23:32:44.046814 12271 raft_consensus.cc:374] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:32:44.046944 12271 raft_consensus.cc:729] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd0f8cf09b145c594af033b7f5b7c91, State: Initialized, Role: FOLLOWER
I20260926 23:32:44.047608 12271 consensus_queue.cc:260] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd5dfbfc00f140739fbdbe6ac2730112" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35151 } } peers { permanent_uuid: "a7331bb1c40a496aa96d0f9cd2a650d5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35841 } } peers { permanent_uuid: "7fd0f8cf09b145c594af033b7f5b7c91" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41495 } }
I20260926 23:32:44.049593 12271 ts_tablet_manager.cc:1428] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260926 23:32:44.055284 12273 raft_consensus.cc:348] T 26069034e69849a1a100b52b97929906 P cd5dfbfc00f140739fbdbe6ac2730112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd5dfbfc00f140739fbdbe6ac2730112" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35151 } } peers { permanent_uuid: "a7331bb1c40a496aa96d0f9cd2a650d5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35841 } } peers { permanent_uuid: "7fd0f8cf09b145c594af033b7f5b7c91" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41495 } }
I20260926 23:32:44.055876 12273 raft_consensus.cc:374] T 26069034e69849a1a100b52b97929906 P cd5dfbfc00f140739fbdbe6ac2730112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:32:44.055989 12273 raft_consensus.cc:729] T 26069034e69849a1a100b52b97929906 P cd5dfbfc00f140739fbdbe6ac2730112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd5dfbfc00f140739fbdbe6ac2730112, State: Initialized, Role: FOLLOWER
I20260926 23:32:44.056762 12273 consensus_queue.cc:260] T 26069034e69849a1a100b52b97929906 P cd5dfbfc00f140739fbdbe6ac2730112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd5dfbfc00f140739fbdbe6ac2730112" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35151 } } peers { permanent_uuid: "a7331bb1c40a496aa96d0f9cd2a650d5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35841 } } peers { permanent_uuid: "7fd0f8cf09b145c594af033b7f5b7c91" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41495 } }
I20260926 23:32:44.057341 12272 raft_consensus.cc:348] T 26069034e69849a1a100b52b97929906 P a7331bb1c40a496aa96d0f9cd2a650d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd5dfbfc00f140739fbdbe6ac2730112" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35151 } } peers { permanent_uuid: "a7331bb1c40a496aa96d0f9cd2a650d5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35841 } } peers { permanent_uuid: "7fd0f8cf09b145c594af033b7f5b7c91" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41495 } }
I20260926 23:32:44.057816 12272 raft_consensus.cc:374] T 26069034e69849a1a100b52b97929906 P a7331bb1c40a496aa96d0f9cd2a650d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:32:44.057930 12272 raft_consensus.cc:729] T 26069034e69849a1a100b52b97929906 P a7331bb1c40a496aa96d0f9cd2a650d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7331bb1c40a496aa96d0f9cd2a650d5, State: Initialized, Role: FOLLOWER
I20260926 23:32:44.058014 12252 heartbeater.cc:503] Master 127.0.97.62:41035 was elected leader, sending a full tablet report...
I20260926 23:32:44.058564 12272 consensus_queue.cc:260] T 26069034e69849a1a100b52b97929906 P a7331bb1c40a496aa96d0f9cd2a650d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd5dfbfc00f140739fbdbe6ac2730112" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35151 } } peers { permanent_uuid: "a7331bb1c40a496aa96d0f9cd2a650d5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35841 } } peers { permanent_uuid: "7fd0f8cf09b145c594af033b7f5b7c91" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41495 } }
I20260926 23:32:44.059839 12273 ts_tablet_manager.cc:1428] T 26069034e69849a1a100b52b97929906 P cd5dfbfc00f140739fbdbe6ac2730112: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20260926 23:32:44.060338 12118 heartbeater.cc:503] Master 127.0.97.62:41035 was elected leader, sending a full tablet report...
I20260926 23:32:44.061103 12272 ts_tablet_manager.cc:1428] T 26069034e69849a1a100b52b97929906 P a7331bb1c40a496aa96d0f9cd2a650d5: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
W20260926 23:32:44.078644 11976 tablet.cc:2369] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:32:44.101986 12277 raft_consensus.cc:482] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:32:44.102361 12277 raft_consensus.cc:504] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd5dfbfc00f140739fbdbe6ac2730112" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35151 } } peers { permanent_uuid: "a7331bb1c40a496aa96d0f9cd2a650d5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35841 } } peers { permanent_uuid: "7fd0f8cf09b145c594af033b7f5b7c91" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41495 } }
I20260926 23:32:44.104543 12277 leader_election.cc:288] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd5dfbfc00f140739fbdbe6ac2730112 (127.0.97.3:35151), a7331bb1c40a496aa96d0f9cd2a650d5 (127.0.97.2:35841)
I20260926 23:32:44.113955 12268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:32:44.105203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50669 (local address 127.0.97.3:35151)
0926 23:32:44.105445 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 23:32:44.105452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:32:44.107145 (+ 1693us) server_negotiation.cc:407] Connection header received
0926 23:32:44.107249 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:32:44.107254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:32:44.107328 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:32:44.107427 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:32:44.108144 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:32:44.109004 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:32:44.111799 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:32:44.112720 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:32:44.112852 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:32:44.112952 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 23:32:44.113493 (+ 541us) server_negotiation.cc:299] Negotiation successful
0926 23:32:44.113658 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 23:32:44.114099 12281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:32:44.104887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35151 (local address 127.0.97.1:50669)
0926 23:32:44.106851 (+ 1964us) negotiation.cc:107] Waiting for socket to connect
0926 23:32:44.106867 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:32:44.107071 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:32:44.107451 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:32:44.107455 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:32:44.107488 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:32:44.107937 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:32:44.107950 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:32:44.109209 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:32:44.109215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:32:44.111574 (+ 2359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:32:44.111591 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:32:44.113013 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:32:44.113027 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:32:44.113256 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:32:44.113301 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 23:32:44.113447 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 23:32:44.113640 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1858,"mutex_wait_us":101,"spinlock_wait_cycles":45568,"thread_start_us":1445,"threads_started":1}
I20260926 23:32:44.114522 12280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:32:44.104863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35841 (local address 127.0.97.1:44133)
0926 23:32:44.106811 (+ 1948us) negotiation.cc:107] Waiting for socket to connect
0926 23:32:44.106841 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 23:32:44.107038 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:32:44.107450 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:32:44.107455 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:32:44.107503 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:32:44.108046 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:32:44.108061 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:32:44.109288 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:32:44.109295 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:32:44.111569 (+ 2274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:32:44.111590 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:32:44.113633 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:32:44.113640 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:32:44.113812 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:32:44.113846 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 23:32:44.113957 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 23:32:44.114114 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1762,"mutex_wait_us":38,"spinlock_wait_cycles":4864,"thread_start_us":1342,"threads_started":1}
I20260926 23:32:44.114555 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:32:44.105226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44133 (local address 127.0.97.2:35841)
0926 23:32:44.105445 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 23:32:44.105452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:32:44.106974 (+ 1522us) server_negotiation.cc:407] Connection header received
0926 23:32:44.107225 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:32:44.107229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:32:44.107313 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:32:44.107386 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:32:44.108231 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:32:44.109033 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:32:44.112201 (+ 3168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:32:44.113328 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:32:44.113495 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:32:44.113610 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 23:32:44.114001 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 23:32:44.114168 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 23:32:44.115823 12207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26069034e69849a1a100b52b97929906" candidate_uuid: "7fd0f8cf09b145c594af033b7f5b7c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd5dfbfc00f140739fbdbe6ac2730112" is_pre_election: true
I20260926 23:32:44.115823 12073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26069034e69849a1a100b52b97929906" candidate_uuid: "7fd0f8cf09b145c594af033b7f5b7c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7331bb1c40a496aa96d0f9cd2a650d5" is_pre_election: true
I20260926 23:32:44.116567 12207 raft_consensus.cc:2454] T 26069034e69849a1a100b52b97929906 P cd5dfbfc00f140739fbdbe6ac2730112 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fd0f8cf09b145c594af033b7f5b7c91 in term 0.
I20260926 23:32:44.116567 12073 raft_consensus.cc:2454] T 26069034e69849a1a100b52b97929906 P a7331bb1c40a496aa96d0f9cd2a650d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fd0f8cf09b145c594af033b7f5b7c91 in term 0.
I20260926 23:32:44.117434 11865 leader_election.cc:302] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fd0f8cf09b145c594af033b7f5b7c91, cd5dfbfc00f140739fbdbe6ac2730112; no voters:
I20260926 23:32:44.117897 12277 raft_consensus.cc:2781] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:32:44.118072 12277 raft_consensus.cc:482] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:32:44.118180 12277 raft_consensus.cc:3037] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:32:44.121141 12277 raft_consensus.cc:504] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd5dfbfc00f140739fbdbe6ac2730112" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35151 } } peers { permanent_uuid: "a7331bb1c40a496aa96d0f9cd2a650d5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35841 } } peers { permanent_uuid: "7fd0f8cf09b145c594af033b7f5b7c91" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41495 } }
I20260926 23:32:44.122588 12277 leader_election.cc:288] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91 [CANDIDATE]: Term 1 election: Requested vote from peers cd5dfbfc00f140739fbdbe6ac2730112 (127.0.97.3:35151), a7331bb1c40a496aa96d0f9cd2a650d5 (127.0.97.2:35841)
I20260926 23:32:44.123445 12073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26069034e69849a1a100b52b97929906" candidate_uuid: "7fd0f8cf09b145c594af033b7f5b7c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7331bb1c40a496aa96d0f9cd2a650d5"
I20260926 23:32:44.123445 12207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26069034e69849a1a100b52b97929906" candidate_uuid: "7fd0f8cf09b145c594af033b7f5b7c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd5dfbfc00f140739fbdbe6ac2730112"
I20260926 23:32:44.123721 12207 raft_consensus.cc:3037] T 26069034e69849a1a100b52b97929906 P cd5dfbfc00f140739fbdbe6ac2730112 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:32:44.123721 12073 raft_consensus.cc:3037] T 26069034e69849a1a100b52b97929906 P a7331bb1c40a496aa96d0f9cd2a650d5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:32:44.127282 12207 raft_consensus.cc:2454] T 26069034e69849a1a100b52b97929906 P cd5dfbfc00f140739fbdbe6ac2730112 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fd0f8cf09b145c594af033b7f5b7c91 in term 1.
I20260926 23:32:44.127282 12073 raft_consensus.cc:2454] T 26069034e69849a1a100b52b97929906 P a7331bb1c40a496aa96d0f9cd2a650d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fd0f8cf09b145c594af033b7f5b7c91 in term 1.
I20260926 23:32:44.127789 11866 leader_election.cc:302] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fd0f8cf09b145c594af033b7f5b7c91, a7331bb1c40a496aa96d0f9cd2a650d5; no voters:
I20260926 23:32:44.128178 12277 raft_consensus.cc:2781] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:32:44.128757 12277 raft_consensus.cc:686] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91 [term 1 LEADER]: Becoming Leader. State: Replica: 7fd0f8cf09b145c594af033b7f5b7c91, State: Running, Role: LEADER
I20260926 23:32:44.129422 12277 consensus_queue.cc:237] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd5dfbfc00f140739fbdbe6ac2730112" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35151 } } peers { permanent_uuid: "a7331bb1c40a496aa96d0f9cd2a650d5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35841 } } peers { permanent_uuid: "7fd0f8cf09b145c594af033b7f5b7c91" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41495 } }
I20260926 23:32:44.135051 11761 catalog_manager.cc:5495] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fd0f8cf09b145c594af033b7f5b7c91 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "7fd0f8cf09b145c594af033b7f5b7c91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd5dfbfc00f140739fbdbe6ac2730112" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7331bb1c40a496aa96d0f9cd2a650d5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fd0f8cf09b145c594af033b7f5b7c91" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41495 } health_report { overall_health: HEALTHY } } }
W20260926 23:32:44.137027 12253 tablet.cc:2369] T 26069034e69849a1a100b52b97929906 P cd5dfbfc00f140739fbdbe6ac2730112: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:32:44.166579 12270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:32:44.159589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58180 (local address 127.0.97.1:41495)
0926 23:32:44.159812 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 23:32:44.159818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:32:44.159837 (+ 19us) server_negotiation.cc:407] Connection header received
0926 23:32:44.159953 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:32:44.159957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:32:44.160050 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:32:44.160236 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:32:44.160613 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:32:44.161449 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:32:44.163972 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:32:44.164452 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:32:44.164625 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:32:44.165282 (+ 657us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:32:44.165978 (+ 696us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:32:44.166118 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 23:32:44.166218 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 23:32:44.166307 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 23:32:44.176770 12073 raft_consensus.cc:1261] T 26069034e69849a1a100b52b97929906 P a7331bb1c40a496aa96d0f9cd2a650d5 [term 1 FOLLOWER]: Refusing update from remote peer 7fd0f8cf09b145c594af033b7f5b7c91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:32:44.176770 12207 raft_consensus.cc:1261] T 26069034e69849a1a100b52b97929906 P cd5dfbfc00f140739fbdbe6ac2730112 [term 1 FOLLOWER]: Refusing update from remote peer 7fd0f8cf09b145c594af033b7f5b7c91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:32:44.177881 12282 consensus_queue.cc:1035] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7331bb1c40a496aa96d0f9cd2a650d5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:32:44.178241 12277 consensus_queue.cc:1035] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd5dfbfc00f140739fbdbe6ac2730112" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:32:44.192713 12286 mvcc.cc:205] Tried to move back new op lower bound from 7333746950856388608 to 7333746950679089152. Current Snapshot: MvccSnapshot[applied={T|T < 7333746950856388608}]
I20260926 23:32:44.195780 12285 mvcc.cc:205] Tried to move back new op lower bound from 7333746950856388608 to 7333746950679089152. Current Snapshot: MvccSnapshot[applied={T|T < 7333746950856388608}]
I20260926 23:32:44.224176 11762 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:32:44.224524 11762 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:32:44.229115 11762 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88d8ec2d4d154c9486b10db65dad994e: Sending DeleteTablet for 3 replicas of tablet 26069034e69849a1a100b52b97929906
I20260926 23:32:44.230611 12053 tablet_service.cc:1514] Processing DeleteTablet for tablet 26069034e69849a1a100b52b97929906 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:32:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33852
I20260926 23:32:44.230611 12187 tablet_service.cc:1514] Processing DeleteTablet for tablet 26069034e69849a1a100b52b97929906 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:32:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33182
I20260926 23:32:44.230839 11910 tablet_service.cc:1514] Processing DeleteTablet for tablet 26069034e69849a1a100b52b97929906 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:32:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58178
I20260926 23:32:44.236989 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 11842
I20260926 23:32:44.243399 12296 tablet_replica.cc:330] T 26069034e69849a1a100b52b97929906 P a7331bb1c40a496aa96d0f9cd2a650d5: stopping tablet replica
I20260926 23:32:44.243680 12295 tablet_replica.cc:330] T 26069034e69849a1a100b52b97929906 P cd5dfbfc00f140739fbdbe6ac2730112: stopping tablet replica
I20260926 23:32:44.244256 12296 raft_consensus.cc:2229] T 26069034e69849a1a100b52b97929906 P a7331bb1c40a496aa96d0f9cd2a650d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:32:44.244251 12295 raft_consensus.cc:2229] T 26069034e69849a1a100b52b97929906 P cd5dfbfc00f140739fbdbe6ac2730112 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:32:44.244933 12295 raft_consensus.cc:2258] T 26069034e69849a1a100b52b97929906 P cd5dfbfc00f140739fbdbe6ac2730112 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:32:44.244935 12296 raft_consensus.cc:2258] T 26069034e69849a1a100b52b97929906 P a7331bb1c40a496aa96d0f9cd2a650d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:32:44.246410 12297 tablet_replica.cc:330] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91: stopping tablet replica
I20260926 23:32:44.247053 12297 raft_consensus.cc:2229] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:32:44.247825 12295 ts_tablet_manager.cc:1905] T 26069034e69849a1a100b52b97929906 P cd5dfbfc00f140739fbdbe6ac2730112: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:32:44.248296 12297 raft_consensus.cc:2258] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:32:44.251122 12296 ts_tablet_manager.cc:1905] T 26069034e69849a1a100b52b97929906 P a7331bb1c40a496aa96d0f9cd2a650d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:32:44.251127 12297 ts_tablet_manager.cc:1905] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:32:44.256340 12295 ts_tablet_manager.cc:1918] T 26069034e69849a1a100b52b97929906 P cd5dfbfc00f140739fbdbe6ac2730112: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:32:44.256541 12295 log.cc:1196] T 26069034e69849a1a100b52b97929906 P cd5dfbfc00f140739fbdbe6ac2730112: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-2/wal/wals/26069034e69849a1a100b52b97929906
I20260926 23:32:44.257071 12295 ts_tablet_manager.cc:1939] T 26069034e69849a1a100b52b97929906 P cd5dfbfc00f140739fbdbe6ac2730112: Deleting consensus metadata
I20260926 23:32:44.259155 12296 ts_tablet_manager.cc:1918] T 26069034e69849a1a100b52b97929906 P a7331bb1c40a496aa96d0f9cd2a650d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:32:44.259265 12297 ts_tablet_manager.cc:1918] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:32:44.259374 12296 log.cc:1196] T 26069034e69849a1a100b52b97929906 P a7331bb1c40a496aa96d0f9cd2a650d5: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-1/wal/wals/26069034e69849a1a100b52b97929906
I20260926 23:32:44.259459 12297 log.cc:1196] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790465400694755-388-0/minicluster-data/ts-0/wal/wals/26069034e69849a1a100b52b97929906
I20260926 23:32:44.259454 11748 catalog_manager.cc:4841] TS cd5dfbfc00f140739fbdbe6ac2730112 (127.0.97.3:35151): tablet 26069034e69849a1a100b52b97929906 (table test-table [id=74bbc73d63c745649a097f5a84ec9960]) successfully deleted
I20260926 23:32:44.259980 12296 ts_tablet_manager.cc:1939] T 26069034e69849a1a100b52b97929906 P a7331bb1c40a496aa96d0f9cd2a650d5: Deleting consensus metadata
I20260926 23:32:44.260010 12297 ts_tablet_manager.cc:1939] T 26069034e69849a1a100b52b97929906 P 7fd0f8cf09b145c594af033b7f5b7c91: Deleting consensus metadata
I20260926 23:32:44.262120 11749 catalog_manager.cc:4841] TS a7331bb1c40a496aa96d0f9cd2a650d5 (127.0.97.2:35841): tablet 26069034e69849a1a100b52b97929906 (table test-table [id=74bbc73d63c745649a097f5a84ec9960]) successfully deleted
I20260926 23:32:44.262092 11747 catalog_manager.cc:4841] TS 7fd0f8cf09b145c594af033b7f5b7c91 (127.0.97.1:41495): tablet 26069034e69849a1a100b52b97929906 (table test-table [id=74bbc73d63c745649a097f5a84ec9960]) successfully deleted
Sep 26 23:32:44 dist-test-slave-gwfx krb5kdc[11285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:32:44.274513 12270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:32:44.239087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58182 (local address 127.0.97.1:41495)
0926 23:32:44.239342 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 23:32:44.239351 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:32:44.249073 (+ 9722us) server_negotiation.cc:407] Connection header received
0926 23:32:44.249282 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:32:44.249289 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:32:44.249413 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:32:44.249548 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:32:44.255850 (+ 6302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:32:44.257201 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:32:44.259873 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:32:44.260427 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:32:44.260529 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:32:44.266273 (+ 5744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:32:44.266297 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:32:44.266328 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:32:44.266396 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:32:44.270824 (+ 4428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:32:44.271810 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:32:44.271822 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:32:44.271827 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:32:44.272044 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:32:44.272495 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:32:44.272501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:32:44.272505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:32:44.273093 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:32:44.273482 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0926 23:32:44.273773 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 23:32:44.274078 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 23:32:44.275481 11970 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:32:44.384066 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 11842
I20260926 23:32:44.403007 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 11981
Sep 26 23:32:44 dist-test-slave-gwfx krb5kdc[11285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:32:44.421166 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:32:44.403981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33858 (local address 127.0.97.2:35841)
0926 23:32:44.404252 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 23:32:44.404258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:32:44.404276 (+ 18us) server_negotiation.cc:407] Connection header received
0926 23:32:44.404346 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:32:44.404350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:32:44.404430 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:32:44.404519 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:32:44.405853 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:32:44.406951 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:32:44.408443 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:32:44.408986 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:32:44.409113 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:32:44.414087 (+ 4974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:32:44.414119 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:32:44.414141 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:32:44.414187 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:32:44.417652 (+ 3465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:32:44.418540 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:32:44.418549 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:32:44.418552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:32:44.418769 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:32:44.419258 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:32:44.419262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:32:44.419264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:32:44.419745 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:32:44.419998 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 23:32:44.420482 (+ 484us) server_negotiation.cc:299] Negotiation successful
0926 23:32:44.420711 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260926 23:32:44.422051 12113 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:32:44.529786 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 11981
I20260926 23:32:44.545610 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 12124
Sep 26 23:32:44 dist-test-slave-gwfx krb5kdc[11285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:32:44.563030 12268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:32:44.546665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33198 (local address 127.0.97.3:35151)
0926 23:32:44.546866 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 23:32:44.546872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:32:44.546893 (+ 21us) server_negotiation.cc:407] Connection header received
0926 23:32:44.547002 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:32:44.547006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:32:44.547083 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:32:44.547171 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:32:44.548578 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:32:44.549381 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:32:44.550794 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:32:44.551272 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:32:44.551384 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:32:44.556288 (+ 4904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:32:44.556310 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:32:44.556339 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:32:44.556397 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:32:44.559607 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:32:44.560565 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:32:44.560579 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:32:44.560582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:32:44.560756 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:32:44.561214 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:32:44.561218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:32:44.561220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:32:44.561713 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:32:44.561986 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0926 23:32:44.562347 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 23:32:44.562553 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 23:32:44.563709 12247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:32:44.661048 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 12124
I20260926 23:32:44.676707 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 11730
I20260926 23:32:44.678177 11792 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:32:44.806262 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 11730
I20260926 23:32:44.826890 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 23:32:45.164230 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 23:32:45.164499 388 mini_ranger.cc:66] Stopping Ranger...
I20260926 23:32:45.511713 388 mini_ranger.cc:68] Stopped Ranger
2026-09-26 23:32:45.512 UTC [11315] LOG: received smart shutdown request
2026-09-26 23:32:45.516 UTC [11315] LOG: background worker "logical replication launcher" (PID 11324) exited with exit code 1
2026-09-26 23:32:45.516 UTC [11319] LOG: shutting down
2026-09-26 23:32:45.680 UTC [11315] LOG: database system is shut down
2026-09-26T23:32:45Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (57064 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:32:45 dist-test-slave-gwfx krb5kdc[12311](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:32:45 dist-test-slave-gwfx krb5kdc[12311](info): set up 2 sockets
Sep 26 23:32:45 dist-test-slave-gwfx krb5kdc[12311](info): commencing operation
krb5kdc: starting...
W20260926 23:32:46.859380 388 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.002s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:32:47 dist-test-slave-gwfx krb5kdc[12311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:32:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:32:47Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260926 23:32:47.223089 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasklIDZWX/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/postgres -l logfile start
2026-09-26 23:32:50.313 UTC [12341] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 23:32:50.314 UTC [12341] LOG: listening on IPv4 address "127.0.97.20", port 54473
2026-09-26 23:32:50.319 UTC [12341] LOG: listening on Unix socket "/tmp/.s.PGSQL.54473"
2026-09-26 23:32:50.349 UTC [12344] LOG: database system was shut down at 2026-09-26 23:32:48 UTC
2026-09-26 23:32:50.353 UTC [12341] LOG: database system is ready to accept connections
I20260926 23:32:51.242857 388 mini_postgres.cc:96] Postgres bound to 54473
2026-09-26 23:32:51.263 UTC [12352] FATAL: database "slave" does not exist
127.0.97.20:54473 - accepting connections
I20260926 23:32:51.264468 388 mini_ranger.cc:161] Starting Ranger...
I20260926 23:32:51.294960 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 23:32:51.938478 388 mini_ranger.cc:87] Created ranger Postgres database
I20260926 23:32:51.938611 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ranger-admin
2026-09-26 23:32:52,256 [I] DB FLAVOR :POSTGRES
2026-09-26 23:32:52,256 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 23:32:52,256 [I] Checking connection..
2026-09-26 23:32:52,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 23:32:52,545 [I] Checking connection passed.
2026-09-26 23:32:52,545 [I] --------- Verifying version history table ---------
2026-09-26 23:32:52,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 23:32:52,825 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 23:32:52,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 23:32:53,120 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 23:32:53,121 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 23:32:53,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 23:32:53.394 UTC [12416] WARNING: there is no transaction in progress
2026-09-26 23:32:53,402 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 23:32:53,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 23:32:53,674 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 23:32:53,674 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 23:32:53,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 23:32:53,973 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 23:32:54,254 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gwfx.c.gcp-upstream.internal','N') ;"
2026-09-26 23:32:54,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 23:32:54,832 [I] Table x_portal_user does not exist in database ranger
2026-09-26 23:32:54,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 23:32:55,125 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 23:32:55,126 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 23:32:55,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 23:32:55,679 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 23:32:55,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 23:32:56.898 UTC [12524] WARNING: there is no transaction in progress
2026-09-26 23:32:56.920 UTC [12524] WARNING: there is no transaction in progress
2026-09-26 23:32:56.941 UTC [12524] WARNING: there is no transaction in progress
2026-09-26 23:32:56.960 UTC [12524] WARNING: there is no transaction in progress
2026-09-26 23:32:56.985 UTC [12524] WARNING: there is no transaction in progress
2026-09-26 23:32:57.009 UTC [12524] WARNING: there is no transaction in progress
2026-09-26 23:32:57.091 UTC [12524] WARNING: there is no transaction in progress
2026-09-26 23:32:57.099 UTC [12524] WARNING: there is no transaction in progress
2026-09-26 23:32:57.121 UTC [12524] WARNING: there is no transaction in progress
2026-09-26 23:32:57.135 UTC [12524] WARNING: there is no transaction in progress
2026-09-26 23:32:57.149 UTC [12524] WARNING: there is no transaction in progress
2026-09-26 23:32:57.168 UTC [12524] WARNING: there is no transaction in progress
2026-09-26 23:32:57.181 UTC [12524] WARNING: there is no transaction in progress
2026-09-26 23:32:57.189 UTC [12524] WARNING: there is no transaction in progress
2026-09-26 23:32:57.198 UTC [12524] WARNING: there is no transaction in progress
2026-09-26 23:32:57.209 UTC [12524] WARNING: there is no transaction in progress
2026-09-26 23:32:58,060 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 23:32:58,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 23:32:58,340 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 23:32:58,341 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 23:32:58,623 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 23:32:58,624 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 23:32:58,924 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 23:32:58,924 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 23:32:59,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 23:32:59,480 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gwfx.c.gcp-upstream.internal';"
2026-09-26 23:32:59,744 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 23:32:59,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 23:33:00,046 [I] DB_PATCHES have already been applied
I20260926 23:33:00.053603 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ranger-admin:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasklIDZWX/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 23:33:00.053719 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260926 23:33:00.071349 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:34677
Sep 26, 2026 11:33:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 11:33:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 11:33:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 11:33:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 11:33:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 11:33:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 11:33:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 23:33:02.185678 388 mini_ranger.cc:160] Time spent starting Ranger: real 10.921s user 0.004s sys 0.059s
Sep 26, 2026 11:33:25 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 11:33:25 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 26, 2026 11:33:25 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 11:33:25 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 11:33:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 11:33:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 11:33:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 11:33:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 11:33:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 11:33:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 11:33:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 11:33:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 11:33:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 11:33:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 11:33:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 11:33:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 11:33:25 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 11:33:26 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 23:33:27.547499 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 23:33:27 dist-test-slave-gwfx krb5kdc[12311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465607, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 23:33:27.760454 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 23:33:27.790385 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 23:33:28.839363 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 23:33:28.839496 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ranger-kms
I20260926 23:33:28.839541 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:54473
2026-09-26 23:33:29,209 [I] DB FLAVOR :POSTGRES
2026-09-26 23:33:29,210 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 23:33:29,210 [I] Checking connection
2026-09-26 23:33:29,210 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54473/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 23:33:29,497 [I] connection success
2026-09-26 23:33:29,498 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 23:33:29,498 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 23:33:29,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54473/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 23:33:29,765 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 23:33:29,766 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 23:33:29,766 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 23:33:29,766 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54473/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 23:33:30,052 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 23:33:30.690603 388 mini_ranger_kms.cc:310] Created kms service
I20260926 23:33:30.936144 388 mini_ranger_kms.cc:326] Created kudu user
I20260926 23:33:30.988622 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 23:33:30 dist-test-slave-gwfx krb5kdc[12311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465607, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 23:33:31.459425 388 mini_ranger_kms.cc:384] Added ranger policy
I20260926 23:33:31.459810 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ranger-kms:/tmp/dist-test-tasklIDZWX/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasklIDZWX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 23:33:31.459896 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260926 23:33:31.481760 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12706
I20260926 23:33:31.481937 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:40423
Sep 26, 2026 11:33:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 11:33:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 11:33:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 11:33:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 11:33:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 11:33:33 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 23:33:33.634876 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.874s user 0.112s sys 0.289s
I20260926 23:33:33.635157 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 23:33:33.635325 388 mini_ranger_kms.cc:401] 127.0.97.20:40423/kms/v1/keys
Sep 26 23:33:33 dist-test-slave-gwfx krb5kdc[12311](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790465613, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 23:33:34 dist-test-slave-gwfx krb5kdc[12311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465613, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 23:33:34 dist-test-slave-gwfx krb5kdc[12311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465613, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 23:33:37 dist-test-slave-gwfx krb5kdc[12311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:33:37.496114 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37509
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37509
--ranger_config_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40423/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:33:37.893541 12756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:33:37.893929 12756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:33:37.894032 12756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:33:37.901935 12756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:33:37.902088 12756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:33:37.902137 12756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:33:37.902192 12756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:33:37.902242 12756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:33:37.913729 12756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:40423/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37509
--ranger_config_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:33:37.915455 12756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:33:37.917112 12756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:33:37.931185 12761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:33:37.931214 12762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:33:37.931279 12764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:33:37.931954 12756 server_base.cc:1034] running on GCE node
I20260926 23:33:37.933036 12756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:33:37.935097 12756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:33:37.936436 12756 hybrid_clock.cc:648] HybridClock initialized: now 1790465617936309 us; error 83 us; skew 500 ppm
I20260926 23:33:37.936908 12756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:33:37 dist-test-slave-gwfx krb5kdc[12311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:33:37.941005 12756 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:33:37.943192 12756 webserver.cc:460] Webserver started at http://127.0.97.62:34329/ using document root <none> and password file <none>
I20260926 23:33:37.944204 12756 fs_manager.cc:359] Metadata directory not provided
I20260926 23:33:37.944353 12756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:33:37.944751 12756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 23:33:37 dist-test-slave-gwfx krb5kdc[12311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 23:33:38.017714 12756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "ef765b2d56bc4b1a9a19dd5f57fb7951"
format_stamp: "Formatted at 2026-09-26 23:33:38 on dist-test-slave-gwfx"
server_key: "b034f29c4087cd911cc2c85218023c4e"
server_key_iv: "bd6eeb8b1db935b458592c3e742bf4c9"
server_key_version: "kuduclusterkey@0"
I20260926 23:33:38.018800 12756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "ef765b2d56bc4b1a9a19dd5f57fb7951"
format_stamp: "Formatted at 2026-09-26 23:33:38 on dist-test-slave-gwfx"
server_key: "b034f29c4087cd911cc2c85218023c4e"
server_key_iv: "bd6eeb8b1db935b458592c3e742bf4c9"
server_key_version: "kuduclusterkey@0"
I20260926 23:33:38.025624 12756 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260926 23:33:38.049453 12771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:33:38.051332 12756 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 23:33:38.051555 12756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "ef765b2d56bc4b1a9a19dd5f57fb7951"
format_stamp: "Formatted at 2026-09-26 23:33:38 on dist-test-slave-gwfx"
server_key: "b034f29c4087cd911cc2c85218023c4e"
server_key_iv: "bd6eeb8b1db935b458592c3e742bf4c9"
server_key_version: "kuduclusterkey@0"
I20260926 23:33:38.051817 12756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:33:38.079778 12756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:33:38.084746 12756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:33:38.085139 12756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:33:38.107262 12756 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37509
I20260926 23:33:38.107262 12822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37509 every 8 connection(s)
I20260926 23:33:38.109323 12756 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:33:38.116206 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:33:38.118069 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 12756
I20260926 23:33:38.118357 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:33:38.119107 388 external_mini_cluster.cc:1428] Setting key 9a1ed8b66aade7bb36e8e27832281664
I20260926 23:33:38.136093 12823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951: Bootstrap starting.
Sep 26 23:33:38 dist-test-slave-gwfx krb5kdc[12311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:33:38.141007 12823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951: Neither blocks nor log segments found. Creating new log.
I20260926 23:33:38.142660 12823 log.cc:824] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951: Log is configured to *not* fsync() on all Append() calls
I20260926 23:33:38.147408 12823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951: No bootstrap required, opened a new log
I20260926 23:33:38.150539 12826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:38.122009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53000 (local address 127.0.97.62:37509)
0926 23:33:38.127771 (+ 5762us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:38.127792 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:38.127860 (+ 68us) server_negotiation.cc:407] Connection header received
0926 23:33:38.130115 (+ 2255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:38.130234 (+ 119us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:38.130993 (+ 759us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:38.131537 (+ 544us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:38.133198 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:38.134751 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:38.136256 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:38.137085 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:38.137272 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:38.141499 (+ 4227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:38.141510 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:38.141532 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:38.141596 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:38.145396 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:38.146292 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:38.146302 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:38.146310 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:38.146490 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:38.146926 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:38.146932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:38.146936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:38.147613 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:38.147968 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:38.148563 (+ 595us) server_negotiation.cc:299] Negotiation successful
0926 23:33:38.149210 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5381,"thread_start_us":5315,"threads_started":1}
I20260926 23:33:38.160302 12823 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef765b2d56bc4b1a9a19dd5f57fb7951" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37509 } }
I20260926 23:33:38.160957 12823 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:33:38.161083 12823 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef765b2d56bc4b1a9a19dd5f57fb7951, State: Initialized, Role: FOLLOWER
I20260926 23:33:38.161948 12823 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef765b2d56bc4b1a9a19dd5f57fb7951" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37509 } }
I20260926 23:33:38.162281 12823 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:33:38.162436 12823 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:33:38.162626 12823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:33:38.166431 12823 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef765b2d56bc4b1a9a19dd5f57fb7951" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37509 } }
I20260926 23:33:38.167394 12823 leader_election.cc:302] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef765b2d56bc4b1a9a19dd5f57fb7951; no voters:
I20260926 23:33:38.168269 12823 leader_election.cc:288] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:33:38.168385 12828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:33:38.169591 12828 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [term 1 LEADER]: Becoming Leader. State: Replica: ef765b2d56bc4b1a9a19dd5f57fb7951, State: Running, Role: LEADER
I20260926 23:33:38.170295 12828 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef765b2d56bc4b1a9a19dd5f57fb7951" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37509 } }
I20260926 23:33:38.171101 12823 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:33:38.174562 12830 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef765b2d56bc4b1a9a19dd5f57fb7951. Latest consensus state: current_term: 1 leader_uuid: "ef765b2d56bc4b1a9a19dd5f57fb7951" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef765b2d56bc4b1a9a19dd5f57fb7951" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37509 } } }
I20260926 23:33:38.175071 12830 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [sys.catalog]: This master's current role is: LEADER
I20260926 23:33:38.175591 12829 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef765b2d56bc4b1a9a19dd5f57fb7951" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef765b2d56bc4b1a9a19dd5f57fb7951" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37509 } } }
I20260926 23:33:38.175933 12829 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [sys.catalog]: This master's current role is: LEADER
I20260926 23:33:38.189004 12835 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:33:38.197958 12835 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:33:38.198940 12823 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 23:33:38.221316 12835 catalog_manager.cc:1347] Generated new cluster ID: b7568243365e426d976458721aa4a092
I20260926 23:33:38.221516 12835 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:33:38.243155 12835 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:33:38.245620 12835 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:33:38.257896 12835 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951: Generated new TSK 0
I20260926 23:33:38.259513 12835 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:33:39.093356 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40423/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37509
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:33:39 dist-test-slave-gwfx krb5kdc[12311](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790465619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 23:33:39.579491 12868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:33:39.579881 12868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:33:39.580054 12868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:33:39.589514 12868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:33:39.589660 12868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:33:39.589820 12868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:33:39.607379 12868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:40423/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:33:39.609651 12868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:33:39.611876 12868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:33:39.639084 12878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:33:39.640276 12879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:33:39.640636 12881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:33:39.642593 12868 server_base.cc:1034] running on GCE node
I20260926 23:33:39.643524 12868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:33:39.644829 12868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:33:39.646239 12868 hybrid_clock.cc:648] HybridClock initialized: now 1790465619646143 us; error 85 us; skew 500 ppm
I20260926 23:33:39.646673 12868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:33:39 dist-test-slave-gwfx krb5kdc[12311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:33:39.651000 12868 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:33:39.653120 12868 webserver.cc:460] Webserver started at http://127.0.97.1:37797/ using document root <none> and password file <none>
I20260926 23:33:39.654116 12868 fs_manager.cc:359] Metadata directory not provided
I20260926 23:33:39.654290 12868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:33:39.654724 12868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 23:33:39 dist-test-slave-gwfx krb5kdc[12311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 23:33:39.677386 12868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-0/data/instance:
uuid: "2464f9c353ea452484ef7117c52f0135"
format_stamp: "Formatted at 2026-09-26 23:33:39 on dist-test-slave-gwfx"
server_key: "4da73e0e32bfab837548b0619b1134e5"
server_key_iv: "5f08647aff23a1975db9b90b7240a310"
server_key_version: "kuduclusterkey@0"
I20260926 23:33:39.678779 12868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2464f9c353ea452484ef7117c52f0135"
format_stamp: "Formatted at 2026-09-26 23:33:39 on dist-test-slave-gwfx"
server_key: "4da73e0e32bfab837548b0619b1134e5"
server_key_iv: "5f08647aff23a1975db9b90b7240a310"
server_key_version: "kuduclusterkey@0"
I20260926 23:33:39.695214 12868 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.005s sys 0.008s
I20260926 23:33:39.727448 12888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:33:39.729934 12868 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.005s sys 0.007s
I20260926 23:33:39.730201 12868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "2464f9c353ea452484ef7117c52f0135"
format_stamp: "Formatted at 2026-09-26 23:33:39 on dist-test-slave-gwfx"
server_key: "4da73e0e32bfab837548b0619b1134e5"
server_key_iv: "5f08647aff23a1975db9b90b7240a310"
server_key_version: "kuduclusterkey@0"
I20260926 23:33:39.730531 12868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:33:39.758039 12868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:33:39.765210 12868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:33:39.765671 12868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:33:39.767072 12868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:33:39.769240 12868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:33:39.769333 12868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:33:39.769467 12868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:33:39.769533 12868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:33:39.825299 12868 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40705
I20260926 23:33:39.825300 13000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40705 every 8 connection(s)
I20260926 23:33:39.827580 12868 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:33:39.837203 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 12868
I20260926 23:33:39.837450 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:33:39.838322 388 external_mini_cluster.cc:1428] Setting key 678d1424189581a95f629a4bb13b1ecf
Sep 26 23:33:39 dist-test-slave-gwfx krb5kdc[12311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 23:33:39.905150 13003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:39.831647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56301 (local address 127.0.97.62:37509)
0926 23:33:39.875108 (+ 43461us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:39.875115 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:39.875141 (+ 26us) server_negotiation.cc:407] Connection header received
0926 23:33:39.875229 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:39.875235 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:39.875344 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:39.875477 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:39.879652 (+ 4175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:39.880874 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:39.883912 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:39.884559 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:39.884685 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:39.895308 (+ 10623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:39.895315 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:39.895319 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:39.895374 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:39.898989 (+ 3615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:39.900433 (+ 1444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:39.900441 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:39.900444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:39.900655 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:39.901383 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:39.901390 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:39.901393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:39.901818 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:39.901985 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:39.904404 (+ 2419us) server_negotiation.cc:299] Negotiation successful
0926 23:33:39.904721 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43287,"thread_start_us":43088,"threads_started":1}
I20260926 23:33:39.908437 13004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:39.832389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37509 (local address 127.0.97.1:56301)
0926 23:33:39.860289 (+ 27900us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:39.860391 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0926 23:33:39.873993 (+ 13602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:33:39.877183 (+ 3190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:33:39.877200 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:33:39.878010 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:33:39.879158 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:39.879202 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:39.881196 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:39.881236 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:39.883615 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:39.883642 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:39.885189 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:39.885198 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:39.885584 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:39.889245 (+ 3661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:33:39.889294 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:33:39.893049 (+ 3755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:33:39.899308 (+ 6259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:39.899324 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:39.899355 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:39.900223 (+ 868us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:39.900841 (+ 618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:39.900846 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:39.900849 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:39.901236 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:39.902111 (+ 875us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:33:39.902124 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:33:39.902799 (+ 675us) client_negotiation.cc:769] Sending connection context
0926 23:33:39.905843 (+ 3044us) client_negotiation.cc:240] Negotiation successful
0926 23:33:39.906631 (+ 788us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27380,"thread_start_us":31444,"threads_started":1}
I20260926 23:33:39.911648 13001 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37509
I20260926 23:33:39.912130 13001 heartbeater.cc:464] Registering TS with master...
I20260926 23:33:39.913249 13001 heartbeater.cc:511] Master 127.0.97.62:37509 requested a full tablet report, sending...
I20260926 23:33:39.916880 12788 ts_manager.cc:194] Registered new tserver with Master: 2464f9c353ea452484ef7117c52f0135 (127.0.97.1:40705)
I20260926 23:33:39.919275 12788 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
W20260926 23:33:39.942759 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.2: real 0.100s user 0.003s sys 0.020s
I20260926 23:33:39.944139 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40423/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37509
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:33:39 dist-test-slave-gwfx krb5kdc[12311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 23:33:40 dist-test-slave-gwfx krb5kdc[12311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260926 23:33:40.514447 13007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:33:40.515126 13007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:33:40.515350 13007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:33:40.528294 13007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:33:40.528501 13007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:33:40.528788 13007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:33:40.546247 13007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:40423/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:33:40.548799 13007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:33:40.551084 13007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:33:40.583743 13014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:33:40.583906 13015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:33:40.589191 13017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:33:40.589820 13007 server_base.cc:1034] running on GCE node
I20260926 23:33:40.591562 13007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:33:40.601030 13007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:33:40.606879 13007 hybrid_clock.cc:648] HybridClock initialized: now 1790465620606732 us; error 127 us; skew 500 ppm
I20260926 23:33:40.607573 13007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:33:40 dist-test-slave-gwfx krb5kdc[12311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:33:40.613400 13007 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:33:40.635244 13007 webserver.cc:460] Webserver started at http://127.0.97.2:35569/ using document root <none> and password file <none>
I20260926 23:33:40.636607 13007 fs_manager.cc:359] Metadata directory not provided
I20260926 23:33:40.636812 13007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:33:40.637322 13007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 23:33:40 dist-test-slave-gwfx krb5kdc[12311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 23:33:40.667119 13007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-1/data/instance:
uuid: "4da58a0ba41b4aeab6a5d41963a89006"
format_stamp: "Formatted at 2026-09-26 23:33:40 on dist-test-slave-gwfx"
server_key: "d2271f24f2e87302ccf9bacbbed51f42"
server_key_iv: "9fa7e2cc7dde6064bae257fd1b629566"
server_key_version: "kuduclusterkey@0"
I20260926 23:33:40.668874 13007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4da58a0ba41b4aeab6a5d41963a89006"
format_stamp: "Formatted at 2026-09-26 23:33:40 on dist-test-slave-gwfx"
server_key: "d2271f24f2e87302ccf9bacbbed51f42"
server_key_iv: "9fa7e2cc7dde6064bae257fd1b629566"
server_key_version: "kuduclusterkey@0"
I20260926 23:33:40.681787 13007 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.011s sys 0.000s
I20260926 23:33:40.710790 13030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:33:40.713199 13007 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.001s
I20260926 23:33:40.713447 13007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "4da58a0ba41b4aeab6a5d41963a89006"
format_stamp: "Formatted at 2026-09-26 23:33:40 on dist-test-slave-gwfx"
server_key: "d2271f24f2e87302ccf9bacbbed51f42"
server_key_iv: "9fa7e2cc7dde6064bae257fd1b629566"
server_key_version: "kuduclusterkey@0"
I20260926 23:33:40.713770 13007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:33:40.734503 13007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:33:40.739322 13007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:33:40.739676 13007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:33:40.740865 13007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:33:40.742523 13007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:33:40.742600 13007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:33:40.742753 13007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:33:40.742813 13007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:33:40.789002 13007 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33129
I20260926 23:33:40.789057 13143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33129 every 8 connection(s)
I20260926 23:33:40.790560 13007 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:33:40.796386 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 13007
I20260926 23:33:40.796567 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:33:40.797176 388 external_mini_cluster.cc:1428] Setting key f80d350ed8c25928e6d390e194ff3568
Sep 26 23:33:40 dist-test-slave-gwfx krb5kdc[12311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:33:40.815238 13146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:40.794139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35087 (local address 127.0.97.62:37509)
0926 23:33:40.794744 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:40.794752 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:40.797851 (+ 3099us) server_negotiation.cc:407] Connection header received
0926 23:33:40.798677 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:40.798708 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:40.798821 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:40.798968 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:40.801052 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:40.801860 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:40.803647 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:40.804087 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:40.804206 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:40.807961 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:40.807967 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:40.807971 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:40.808019 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:40.811008 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:40.812052 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:40.812058 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:40.812062 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:40.812220 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:40.812744 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:40.812749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:40.812752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:40.813149 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:40.813301 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:40.814472 (+ 1171us) server_negotiation.cc:299] Negotiation successful
0926 23:33:40.814805 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":318,"threads_started":1}
I20260926 23:33:40.817205 13147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:40.794835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37509 (local address 127.0.97.2:35087)
0926 23:33:40.795812 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:40.795870 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 23:33:40.797978 (+ 2108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:33:40.799487 (+ 1509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:33:40.799501 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:33:40.800083 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:33:40.800851 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:40.800871 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:40.802019 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:40.802023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:40.803492 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:40.803506 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:40.804351 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:40.804358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:40.804608 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:40.805347 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:33:40.805374 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:33:40.807689 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:33:40.811185 (+ 3496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:40.811198 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:40.811222 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:40.811885 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:40.812324 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:40.812329 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:40.812332 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:40.812630 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:40.813276 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:33:40.813286 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:33:40.813799 (+ 513us) client_negotiation.cc:769] Sending connection context
0926 23:33:40.814341 (+ 542us) client_negotiation.cc:240] Negotiation successful
0926 23:33:40.815184 (+ 843us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":457,"threads_started":1}
I20260926 23:33:40.820039 13144 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37509
I20260926 23:33:40.820662 13144 heartbeater.cc:464] Registering TS with master...
I20260926 23:33:40.822366 13144 heartbeater.cc:511] Master 127.0.97.62:37509 requested a full tablet report, sending...
I20260926 23:33:40.828737 12788 ts_manager.cc:194] Registered new tserver with Master: 4da58a0ba41b4aeab6a5d41963a89006 (127.0.97.2:33129)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 23:33:40.830020 12788 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35087
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:33:40.870641 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40423/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37509
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:33:40.926551 13001 heartbeater.cc:503] Master 127.0.97.62:37509 was elected leader, sending a full tablet report...
W20260926 23:33:41.275447 13150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:33:41.275825 13150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:33:41.275928 13150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:33:41.283972 13150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:33:41.284125 13150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:33:41.284253 13150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:33:41.295765 13150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:40423/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:33:41.297569 13150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:33:41.299403 13150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:33:41.317488 13158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:33:41.317627 13156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:33:41.317677 13155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:33:41.319018 13150 server_base.cc:1034] running on GCE node
I20260926 23:33:41.319974 13150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:33:41.321439 13150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:33:41.322719 13150 hybrid_clock.cc:648] HybridClock initialized: now 1790465621322631 us; error 57 us; skew 500 ppm
I20260926 23:33:41.323153 13150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:33:41 dist-test-slave-gwfx krb5kdc[12311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:33:41.327495 13150 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:33:41.329677 13150 webserver.cc:460] Webserver started at http://127.0.97.3:33603/ using document root <none> and password file <none>
I20260926 23:33:41.330785 13150 fs_manager.cc:359] Metadata directory not provided
I20260926 23:33:41.330947 13150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:33:41.331341 13150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 23:33:41 dist-test-slave-gwfx krb5kdc[12311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 23:33:41.351351 13150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-2/data/instance:
uuid: "cc881ede23e848e2bfd22a918a751e0f"
format_stamp: "Formatted at 2026-09-26 23:33:41 on dist-test-slave-gwfx"
server_key: "9e4fbbf5e4b91d520445fc28726fabe6"
server_key_iv: "6a82ae7fe32b0574e08c8d4e00da8bf8"
server_key_version: "kuduclusterkey@0"
I20260926 23:33:41.352371 13150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-2/wal/instance:
uuid: "cc881ede23e848e2bfd22a918a751e0f"
format_stamp: "Formatted at 2026-09-26 23:33:41 on dist-test-slave-gwfx"
server_key: "9e4fbbf5e4b91d520445fc28726fabe6"
server_key_iv: "6a82ae7fe32b0574e08c8d4e00da8bf8"
server_key_version: "kuduclusterkey@0"
I20260926 23:33:41.359201 13150 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 23:33:41.378266 13165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:33:41.380352 13150 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:33:41.380573 13150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "cc881ede23e848e2bfd22a918a751e0f"
format_stamp: "Formatted at 2026-09-26 23:33:41 on dist-test-slave-gwfx"
server_key: "9e4fbbf5e4b91d520445fc28726fabe6"
server_key_iv: "6a82ae7fe32b0574e08c8d4e00da8bf8"
server_key_version: "kuduclusterkey@0"
I20260926 23:33:41.380844 13150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:33:41.398113 13150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:33:41.403527 13150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:33:41.403898 13150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:33:41.405154 13150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:33:41.407033 13150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:33:41.407130 13150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:33:41.407261 13150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:33:41.407320 13150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:33:41.442248 13150 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42817
I20260926 23:33:41.442289 13277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42817 every 8 connection(s)
I20260926 23:33:41.443936 13150 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:33:41.444478 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 13150
I20260926 23:33:41.444685 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:33:41.445408 388 external_mini_cluster.cc:1428] Setting key b46591dfce9337782e6fd602584581cc
Sep 26 23:33:41 dist-test-slave-gwfx krb5kdc[12311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:33:41.468297 13280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:41.447347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44895 (local address 127.0.97.62:37509)
0926 23:33:41.447978 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:41.447985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:41.449457 (+ 1472us) server_negotiation.cc:407] Connection header received
0926 23:33:41.451603 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:41.451609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:41.451702 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:41.451815 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:41.453682 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:41.454737 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:41.456590 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:41.457123 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:41.457251 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:41.461401 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:41.461406 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:41.461410 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:41.461469 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:41.464626 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:41.465546 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:41.465553 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:41.465558 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:41.465708 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:41.466184 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:41.466189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:41.466193 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:41.466518 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:41.466662 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:41.467580 (+ 918us) server_negotiation.cc:299] Negotiation successful
0926 23:33:41.467885 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":303,"threads_started":1}
I20260926 23:33:41.469568 13281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:41.448245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37509 (local address 127.0.97.3:44895)
0926 23:33:41.449245 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:41.449319 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0926 23:33:41.451293 (+ 1974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:33:41.452055 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:33:41.452065 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:33:41.452670 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:33:41.453466 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:41.453496 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:41.454916 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:41.454921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:41.456412 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:41.456427 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:41.457428 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:41.457438 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:41.457714 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:41.458501 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:33:41.458530 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:33:41.461110 (+ 2580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:33:41.464814 (+ 3704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:41.464826 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:41.464847 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:41.465415 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:41.465828 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:41.465834 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:41.465838 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:41.466069 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:41.466653 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:33:41.466663 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:33:41.467158 (+ 495us) client_negotiation.cc:769] Sending connection context
0926 23:33:41.467472 (+ 314us) client_negotiation.cc:240] Negotiation successful
0926 23:33:41.468165 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":397,"threads_started":1}
I20260926 23:33:41.471504 13278 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37509
I20260926 23:33:41.471931 13278 heartbeater.cc:464] Registering TS with master...
I20260926 23:33:41.472836 13278 heartbeater.cc:511] Master 127.0.97.62:37509 requested a full tablet report, sending...
I20260926 23:33:41.474908 12788 ts_manager.cc:194] Registered new tserver with Master: cc881ede23e848e2bfd22a918a751e0f (127.0.97.3:42817)
I20260926 23:33:41.476091 12788 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44895
I20260926 23:33:41.476454 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:33:41.478547 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 12868
Sep 26 23:33:41 dist-test-slave-gwfx krb5kdc[12311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:33:41.501245 13283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:41.479911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34814 (local address 127.0.97.1:40705)
0926 23:33:41.480764 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:41.480773 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:41.480802 (+ 29us) server_negotiation.cc:407] Connection header received
0926 23:33:41.480877 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:41.480886 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:41.481165 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:41.481353 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:41.483351 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:41.484645 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:41.486929 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:41.487660 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:41.487846 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:41.493571 (+ 5725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:41.493596 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:41.493619 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:41.493678 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:41.497208 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:41.498431 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:41.498443 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:41.498446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:41.498669 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:41.499198 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:41.499203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:41.499205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:41.499751 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:41.500064 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:41.500487 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 23:33:41.500791 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":477,"threads_started":1}
I20260926 23:33:41.503260 12996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:33:41.596362 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 12868
I20260926 23:33:41.612422 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 13007
Sep 26 23:33:41 dist-test-slave-gwfx krb5kdc[12311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:33:41.638908 13285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:41.618171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34478 (local address 127.0.97.2:33129)
0926 23:33:41.619005 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:41.619014 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:41.619050 (+ 36us) server_negotiation.cc:407] Connection header received
0926 23:33:41.619133 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:41.619143 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:41.619450 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:41.619673 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:41.621103 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:41.622171 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:41.623675 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:41.624292 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:41.624423 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:41.630350 (+ 5927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:41.630482 (+ 132us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:41.630739 (+ 257us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:41.630829 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:41.634749 (+ 3920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:41.635970 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:41.635979 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:41.635981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:41.636238 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:41.636793 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:41.636799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:41.636803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:41.637390 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:41.637866 (+ 476us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:41.638138 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 23:33:41.638431 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":376,"threads_started":1}
I20260926 23:33:41.640972 13139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:33:41.733014 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 13007
I20260926 23:33:41.748188 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 13150
Sep 26 23:33:41 dist-test-slave-gwfx krb5kdc[12311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:33:41.768268 13287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:41.749507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39812 (local address 127.0.97.3:42817)
0926 23:33:41.750265 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:41.750274 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:41.750308 (+ 34us) server_negotiation.cc:407] Connection header received
0926 23:33:41.750448 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:41.750456 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:41.750825 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:41.751015 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:41.752514 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:41.753614 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:41.755110 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:41.755765 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:41.755909 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:41.761243 (+ 5334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:41.761266 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:41.761284 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:41.761345 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:41.764436 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:41.765412 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:41.765419 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:41.765422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:41.765575 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:41.766060 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:41.766063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:41.766065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:41.766474 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:41.766781 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:41.767441 (+ 660us) server_negotiation.cc:299] Negotiation successful
0926 23:33:41.767703 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":375,"threads_started":1}
I20260926 23:33:41.770058 13273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:33:41.858448 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 13150
I20260926 23:33:41.873495 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 12756
I20260926 23:33:41.875304 12818 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:33:41.988268 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 12756
I20260926 23:33:42.010167 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37509
--webserver_interface=127.0.97.62
--webserver_port=34329
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37509
--ranger_config_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40423/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:40423/kms with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:33:42.378805 13290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:33:42.379146 13290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:33:42.379264 13290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:33:42.386847 13290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:33:42.386991 13290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:33:42.387038 13290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:33:42.387120 13290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:33:42.387185 13290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:33:42.397846 13290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:40423/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37509
--ranger_config_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:33:42.399355 13290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:33:42.400764 13290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:33:42.415269 13295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:33:42.415514 13296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:33:42.415583 13298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:33:42.420907 13290 server_base.cc:1034] running on GCE node
I20260926 23:33:42.421911 13290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:33:42.423969 13290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:33:42.425432 13290 hybrid_clock.cc:648] HybridClock initialized: now 1790465622425393 us; error 110 us; skew 500 ppm
I20260926 23:33:42.425915 13290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:33:42 dist-test-slave-gwfx krb5kdc[12311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:33:42.430027 13290 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:33:42.432570 13290 webserver.cc:460] Webserver started at http://127.0.97.62:34329/ using document root <none> and password file <none>
I20260926 23:33:42.433383 13290 fs_manager.cc:359] Metadata directory not provided
I20260926 23:33:42.433532 13290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 23:33:42 dist-test-slave-gwfx krb5kdc[12311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 23:33:42.507859 13290 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 23:33:42.511673 13306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:33:42.513468 13290 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 23:33:42.513712 13290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "ef765b2d56bc4b1a9a19dd5f57fb7951"
format_stamp: "Formatted at 2026-09-26 23:33:38 on dist-test-slave-gwfx"
server_key: "b034f29c4087cd911cc2c85218023c4e"
server_key_iv: "bd6eeb8b1db935b458592c3e742bf4c9"
server_key_version: "kuduclusterkey@0"
I20260926 23:33:42.514873 13290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:33:42.539939 13290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:33:42.544162 13290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:33:42.544453 13290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:33:42.565084 13290 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37509
I20260926 23:33:42.565155 13357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37509 every 8 connection(s)
I20260926 23:33:42.566757 13290 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:33:42.570906 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 13290
I20260926 23:33:42.571794 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40423/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:40705
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37509
--webserver_port=37797
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:40423/kms with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:33:42.576083 13358 sys_catalog.cc:263] Verifying existing consensus state
I20260926 23:33:42.580408 13358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951: Bootstrap starting.
I20260926 23:33:42.604532 13358 log.cc:824] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951: Log is configured to *not* fsync() on all Append() calls
I20260926 23:33:42.616616 13358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 23:33:42.617287 13358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951: Bootstrap complete.
I20260926 23:33:42.627780 13358 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef765b2d56bc4b1a9a19dd5f57fb7951" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37509 } }
I20260926 23:33:42.628911 13358 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef765b2d56bc4b1a9a19dd5f57fb7951, State: Initialized, Role: FOLLOWER
I20260926 23:33:42.629860 13358 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef765b2d56bc4b1a9a19dd5f57fb7951" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37509 } }
I20260926 23:33:42.630152 13358 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:33:42.630275 13358 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:33:42.630451 13358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [term 1 FOLLOWER]: Advancing to term 2
I20260926 23:33:42.635488 13358 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef765b2d56bc4b1a9a19dd5f57fb7951" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37509 } }
I20260926 23:33:42.636148 13358 leader_election.cc:302] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef765b2d56bc4b1a9a19dd5f57fb7951; no voters:
I20260926 23:33:42.636888 13358 leader_election.cc:288] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 23:33:42.637049 13362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 23:33:42.638279 13362 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [term 2 LEADER]: Becoming Leader. State: Replica: ef765b2d56bc4b1a9a19dd5f57fb7951, State: Running, Role: LEADER
I20260926 23:33:42.639099 13362 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef765b2d56bc4b1a9a19dd5f57fb7951" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37509 } }
I20260926 23:33:42.639612 13358 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:33:42.642263 13364 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef765b2d56bc4b1a9a19dd5f57fb7951. Latest consensus state: current_term: 2 leader_uuid: "ef765b2d56bc4b1a9a19dd5f57fb7951" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef765b2d56bc4b1a9a19dd5f57fb7951" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37509 } } }
I20260926 23:33:42.642746 13364 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [sys.catalog]: This master's current role is: LEADER
I20260926 23:33:42.643433 13363 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ef765b2d56bc4b1a9a19dd5f57fb7951" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef765b2d56bc4b1a9a19dd5f57fb7951" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37509 } } }
I20260926 23:33:42.643816 13363 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951 [sys.catalog]: This master's current role is: LEADER
I20260926 23:33:42.653934 13368 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:33:42.661101 13368 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:33:42.662971 13368 catalog_manager.cc:1259] Loaded cluster ID: b7568243365e426d976458721aa4a092
I20260926 23:33:42.663206 13368 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:33:42.668465 13368 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:33:42.670431 13368 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951: Loaded TSK: 0
I20260926 23:33:42.672472 13368 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:33:42.681730 13358 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260926 23:33:43.114547 13360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:33:43.115159 13360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:33:43.115379 13360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:33:43.128021 13360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:33:43.128222 13360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:33:43.128449 13360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:33:43.145635 13360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:40423/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:40705
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=37797
--tserver_master_addrs=127.0.97.62:37509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:33:43.148065 13360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:33:43.150421 13360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:33:43.187091 13395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:33:43.187981 13396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:33:43.190963 13360 server_base.cc:1034] running on GCE node
W20260926 23:33:43.192176 13398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:33:43.193073 13360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:33:43.194568 13360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:33:43.195995 13360 hybrid_clock.cc:648] HybridClock initialized: now 1790465623195909 us; error 75 us; skew 500 ppm
I20260926 23:33:43.196444 13360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:33:43 dist-test-slave-gwfx krb5kdc[12311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:33:43.200310 13360 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:33:43.202569 13360 webserver.cc:460] Webserver started at http://127.0.97.1:37797/ using document root <none> and password file <none>
I20260926 23:33:43.203723 13360 fs_manager.cc:359] Metadata directory not provided
I20260926 23:33:43.203912 13360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 23:33:43 dist-test-slave-gwfx krb5kdc[12311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 23:33:43.235548 13360 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.007s sys 0.000s
I20260926 23:33:43.243770 13406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:33:43.246603 13360 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.004s sys 0.004s
I20260926 23:33:43.246959 13360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "2464f9c353ea452484ef7117c52f0135"
format_stamp: "Formatted at 2026-09-26 23:33:39 on dist-test-slave-gwfx"
server_key: "4da73e0e32bfab837548b0619b1134e5"
server_key_iv: "5f08647aff23a1975db9b90b7240a310"
server_key_version: "kuduclusterkey@0"
I20260926 23:33:43.249003 13360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:33:43.291965 13360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:33:43.298928 13360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:33:43.299255 13360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:33:43.300657 13360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:33:43.302632 13360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:33:43.302902 13360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 23:33:43.303066 13360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:33:43.303156 13360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:33:43.391518 13360 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40705
I20260926 23:33:43.391539 13519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40705 every 8 connection(s)
I20260926 23:33:43.393644 13360 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:33:43.397644 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 13360
I20260926 23:33:43.398588 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40423/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33129
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37509
--webserver_port=35569
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:40423/kms with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:33:43 dist-test-slave-gwfx krb5kdc[12311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:33:43.434221 13522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:43.398431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37509 (local address 127.0.97.1:33179)
0926 23:33:43.399560 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:43.399652 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0926 23:33:43.402365 (+ 2713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:33:43.408427 (+ 6062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:33:43.408441 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:33:43.409172 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:33:43.410130 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:43.410163 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:43.413208 (+ 3045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:43.413215 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:43.415737 (+ 2522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:43.415786 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:43.418151 (+ 2365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:43.418162 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:43.418491 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:43.419460 (+ 969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:33:43.419497 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:33:43.422714 (+ 3217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:33:43.427989 (+ 5275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:43.428002 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:43.428025 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:43.428832 (+ 807us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:43.429478 (+ 646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:43.429483 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:43.429487 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:43.429753 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:43.431022 (+ 1269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:33:43.431032 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:33:43.431499 (+ 467us) client_negotiation.cc:769] Sending connection context
0926 23:33:43.431831 (+ 332us) client_negotiation.cc:240] Negotiation successful
0926 23:33:43.432576 (+ 745us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":3111,"threads_started":1}
I20260926 23:33:43.437956 13523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:43.399856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33179 (local address 127.0.97.62:37509)
0926 23:33:43.403213 (+ 3357us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:43.403234 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:43.403294 (+ 60us) server_negotiation.cc:407] Connection header received
0926 23:33:43.403819 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:43.403843 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:43.404448 (+ 605us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:43.404936 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:43.410501 (+ 5565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:43.412827 (+ 2326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:43.416060 (+ 3233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:43.417333 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:43.417566 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:43.423156 (+ 5590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:43.423172 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:43.423199 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:43.423284 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:43.427707 (+ 4423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:43.429056 (+ 1349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:43.429068 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:43.429076 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:43.429313 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:43.429894 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:43.429900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:43.429904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:43.430679 (+ 775us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:43.430917 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:43.435529 (+ 4612us) server_negotiation.cc:299] Negotiation successful
0926 23:33:43.436163 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3087,"thread_start_us":2840,"threads_started":1}
I20260926 23:33:43.443014 13520 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37509
I20260926 23:33:43.443526 13520 heartbeater.cc:464] Registering TS with master...
I20260926 23:33:43.444792 13520 heartbeater.cc:511] Master 127.0.97.62:37509 requested a full tablet report, sending...
W20260926 23:33:43.449677 13520 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37509 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 23:33:43.450075 13520 heartbeater.cc:464] Registering TS with master...
I20260926 23:33:43.450506 13520 heartbeater.cc:511] Master 127.0.97.62:37509 requested a full tablet report, sending...
I20260926 23:33:43.457026 13520 heartbeater.cc:464] Registering TS with master...
I20260926 23:33:43.457595 13520 heartbeater.cc:511] Master 127.0.97.62:37509 requested a full tablet report, sending...
W20260926 23:33:43.459415 13520 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 23:33:43.928288 13524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:33:43.928694 13524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:33:43.928882 13524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:33:43.937175 13524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:33:43.937350 13524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:33:43.937536 13524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:33:43.949436 13524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:40423/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=35569
--tserver_master_addrs=127.0.97.62:37509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:33:43.951387 13524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:33:43.953099 13524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:33:43.973318 13538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:33:43.977499 13539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:33:43.983338 13524 server_base.cc:1034] running on GCE node
W20260926 23:33:43.983808 13541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:33:43.984934 13524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:33:43.998085 13524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:33:43.999621 13524 hybrid_clock.cc:648] HybridClock initialized: now 1790465623999528 us; error 114 us; skew 500 ppm
I20260926 23:33:44.000218 13524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:33:44 dist-test-slave-gwfx krb5kdc[12311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:33:44.011215 13524 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:33:44.016621 13524 webserver.cc:460] Webserver started at http://127.0.97.2:35569/ using document root <none> and password file <none>
I20260926 23:33:44.018134 13524 fs_manager.cc:359] Metadata directory not provided
I20260926 23:33:44.018450 13524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 23:33:44 dist-test-slave-gwfx krb5kdc[12311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 23:33:44.061051 13524 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.006s sys 0.000s
I20260926 23:33:44.065925 13551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:33:44.068395 13524 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260926 23:33:44.068671 13524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "4da58a0ba41b4aeab6a5d41963a89006"
format_stamp: "Formatted at 2026-09-26 23:33:40 on dist-test-slave-gwfx"
server_key: "d2271f24f2e87302ccf9bacbbed51f42"
server_key_iv: "9fa7e2cc7dde6064bae257fd1b629566"
server_key_version: "kuduclusterkey@0"
I20260926 23:33:44.070466 13524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:33:44.094075 13524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
Sep 26 23:33:44 dist-test-slave-gwfx krb5kdc[12311](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790465624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:33:44.101142 13524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:33:44.101591 13524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:33:44.103243 13524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:33:44.107662 13524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:33:44.107829 13524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:33:44.107987 13524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:33:44.108080 13524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:33:44.188726 13524 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33129
I20260926 23:33:44.188984 13663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33129 every 8 connection(s)
I20260926 23:33:44.191282 13524 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:33:44.197734 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 13524
I20260926 23:33:44.198607 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40423/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:42817
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37509
--webserver_port=33603
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:40423/kms with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:33:44 dist-test-slave-gwfx krb5kdc[12311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:33:44.254205 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:44.196686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37509 (local address 127.0.97.2:35641)
0926 23:33:44.223855 (+ 27169us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:44.223959 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0926 23:33:44.229632 (+ 5673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:33:44.231311 (+ 1679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:33:44.231328 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:33:44.232121 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:33:44.233285 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:44.233328 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:44.235242 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:44.235248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:44.237851 (+ 2603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:44.237880 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:44.239221 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:44.239233 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:44.239738 (+ 505us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:44.240833 (+ 1095us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:33:44.240879 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:33:44.244034 (+ 3155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:33:44.248762 (+ 4728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:44.248773 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:44.248792 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:44.249520 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:44.250103 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:44.250109 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:44.250112 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:44.250402 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:44.251222 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:33:44.251231 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:33:44.251679 (+ 448us) client_negotiation.cc:769] Sending connection context
0926 23:33:44.252040 (+ 361us) client_negotiation.cc:240] Negotiation successful
0926 23:33:44.252711 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26571,"thread_start_us":29856,"threads_started":1}
I20260926 23:33:44.255517 13667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:44.195110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35641 (local address 127.0.97.62:37509)
0926 23:33:44.228798 (+ 33688us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:44.228807 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:44.228849 (+ 42us) server_negotiation.cc:407] Connection header received
0926 23:33:44.230232 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:44.230240 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:44.230365 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:44.230491 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:44.233613 (+ 3122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:44.234986 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:44.238104 (+ 3118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:44.238845 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:44.239093 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:44.244448 (+ 5355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:44.244457 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:44.244462 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:44.244516 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:44.248531 (+ 4015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:44.249732 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:44.249740 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:44.249744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:44.249943 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:44.250567 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:44.250573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:44.250576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:44.251056 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:44.251221 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:44.254744 (+ 3523us) server_negotiation.cc:299] Negotiation successful
0926 23:33:44.255086 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33471,"thread_start_us":33740,"threads_started":1}
I20260926 23:33:44.258152 13665 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37509
I20260926 23:33:44.258582 13665 heartbeater.cc:464] Registering TS with master...
I20260926 23:33:44.259778 13665 heartbeater.cc:511] Master 127.0.97.62:37509 requested a full tablet report, sending...
I20260926 23:33:44.262724 13323 ts_manager.cc:194] Registered new tserver with Master: 4da58a0ba41b4aeab6a5d41963a89006 (127.0.97.2:33129)
I20260926 23:33:44.265688 13323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35641
I20260926 23:33:44.459569 13520 heartbeater.cc:464] Registering TS with master...
I20260926 23:33:44.460194 13520 heartbeater.cc:511] Master 127.0.97.62:37509 requested a full tablet report, sending...
I20260926 23:33:44.461833 13323 ts_manager.cc:194] Registered new tserver with Master: 2464f9c353ea452484ef7117c52f0135 (127.0.97.1:40705)
I20260926 23:33:44.463266 13323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33179
Sep 26 23:33:44 dist-test-slave-gwfx krb5kdc[12311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260926 23:33:44.753257 13669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:33:44.753777 13669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:33:44.753969 13669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:33:44.765218 13669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:33:44.765386 13669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:33:44.765611 13669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:33:44.781816 13669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:40423/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:42817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=33603
--tserver_master_addrs=127.0.97.62:37509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:33:44.784112 13669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:33:44.786232 13669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:33:44.809558 13674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:33:44.811386 13675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:33:44.824806 13677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:33:44.825773 13669 server_base.cc:1034] running on GCE node
I20260926 23:33:44.826906 13669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:33:44.828619 13669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:33:44.829980 13669 hybrid_clock.cc:648] HybridClock initialized: now 1790465624829906 us; error 67 us; skew 500 ppm
I20260926 23:33:44.830449 13669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:33:44 dist-test-slave-gwfx krb5kdc[12311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:33:44.835536 13669 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:33:44.838164 13669 webserver.cc:460] Webserver started at http://127.0.97.3:33603/ using document root <none> and password file <none>
I20260926 23:33:44.839437 13669 fs_manager.cc:359] Metadata directory not provided
I20260926 23:33:44.839627 13669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 23:33:44 dist-test-slave-gwfx krb5kdc[12311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 23:33:44.868736 13669 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260926 23:33:44.873343 13685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:33:44.875456 13669 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 23:33:44.875720 13669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "cc881ede23e848e2bfd22a918a751e0f"
format_stamp: "Formatted at 2026-09-26 23:33:41 on dist-test-slave-gwfx"
server_key: "9e4fbbf5e4b91d520445fc28726fabe6"
server_key_iv: "6a82ae7fe32b0574e08c8d4e00da8bf8"
server_key_version: "kuduclusterkey@0"
I20260926 23:33:44.877452 13669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:33:44.900633 13669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:33:44.906723 13669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:33:44.907166 13669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:33:44.908679 13669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:33:44.911679 13669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:33:44.911921 13669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 23:33:44.912098 13669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:33:44.912267 13669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 23:33:44 dist-test-slave-gwfx krb5kdc[12311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 23:33:45.177006 13669 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42817
I20260926 23:33:45.177053 13798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42817 every 8 connection(s)
I20260926 23:33:45.179595 13669 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:33:45.183856 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 13669
I20260926 23:33:45.237440 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:45.185158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53014 (local address 127.0.97.62:37509)
0926 23:33:45.202531 (+ 17373us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:45.202539 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:45.202565 (+ 26us) server_negotiation.cc:407] Connection header received
0926 23:33:45.202648 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:45.202654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:45.202819 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:45.202960 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:45.211623 (+ 8663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.212978 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.218233 (+ 5255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.219340 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.220760 (+ 1420us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:45.224266 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:45.224275 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:45.224282 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:45.224391 (+ 109us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:45.230047 (+ 5656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:45.231425 (+ 1378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:45.231432 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:45.231436 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:45.231670 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:45.232536 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:45.232542 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:45.232546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:45.233028 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:45.235626 (+ 2598us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:45.235839 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 23:33:45.236918 (+ 1079us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17202,"thread_start_us":17063,"threads_started":1}
Sep 26 23:33:45 dist-test-slave-gwfx krb5kdc[12311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:33:45.271306 13803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:45.217624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37509 (local address 127.0.97.3:34731)
0926 23:33:45.241973 (+ 24349us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:45.242045 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0926 23:33:45.244714 (+ 2669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:33:45.245766 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:33:45.245779 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:33:45.246379 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:33:45.247376 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:45.247410 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.249254 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.249258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:45.250823 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:45.250838 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.252006 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.252013 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:45.252419 (+ 406us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:45.254845 (+ 2426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:33:45.255033 (+ 188us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:33:45.258901 (+ 3868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:33:45.264121 (+ 5220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:45.264136 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:45.264165 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:45.265047 (+ 882us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:45.265763 (+ 716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:45.265769 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:45.265773 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:45.266175 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:45.267352 (+ 1177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:33:45.267363 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:33:45.268051 (+ 688us) client_negotiation.cc:769] Sending connection context
0926 23:33:45.268477 (+ 426us) client_negotiation.cc:240] Negotiation successful
0926 23:33:45.269324 (+ 847us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23896,"thread_start_us":23989,"threads_started":1}
I20260926 23:33:45.271845 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:45.218258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34731 (local address 127.0.97.62:37509)
0926 23:33:45.236753 (+ 18495us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:45.236759 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:45.243802 (+ 7043us) server_negotiation.cc:407] Connection header received
0926 23:33:45.245131 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:45.245138 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:45.245279 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:45.245412 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:45.247656 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.248992 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.250995 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.251672 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.251823 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:45.259350 (+ 7527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:45.259358 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:45.259363 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:45.259426 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:45.263708 (+ 4282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:45.265326 (+ 1618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:45.265332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:45.265336 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:45.265549 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:45.266396 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:45.266402 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:45.266406 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:45.267158 (+ 752us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:45.270879 (+ 3721us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:45.271096 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 23:33:45.271432 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18341,"thread_start_us":18512,"threads_started":1}
I20260926 23:33:45.272814 13665 heartbeater.cc:503] Master 127.0.97.62:37509 was elected leader, sending a full tablet report...
I20260926 23:33:45.273583 13799 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37509
I20260926 23:33:45.274019 13799 heartbeater.cc:464] Registering TS with master...
I20260926 23:33:45.275415 13799 heartbeater.cc:511] Master 127.0.97.62:37509 requested a full tablet report, sending...
I20260926 23:33:45.277796 13323 ts_manager.cc:194] Registered new tserver with Master: cc881ede23e848e2bfd22a918a751e0f (127.0.97.3:42817)
I20260926 23:33:45.278918 13323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34731
I20260926 23:33:45.282135 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:33:45.302816 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:45.286274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53020 (local address 127.0.97.62:37509)
0926 23:33:45.286500 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:45.286507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:45.287050 (+ 543us) server_negotiation.cc:407] Connection header received
0926 23:33:45.287251 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:45.287257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:45.287351 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:45.287442 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:45.289269 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.290066 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.292082 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.292649 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.292753 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:45.295907 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:45.295913 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:45.295916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:45.295978 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:45.299291 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:45.300291 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:45.300295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:45.300298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:45.300484 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:45.300953 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:45.300957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:45.300960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:45.301384 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:45.301673 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:45.302074 (+ 401us) server_negotiation.cc:299] Negotiation successful
0926 23:33:45.302378 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 23:33:45.310865 13323 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:33:45.324113 13323 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:33:45.392433 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:45.346319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40705 (local address 127.0.0.1:34820)
0926 23:33:45.347764 (+ 1445us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:45.347826 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 23:33:45.348121 (+ 295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:33:45.376753 (+ 28632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:33:45.376759 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:33:45.376791 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:33:45.377268 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:45.377280 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.386487 (+ 9207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.386495 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:45.388797 (+ 2302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:45.388817 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.391123 (+ 2306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.391131 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:45.391418 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:45.391559 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 23:33:45.391723 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 23:33:45.392006 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1060,"threads_started":1}
I20260926 23:33:45.398378 13823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:45.346958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39822 (local address 127.0.97.3:42817)
0926 23:33:45.359373 (+ 12415us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:45.359387 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:45.372096 (+ 12709us) server_negotiation.cc:407] Connection header received
0926 23:33:45.374015 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:45.374026 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:45.374455 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:45.374738 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:33:45.377691 (+ 2953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.380273 (+ 2582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.387809 (+ 7536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.389655 (+ 1846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.389937 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:45.390352 (+ 415us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:45.397669 (+ 7317us) server_negotiation.cc:299] Negotiation successful
0926 23:33:45.397952 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12186,"thread_start_us":11990,"threads_started":1}
I20260926 23:33:45.398469 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:45.347405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34820 (local address 127.0.97.1:40705)
0926 23:33:45.373200 (+ 25795us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:45.373213 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:45.373252 (+ 39us) server_negotiation.cc:407] Connection header received
0926 23:33:45.373426 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:45.373439 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:45.376443 (+ 3004us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:45.382811 (+ 6368us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:33:45.383243 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.386229 (+ 2986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.389076 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.390811 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.397340 (+ 6529us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:45.397727 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:45.397819 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 23:33:45.398070 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25534,"thread_start_us":37343,"threads_started":1}
I20260926 23:33:45.398988 13825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:45.346777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42817 (local address 127.0.0.1:39822)
0926 23:33:45.371754 (+ 24977us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:45.371793 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 23:33:45.373721 (+ 1928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:33:45.375258 (+ 1537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:33:45.375268 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:33:45.375312 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:33:45.375894 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:45.375909 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.380557 (+ 4648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.380563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:45.387521 (+ 6958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:45.387548 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.396388 (+ 8840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.396396 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:45.396671 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:45.396720 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 23:33:45.398403 (+ 1683us) client_negotiation.cc:240] Negotiation successful
0926 23:33:45.398608 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24703,"thread_start_us":889,"threads_started":1}
I20260926 23:33:45.404337 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:45.353006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33129 (local address 127.0.0.1:34480)
0926 23:33:45.393151 (+ 40145us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:45.393178 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 23:33:45.393341 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:33:45.394468 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:33:45.394474 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:33:45.394510 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:33:45.394795 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:45.394810 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.399322 (+ 4512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.399329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:45.401317 (+ 1988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:45.401335 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.403467 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.403475 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:45.403697 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:45.403748 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 23:33:45.403829 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 23:33:45.403998 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39981,"thread_start_us":19135,"threads_started":1}
I20260926 23:33:45.404486 13733 tablet_service.cc:1467] Processing CreateTablet for tablet 5a44721a6ed34f63b867bde7aea54b75 (DEFAULT_TABLE table=test-table [id=abb70f5644f44759b0d005030acf1de2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:33:45.404486 13455 tablet_service.cc:1467] Processing CreateTablet for tablet 5a44721a6ed34f63b867bde7aea54b75 (DEFAULT_TABLE table=test-table [id=abb70f5644f44759b0d005030acf1de2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:33:45.406108 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:45.353175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34480 (local address 127.0.97.2:33129)
0926 23:33:45.384724 (+ 31549us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:45.384736 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:45.393538 (+ 8802us) server_negotiation.cc:407] Connection header received
0926 23:33:45.393701 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:45.393711 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:45.394141 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:45.395039 (+ 898us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:33:45.395365 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.397068 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.401586 (+ 4518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.403225 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.405131 (+ 1906us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:45.405432 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:45.405525 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 23:33:45.405730 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31275,"thread_start_us":31006,"threads_started":1}
I20260926 23:33:45.406469 13733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a44721a6ed34f63b867bde7aea54b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:33:45.406492 13455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a44721a6ed34f63b867bde7aea54b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:33:45.411263 13599 tablet_service.cc:1467] Processing CreateTablet for tablet 5a44721a6ed34f63b867bde7aea54b75 (DEFAULT_TABLE table=test-table [id=abb70f5644f44759b0d005030acf1de2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:33:45.413815 13599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a44721a6ed34f63b867bde7aea54b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:33:45.431272 13827 tablet_bootstrap.cc:492] T 5a44721a6ed34f63b867bde7aea54b75 P cc881ede23e848e2bfd22a918a751e0f: Bootstrap starting.
I20260926 23:33:45.439190 13827 tablet_bootstrap.cc:654] T 5a44721a6ed34f63b867bde7aea54b75 P cc881ede23e848e2bfd22a918a751e0f: Neither blocks nor log segments found. Creating new log.
I20260926 23:33:45.441465 13828 tablet_bootstrap.cc:492] T 5a44721a6ed34f63b867bde7aea54b75 P 4da58a0ba41b4aeab6a5d41963a89006: Bootstrap starting.
I20260926 23:33:45.442516 13829 tablet_bootstrap.cc:492] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135: Bootstrap starting.
I20260926 23:33:45.443449 13827 log.cc:824] T 5a44721a6ed34f63b867bde7aea54b75 P cc881ede23e848e2bfd22a918a751e0f: Log is configured to *not* fsync() on all Append() calls
I20260926 23:33:45.448333 13829 tablet_bootstrap.cc:654] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135: Neither blocks nor log segments found. Creating new log.
I20260926 23:33:45.448333 13828 tablet_bootstrap.cc:654] T 5a44721a6ed34f63b867bde7aea54b75 P 4da58a0ba41b4aeab6a5d41963a89006: Neither blocks nor log segments found. Creating new log.
I20260926 23:33:45.449509 13827 tablet_bootstrap.cc:492] T 5a44721a6ed34f63b867bde7aea54b75 P cc881ede23e848e2bfd22a918a751e0f: No bootstrap required, opened a new log
I20260926 23:33:45.449949 13827 ts_tablet_manager.cc:1397] T 5a44721a6ed34f63b867bde7aea54b75 P cc881ede23e848e2bfd22a918a751e0f: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.000s
I20260926 23:33:45.452102 13829 log.cc:824] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135: Log is configured to *not* fsync() on all Append() calls
I20260926 23:33:45.456271 13828 log.cc:824] T 5a44721a6ed34f63b867bde7aea54b75 P 4da58a0ba41b4aeab6a5d41963a89006: Log is configured to *not* fsync() on all Append() calls
I20260926 23:33:45.461817 13827 raft_consensus.cc:348] T 5a44721a6ed34f63b867bde7aea54b75 P cc881ede23e848e2bfd22a918a751e0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2464f9c353ea452484ef7117c52f0135" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40705 } } peers { permanent_uuid: "cc881ede23e848e2bfd22a918a751e0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } } peers { permanent_uuid: "4da58a0ba41b4aeab6a5d41963a89006" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33129 } }
I20260926 23:33:45.462486 13827 raft_consensus.cc:374] T 5a44721a6ed34f63b867bde7aea54b75 P cc881ede23e848e2bfd22a918a751e0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:33:45.462603 13827 raft_consensus.cc:729] T 5a44721a6ed34f63b867bde7aea54b75 P cc881ede23e848e2bfd22a918a751e0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc881ede23e848e2bfd22a918a751e0f, State: Initialized, Role: FOLLOWER
I20260926 23:33:45.465118 13827 consensus_queue.cc:260] T 5a44721a6ed34f63b867bde7aea54b75 P cc881ede23e848e2bfd22a918a751e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2464f9c353ea452484ef7117c52f0135" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40705 } } peers { permanent_uuid: "cc881ede23e848e2bfd22a918a751e0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } } peers { permanent_uuid: "4da58a0ba41b4aeab6a5d41963a89006" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33129 } }
I20260926 23:33:45.466857 13799 heartbeater.cc:503] Master 127.0.97.62:37509 was elected leader, sending a full tablet report...
I20260926 23:33:45.467542 13828 tablet_bootstrap.cc:492] T 5a44721a6ed34f63b867bde7aea54b75 P 4da58a0ba41b4aeab6a5d41963a89006: No bootstrap required, opened a new log
I20260926 23:33:45.467659 13829 tablet_bootstrap.cc:492] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135: No bootstrap required, opened a new log
I20260926 23:33:45.467919 13828 ts_tablet_manager.cc:1397] T 5a44721a6ed34f63b867bde7aea54b75 P 4da58a0ba41b4aeab6a5d41963a89006: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.005s
I20260926 23:33:45.467979 13829 ts_tablet_manager.cc:1397] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.007s
I20260926 23:33:45.468833 13520 heartbeater.cc:503] Master 127.0.97.62:37509 was elected leader, sending a full tablet report...
I20260926 23:33:45.479456 13827 ts_tablet_manager.cc:1428] T 5a44721a6ed34f63b867bde7aea54b75 P cc881ede23e848e2bfd22a918a751e0f: Time spent starting tablet: real 0.029s user 0.008s sys 0.007s
I20260926 23:33:45.481501 13828 raft_consensus.cc:348] T 5a44721a6ed34f63b867bde7aea54b75 P 4da58a0ba41b4aeab6a5d41963a89006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2464f9c353ea452484ef7117c52f0135" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40705 } } peers { permanent_uuid: "cc881ede23e848e2bfd22a918a751e0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } } peers { permanent_uuid: "4da58a0ba41b4aeab6a5d41963a89006" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33129 } }
I20260926 23:33:45.481501 13829 raft_consensus.cc:348] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2464f9c353ea452484ef7117c52f0135" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40705 } } peers { permanent_uuid: "cc881ede23e848e2bfd22a918a751e0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } } peers { permanent_uuid: "4da58a0ba41b4aeab6a5d41963a89006" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33129 } }
I20260926 23:33:45.482148 13829 raft_consensus.cc:374] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:33:45.482304 13828 raft_consensus.cc:374] T 5a44721a6ed34f63b867bde7aea54b75 P 4da58a0ba41b4aeab6a5d41963a89006 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:33:45.482366 13829 raft_consensus.cc:729] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2464f9c353ea452484ef7117c52f0135, State: Initialized, Role: FOLLOWER
I20260926 23:33:45.482487 13828 raft_consensus.cc:729] T 5a44721a6ed34f63b867bde7aea54b75 P 4da58a0ba41b4aeab6a5d41963a89006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da58a0ba41b4aeab6a5d41963a89006, State: Initialized, Role: FOLLOWER
I20260926 23:33:45.483335 13828 consensus_queue.cc:260] T 5a44721a6ed34f63b867bde7aea54b75 P 4da58a0ba41b4aeab6a5d41963a89006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2464f9c353ea452484ef7117c52f0135" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40705 } } peers { permanent_uuid: "cc881ede23e848e2bfd22a918a751e0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } } peers { permanent_uuid: "4da58a0ba41b4aeab6a5d41963a89006" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33129 } }
I20260926 23:33:45.483337 13829 consensus_queue.cc:260] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2464f9c353ea452484ef7117c52f0135" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40705 } } peers { permanent_uuid: "cc881ede23e848e2bfd22a918a751e0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } } peers { permanent_uuid: "4da58a0ba41b4aeab6a5d41963a89006" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33129 } }
I20260926 23:33:45.485949 13828 ts_tablet_manager.cc:1428] T 5a44721a6ed34f63b867bde7aea54b75 P 4da58a0ba41b4aeab6a5d41963a89006: Time spent starting tablet: real 0.018s user 0.006s sys 0.008s
I20260926 23:33:45.488375 13829 ts_tablet_manager.cc:1428] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135: Time spent starting tablet: real 0.020s user 0.013s sys 0.004s
W20260926 23:33:45.492208 13802 tablet.cc:2369] T 5a44721a6ed34f63b867bde7aea54b75 P cc881ede23e848e2bfd22a918a751e0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:33:45.551898 13835 raft_consensus.cc:482] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:33:45.552348 13835 raft_consensus.cc:504] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2464f9c353ea452484ef7117c52f0135" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40705 } } peers { permanent_uuid: "cc881ede23e848e2bfd22a918a751e0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } } peers { permanent_uuid: "4da58a0ba41b4aeab6a5d41963a89006" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33129 } }
I20260926 23:33:45.555353 13835 leader_election.cc:288] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc881ede23e848e2bfd22a918a751e0f (127.0.97.3:42817), 4da58a0ba41b4aeab6a5d41963a89006 (127.0.97.2:33129)
I20260926 23:33:45.564505 13823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:45.555967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36533 (local address 127.0.97.3:42817)
0926 23:33:45.556264 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:45.556273 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:45.557389 (+ 1116us) server_negotiation.cc:407] Connection header received
0926 23:33:45.557530 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:45.557536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:45.557624 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:45.557735 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:33:45.558609 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.559824 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.562430 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.563368 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.563525 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:45.563673 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:45.564054 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 23:33:45.564216 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260926 23:33:45.564688 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:45.556173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57741 (local address 127.0.97.2:33129)
0926 23:33:45.556384 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:45.556390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:45.557214 (+ 824us) server_negotiation.cc:407] Connection header received
0926 23:33:45.557549 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:45.557556 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:45.557648 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:45.557709 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:33:45.558652 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.559480 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.562368 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.563597 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.563737 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:45.563845 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:45.564233 (+ 388us) server_negotiation.cc:299] Negotiation successful
0926 23:33:45.564398 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 23:33:45.565539 13837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:45.555656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33129 (local address 127.0.97.1:57741)
0926 23:33:45.556975 (+ 1319us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:45.557071 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0926 23:33:45.557320 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:33:45.557945 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:33:45.557949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:33:45.557992 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:33:45.558461 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:45.558472 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.559647 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.559654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:45.562156 (+ 2502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:45.562182 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.563854 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.563860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:45.564032 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:45.564071 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 23:33:45.564920 (+ 849us) client_negotiation.cc:240] Negotiation successful
0926 23:33:45.565087 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1139,"mutex_wait_us":32,"thread_start_us":1278,"threads_started":1}
I20260926 23:33:45.565917 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:45.555726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42817 (local address 127.0.97.1:36533)
0926 23:33:45.557264 (+ 1538us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:45.557287 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:33:45.557404 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:33:45.557776 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:33:45.557781 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:33:45.557810 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:33:45.558396 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:45.558411 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.559984 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.559990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:45.562178 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:45.562189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.563619 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.563626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:45.563844 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:45.563887 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 23:33:45.564008 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 23:33:45.564195 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1420,"mutex_wait_us":9,"thread_start_us":770,"threads_started":1}
I20260926 23:33:45.567001 13619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a44721a6ed34f63b867bde7aea54b75" candidate_uuid: "2464f9c353ea452484ef7117c52f0135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da58a0ba41b4aeab6a5d41963a89006" is_pre_election: true
I20260926 23:33:45.567221 13754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a44721a6ed34f63b867bde7aea54b75" candidate_uuid: "2464f9c353ea452484ef7117c52f0135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc881ede23e848e2bfd22a918a751e0f" is_pre_election: true
I20260926 23:33:45.567782 13619 raft_consensus.cc:2454] T 5a44721a6ed34f63b867bde7aea54b75 P 4da58a0ba41b4aeab6a5d41963a89006 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2464f9c353ea452484ef7117c52f0135 in term 0.
I20260926 23:33:45.567875 13754 raft_consensus.cc:2454] T 5a44721a6ed34f63b867bde7aea54b75 P cc881ede23e848e2bfd22a918a751e0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2464f9c353ea452484ef7117c52f0135 in term 0.
I20260926 23:33:45.568851 13409 leader_election.cc:302] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2464f9c353ea452484ef7117c52f0135, 4da58a0ba41b4aeab6a5d41963a89006, cc881ede23e848e2bfd22a918a751e0f; no voters:
I20260926 23:33:45.569463 13835 raft_consensus.cc:2781] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:33:45.569646 13835 raft_consensus.cc:482] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:33:45.569743 13835 raft_consensus.cc:3037] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:33:45.573673 13835 raft_consensus.cc:504] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2464f9c353ea452484ef7117c52f0135" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40705 } } peers { permanent_uuid: "cc881ede23e848e2bfd22a918a751e0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } } peers { permanent_uuid: "4da58a0ba41b4aeab6a5d41963a89006" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33129 } }
I20260926 23:33:45.575477 13835 leader_election.cc:288] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135 [CANDIDATE]: Term 1 election: Requested vote from peers cc881ede23e848e2bfd22a918a751e0f (127.0.97.3:42817), 4da58a0ba41b4aeab6a5d41963a89006 (127.0.97.2:33129)
I20260926 23:33:45.575474 13754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a44721a6ed34f63b867bde7aea54b75" candidate_uuid: "2464f9c353ea452484ef7117c52f0135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc881ede23e848e2bfd22a918a751e0f"
I20260926 23:33:45.575753 13754 raft_consensus.cc:3037] T 5a44721a6ed34f63b867bde7aea54b75 P cc881ede23e848e2bfd22a918a751e0f [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:33:45.576380 13619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a44721a6ed34f63b867bde7aea54b75" candidate_uuid: "2464f9c353ea452484ef7117c52f0135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da58a0ba41b4aeab6a5d41963a89006"
I20260926 23:33:45.576647 13619 raft_consensus.cc:3037] T 5a44721a6ed34f63b867bde7aea54b75 P 4da58a0ba41b4aeab6a5d41963a89006 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:33:45.579494 13754 raft_consensus.cc:2454] T 5a44721a6ed34f63b867bde7aea54b75 P cc881ede23e848e2bfd22a918a751e0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2464f9c353ea452484ef7117c52f0135 in term 1.
I20260926 23:33:45.580281 13407 leader_election.cc:302] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2464f9c353ea452484ef7117c52f0135, cc881ede23e848e2bfd22a918a751e0f; no voters:
I20260926 23:33:45.581043 13619 raft_consensus.cc:2454] T 5a44721a6ed34f63b867bde7aea54b75 P 4da58a0ba41b4aeab6a5d41963a89006 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2464f9c353ea452484ef7117c52f0135 in term 1.
I20260926 23:33:45.582360 13838 raft_consensus.cc:2781] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:33:45.582628 13838 raft_consensus.cc:686] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135 [term 1 LEADER]: Becoming Leader. State: Replica: 2464f9c353ea452484ef7117c52f0135, State: Running, Role: LEADER
I20260926 23:33:45.583498 13838 consensus_queue.cc:237] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2464f9c353ea452484ef7117c52f0135" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40705 } } peers { permanent_uuid: "cc881ede23e848e2bfd22a918a751e0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } } peers { permanent_uuid: "4da58a0ba41b4aeab6a5d41963a89006" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33129 } }
I20260926 23:33:45.590452 13322 catalog_manager.cc:5495] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2464f9c353ea452484ef7117c52f0135 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "2464f9c353ea452484ef7117c52f0135" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2464f9c353ea452484ef7117c52f0135" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc881ede23e848e2bfd22a918a751e0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da58a0ba41b4aeab6a5d41963a89006" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33129 } health_report { overall_health: UNKNOWN } } }
I20260926 23:33:45.629590 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:45.622138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34830 (local address 127.0.97.1:40705)
0926 23:33:45.622364 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:45.622371 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:45.622395 (+ 24us) server_negotiation.cc:407] Connection header received
0926 23:33:45.622650 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:45.622656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:45.622831 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:45.623013 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:33:45.623477 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.624657 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.626441 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.627178 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.627319 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:45.627971 (+ 652us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:33:45.628871 (+ 900us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:33:45.629038 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:45.629162 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 23:33:45.629276 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 23:33:45.640867 13619 raft_consensus.cc:1261] T 5a44721a6ed34f63b867bde7aea54b75 P 4da58a0ba41b4aeab6a5d41963a89006 [term 1 FOLLOWER]: Refusing update from remote peer 2464f9c353ea452484ef7117c52f0135: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:33:45.640903 13754 raft_consensus.cc:1261] T 5a44721a6ed34f63b867bde7aea54b75 P cc881ede23e848e2bfd22a918a751e0f [term 1 FOLLOWER]: Refusing update from remote peer 2464f9c353ea452484ef7117c52f0135: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:33:45.642293 13838 consensus_queue.cc:1035] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da58a0ba41b4aeab6a5d41963a89006" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:33:45.642709 13835 consensus_queue.cc:1035] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc881ede23e848e2bfd22a918a751e0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:33:45.662190 13841 mvcc.cc:205] Tried to move back new op lower bound from 7333747202610688000 to 7333747202396565504. Current Snapshot: MvccSnapshot[applied={T|T < 7333747202610688000}]
I20260926 23:33:45.666018 13843 mvcc.cc:205] Tried to move back new op lower bound from 7333747202610688000 to 7333747202396565504. Current Snapshot: MvccSnapshot[applied={T|T < 7333747202610688000}]
I20260926 23:33:45.698170 13322 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:33:45.698565 13322 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:33:45.704027 13322 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef765b2d56bc4b1a9a19dd5f57fb7951: Sending DeleteTablet for 3 replicas of tablet 5a44721a6ed34f63b867bde7aea54b75
I20260926 23:33:45.706327 13455 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a44721a6ed34f63b867bde7aea54b75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:33:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34820
I20260926 23:33:45.707134 13733 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a44721a6ed34f63b867bde7aea54b75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:33:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39822
I20260926 23:33:45.712268 13599 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a44721a6ed34f63b867bde7aea54b75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:33:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34480
I20260926 23:33:45.712852 13852 tablet_replica.cc:330] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135: stopping tablet replica
I20260926 23:33:45.713537 13852 raft_consensus.cc:2229] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:33:45.714342 13851 tablet_replica.cc:330] T 5a44721a6ed34f63b867bde7aea54b75 P cc881ede23e848e2bfd22a918a751e0f: stopping tablet replica
I20260926 23:33:45.715317 13852 raft_consensus.cc:2258] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:33:45.715988 13851 raft_consensus.cc:2229] T 5a44721a6ed34f63b867bde7aea54b75 P cc881ede23e848e2bfd22a918a751e0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:33:45.716660 13851 raft_consensus.cc:2258] T 5a44721a6ed34f63b867bde7aea54b75 P cc881ede23e848e2bfd22a918a751e0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:33:45.718248 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 13360
I20260926 23:33:45.725297 13852 ts_tablet_manager.cc:1905] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:33:45.732930 13853 tablet_replica.cc:330] T 5a44721a6ed34f63b867bde7aea54b75 P 4da58a0ba41b4aeab6a5d41963a89006: stopping tablet replica
I20260926 23:33:45.733788 13853 raft_consensus.cc:2229] T 5a44721a6ed34f63b867bde7aea54b75 P 4da58a0ba41b4aeab6a5d41963a89006 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:33:45.734529 13853 raft_consensus.cc:2258] T 5a44721a6ed34f63b867bde7aea54b75 P 4da58a0ba41b4aeab6a5d41963a89006 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:33:45.741030 13852 ts_tablet_manager.cc:1918] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:33:45.741353 13852 log.cc:1196] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-0/wal/wals/5a44721a6ed34f63b867bde7aea54b75
I20260926 23:33:45.742007 13852 ts_tablet_manager.cc:1939] T 5a44721a6ed34f63b867bde7aea54b75 P 2464f9c353ea452484ef7117c52f0135: Deleting consensus metadata
I20260926 23:33:45.742080 13853 ts_tablet_manager.cc:1905] T 5a44721a6ed34f63b867bde7aea54b75 P 4da58a0ba41b4aeab6a5d41963a89006: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:33:45.742080 13851 ts_tablet_manager.cc:1905] T 5a44721a6ed34f63b867bde7aea54b75 P cc881ede23e848e2bfd22a918a751e0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:33:45.744757 13308 catalog_manager.cc:4841] TS 2464f9c353ea452484ef7117c52f0135 (127.0.97.1:40705): tablet 5a44721a6ed34f63b867bde7aea54b75 (table test-table [id=abb70f5644f44759b0d005030acf1de2]) successfully deleted
I20260926 23:33:45.750015 13853 ts_tablet_manager.cc:1918] T 5a44721a6ed34f63b867bde7aea54b75 P 4da58a0ba41b4aeab6a5d41963a89006: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:33:45.750299 13853 log.cc:1196] T 5a44721a6ed34f63b867bde7aea54b75 P 4da58a0ba41b4aeab6a5d41963a89006: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-1/wal/wals/5a44721a6ed34f63b867bde7aea54b75
I20260926 23:33:45.751124 13853 ts_tablet_manager.cc:1939] T 5a44721a6ed34f63b867bde7aea54b75 P 4da58a0ba41b4aeab6a5d41963a89006: Deleting consensus metadata
I20260926 23:33:45.751680 13851 ts_tablet_manager.cc:1918] T 5a44721a6ed34f63b867bde7aea54b75 P cc881ede23e848e2bfd22a918a751e0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:33:45.751881 13851 log.cc:1196] T 5a44721a6ed34f63b867bde7aea54b75 P cc881ede23e848e2bfd22a918a751e0f: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790465400694755-388-0/minicluster-data/ts-2/wal/wals/5a44721a6ed34f63b867bde7aea54b75
I20260926 23:33:45.753819 13851 ts_tablet_manager.cc:1939] T 5a44721a6ed34f63b867bde7aea54b75 P cc881ede23e848e2bfd22a918a751e0f: Deleting consensus metadata
I20260926 23:33:45.753979 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:45.720280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34844 (local address 127.0.97.1:40705)
0926 23:33:45.721025 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:45.721033 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:45.721054 (+ 21us) server_negotiation.cc:407] Connection header received
0926 23:33:45.721136 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:45.721143 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:45.721252 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:45.721366 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:45.725778 (+ 4412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.727199 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.729327 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.730123 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.730267 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:45.737079 (+ 6812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:45.737099 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:45.737123 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:45.737185 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:45.746376 (+ 9191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:45.750367 (+ 3991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:45.750477 (+ 110us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:45.750481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:45.750760 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:45.751407 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:45.751412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:45.751415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:45.752210 (+ 795us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:45.752641 (+ 431us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:45.753160 (+ 519us) server_negotiation.cc:299] Negotiation successful
0926 23:33:45.753462 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588}
I20260926 23:33:45.753988 13309 catalog_manager.cc:4841] TS 4da58a0ba41b4aeab6a5d41963a89006 (127.0.97.2:33129): tablet 5a44721a6ed34f63b867bde7aea54b75 (table test-table [id=abb70f5644f44759b0d005030acf1de2]) successfully deleted
I20260926 23:33:45.755288 13515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:33:45.756282 13307 catalog_manager.cc:4841] TS cc881ede23e848e2bfd22a918a751e0f (127.0.97.3:42817): tablet 5a44721a6ed34f63b867bde7aea54b75 (table test-table [id=abb70f5644f44759b0d005030acf1de2]) successfully deleted
I20260926 23:33:45.867791 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 13360
I20260926 23:33:45.885820 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 13524
I20260926 23:33:45.902487 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:45.886836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34484 (local address 127.0.97.2:33129)
0926 23:33:45.887049 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:45.887056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:45.887079 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:33:45.887338 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:45.887345 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:45.887451 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:45.887543 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:45.888999 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.889831 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.891585 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:45.892044 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:45.892169 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:45.895190 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:45.895210 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:45.895233 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:45.895283 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:45.898971 (+ 3688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:45.900021 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:45.900030 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:45.900033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:45.900213 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:45.900606 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:45.900610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:45.900612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:45.901081 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:45.901452 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:45.901791 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 23:33:45.902073 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 23:33:45.903406 13659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:33:46.001660 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 13524
I20260926 23:33:46.019116 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 13669
I20260926 23:33:46.037261 13823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:46.020357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39838 (local address 127.0.97.3:42817)
0926 23:33:46.020588 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:46.020593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:46.020618 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:33:46.020731 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:46.020736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:46.020824 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:46.020928 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:46.022739 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:46.023633 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:46.025153 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:46.025707 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:46.025845 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:46.029151 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:46.029225 (+ 74us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:46.029248 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:46.029304 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:46.032738 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:46.033708 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:46.033721 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:46.033725 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:46.033921 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:46.034455 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:46.034458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:46.034460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:46.035828 (+ 1368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:46.036205 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:46.036620 (+ 415us) server_negotiation.cc:299] Negotiation successful
0926 23:33:46.036890 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 23:33:46.038203 13794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:33:46.135536 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 13669
I20260926 23:33:46.151016 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 13290
I20260926 23:33:46.152424 13353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:33:46.278776 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 13290
I20260926 23:33:46.304239 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 23:33:46.641417 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 23:33:46.641667 388 mini_ranger.cc:66] Stopping Ranger...
I20260926 23:33:46.989030 388 mini_ranger.cc:68] Stopped Ranger
2026-09-26 23:33:46.989 UTC [12341] LOG: received smart shutdown request
2026-09-26 23:33:46.993 UTC [12341] LOG: background worker "logical replication launcher" (PID 12350) exited with exit code 1
2026-09-26 23:33:46.993 UTC [12345] LOG: shutting down
2026-09-26 23:33:47.206 UTC [12341] LOG: database system is shut down
2026-09-26T23:33:47Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (61518 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:33:47 dist-test-slave-gwfx krb5kdc[13866](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:33:47 dist-test-slave-gwfx krb5kdc[13866](info): set up 2 sockets
Sep 26 23:33:47 dist-test-slave-gwfx krb5kdc[13866](info): commencing operation
krb5kdc: starting...
W20260926 23:33:48.359375 388 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.010s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:33:48 dist-test-slave-gwfx krb5kdc[13866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:33:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:33:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:33:48.634733 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36313
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36313,127.0.97.61:35887,127.0.97.60:33859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:33:49.043411 13879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:33:49.043849 13879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:33:49.043991 13879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:33:49.053942 13879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:33:49.054098 13879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:33:49.054158 13879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:33:49.054255 13879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:33:49.054317 13879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:33:49.066598 13879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36313,127.0.97.61:35887,127.0.97.60:33859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:33:49.068521 13879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:33:49.070312 13879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:33:49.085645 13884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:33:49.085507 13885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:33:49.085507 13887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:33:49.086544 13879 server_base.cc:1034] running on GCE node
I20260926 23:33:49.087610 13879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:33:49.089577 13879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:33:49.090885 13879 hybrid_clock.cc:648] HybridClock initialized: now 1790465629090818 us; error 52 us; skew 500 ppm
I20260926 23:33:49.091385 13879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:33:49 dist-test-slave-gwfx krb5kdc[13866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:33:49.100628 13879 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:33:49.102869 13879 webserver.cc:460] Webserver started at http://127.0.97.62:45039/ using document root <none> and password file <none>
I20260926 23:33:49.103777 13879 fs_manager.cc:359] Metadata directory not provided
I20260926 23:33:49.103927 13879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:33:49.104310 13879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:33:49.108117 13879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "58b48d8fb2104fe2b82b3ea1565d2820"
format_stamp: "Formatted at 2026-09-26 23:33:49 on dist-test-slave-gwfx"
server_key: "61c164c480ee0b7d99b8ed1a8eab634f"
server_key_iv: "f8870246815d2e1bee2cebc8048ab291"
server_key_version: "encryptionkey@0"
I20260926 23:33:49.109030 13879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "58b48d8fb2104fe2b82b3ea1565d2820"
format_stamp: "Formatted at 2026-09-26 23:33:49 on dist-test-slave-gwfx"
server_key: "61c164c480ee0b7d99b8ed1a8eab634f"
server_key_iv: "f8870246815d2e1bee2cebc8048ab291"
server_key_version: "encryptionkey@0"
I20260926 23:33:49.115388 13879 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 23:33:49.120005 13894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:33:49.121968 13879 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260926 23:33:49.122191 13879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "58b48d8fb2104fe2b82b3ea1565d2820"
format_stamp: "Formatted at 2026-09-26 23:33:49 on dist-test-slave-gwfx"
server_key: "61c164c480ee0b7d99b8ed1a8eab634f"
server_key_iv: "f8870246815d2e1bee2cebc8048ab291"
server_key_version: "encryptionkey@0"
I20260926 23:33:49.122452 13879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:33:49.145942 13879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:33:49.151801 13879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:33:49.152366 13879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:33:49.173655 13879 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36313
I20260926 23:33:49.173678 13945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36313 every 8 connection(s)
I20260926 23:33:49.175424 13879 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:33:49.180550 13946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:33:49.184765 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 13879
I20260926 23:33:49.185050 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:33:49.185968 388 external_mini_cluster.cc:1428] Setting key 4beb4eeeaac42157b392c730a4814965
I20260926 23:33:49.191094 13946 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36313 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:33:49 dist-test-slave-gwfx krb5kdc[13866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260926 23:33:49.222842 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:49.196003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36313 (local address 127.0.0.1:54370)
0926 23:33:49.197227 (+ 1224us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:49.197290 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 23:33:49.200013 (+ 2723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:33:49.201235 (+ 1222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:33:49.201258 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:33:49.202262 (+ 1004us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:33:49.202869 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:49.202904 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:49.204798 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:49.204804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:49.206752 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:49.206780 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:49.208087 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:49.208095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:49.208293 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:49.209157 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:33:49.209194 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:33:49.212851 (+ 3657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:33:49.217231 (+ 4380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:49.217245 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:49.217258 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:49.217966 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:49.218631 (+ 665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:49.218635 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:49.218638 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:49.218868 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:49.220252 (+ 1384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:33:49.220260 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:33:49.220532 (+ 272us) client_negotiation.cc:769] Sending connection context
0926 23:33:49.220811 (+ 279us) client_negotiation.cc:240] Negotiation successful
0926 23:33:49.221372 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"thread_start_us":577,"threads_started":1}
I20260926 23:33:49.222874 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:49.196003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54370 (local address 127.0.97.62:36313)
0926 23:33:49.197357 (+ 1354us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:49.197367 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:49.197500 (+ 133us) server_negotiation.cc:407] Connection header received
0926 23:33:49.200584 (+ 3084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:49.200592 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:49.200956 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:49.201222 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:49.203143 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:49.204606 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:49.206938 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:49.207784 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:49.207963 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:49.213300 (+ 5337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:49.213313 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:49.213333 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:49.213401 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:49.217014 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:49.218202 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:49.218229 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:49.218233 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:49.218492 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:49.218984 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:49.218989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:49.218991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:49.219992 (+ 1001us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:49.220316 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:49.221005 (+ 689us) server_negotiation.cc:299] Negotiation successful
0926 23:33:49.221573 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":193,"server-negotiator.queue_time_us":1106,"spinlock_wait_cycles":94080,"thread_start_us":859,"threads_started":1}
I20260926 23:33:49.226930 13946 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35887 } has no permanent_uuid. Determining permanent_uuid...
I20260926 23:33:49.228487 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:49.227885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35887 (local address 127.0.0.1:38416)
0926 23:33:49.228144 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:49.228307 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260926 23:33:49.228786 13897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35887: connect: Connection refused (error 111)
I20260926 23:33:49.230863 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:49.230353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35887 (local address 127.0.0.1:38420)
0926 23:33:49.230528 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:49.230632 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 23:33:49.231060 13946 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:35887: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35887: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260926 23:33:49.254101 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:35887
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36313,127.0.97.61:35887,127.0.97.60:33859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:33:49.280535 13946 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35887 } attempt: 1
I20260926 23:33:49.282073 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:49.281487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35887 (local address 127.0.0.1:38432)
0926 23:33:49.281745 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:49.281872 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 23:33:49.283421 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:49.282915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35887 (local address 127.0.0.1:38438)
0926 23:33:49.283114 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:49.283206 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 23:33:49.283655 13946 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:35887: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35887: connect: Connection refused (error 111)
I20260926 23:33:49.351974 13946 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35887 } attempt: 2
I20260926 23:33:49.353389 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:49.352850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35887 (local address 127.0.0.1:38450)
0926 23:33:49.353092 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:49.353207 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 23:33:49.354660 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:49.354060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35887 (local address 127.0.0.1:38456)
0926 23:33:49.354316 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:49.354445 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260926 23:33:49.354943 13946 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:35887: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35887: connect: Connection refused (error 111)
I20260926 23:33:49.446368 13946 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35887 } attempt: 3
I20260926 23:33:49.447932 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:49.447288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35887 (local address 127.0.0.1:38464)
0926 23:33:49.447576 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:49.447707 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260926 23:33:49.449249 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:49.448761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35887 (local address 127.0.0.1:38474)
0926 23:33:49.448935 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:49.449035 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 23:33:49.449520 13946 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:35887: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35887: connect: Connection refused (error 111)
I20260926 23:33:49.592823 13946 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35887 } attempt: 4
I20260926 23:33:49.594538 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:49.593777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35887 (local address 127.0.0.1:38476)
0926 23:33:49.594149 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:49.594319 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191}
I20260926 23:33:49.595980 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:49.595540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35887 (local address 127.0.0.1:38478)
0926 23:33:49.595702 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:49.595821 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 23:33:49.596213 13946 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:35887: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35887: connect: Connection refused (error 111)
W20260926 23:33:49.659339 13953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:33:49.659744 13953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:33:49.659895 13953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:33:49.668215 13953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:33:49.668383 13953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:33:49.668440 13953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:33:49.668529 13953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:33:49.668587 13953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:33:49.680161 13953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36313,127.0.97.61:35887,127.0.97.60:33859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:35887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:33:49.682097 13953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:33:49.683841 13953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:33:49.698545 13961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:33:49.698581 13959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:33:49.698608 13958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:33:49.699582 13953 server_base.cc:1034] running on GCE node
I20260926 23:33:49.700727 13953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:33:49.702708 13953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:33:49.703963 13953 hybrid_clock.cc:648] HybridClock initialized: now 1790465629703896 us; error 60 us; skew 500 ppm
I20260926 23:33:49.704406 13953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:33:49 dist-test-slave-gwfx krb5kdc[13866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:33:49.708532 13953 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260926 23:33:49.710527 13953 webserver.cc:460] Webserver started at http://127.0.97.61:45317/ using document root <none> and password file <none>
I20260926 23:33:49.711493 13953 fs_manager.cc:359] Metadata directory not provided
I20260926 23:33:49.711637 13953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:33:49.712004 13953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:33:49.715925 13953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-1/data/instance:
uuid: "5697e87853b4414db9727f49a1e4d2bf"
format_stamp: "Formatted at 2026-09-26 23:33:49 on dist-test-slave-gwfx"
server_key: "d964b2e64a18ca5f0d5b45172c287250"
server_key_iv: "925cf57aaf0e89867169a22307d0401b"
server_key_version: "encryptionkey@0"
I20260926 23:33:49.716921 13953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-1/wal/instance:
uuid: "5697e87853b4414db9727f49a1e4d2bf"
format_stamp: "Formatted at 2026-09-26 23:33:49 on dist-test-slave-gwfx"
server_key: "d964b2e64a18ca5f0d5b45172c287250"
server_key_iv: "925cf57aaf0e89867169a22307d0401b"
server_key_version: "encryptionkey@0"
I20260926 23:33:49.724365 13953 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260926 23:33:49.729475 13968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:33:49.731542 13953 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 23:33:49.731787 13953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-1/wal
uuid: "5697e87853b4414db9727f49a1e4d2bf"
format_stamp: "Formatted at 2026-09-26 23:33:49 on dist-test-slave-gwfx"
server_key: "d964b2e64a18ca5f0d5b45172c287250"
server_key_iv: "925cf57aaf0e89867169a22307d0401b"
server_key_version: "encryptionkey@0"
I20260926 23:33:49.732046 13953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:33:49.758579 13953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:33:49.764503 13953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:33:49.764843 13953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:33:49.786345 13953 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:35887
I20260926 23:33:49.786414 14019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:35887 every 8 connection(s)
I20260926 23:33:49.788251 13953 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-1/data/info.pb
I20260926 23:33:49.793989 14020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:33:49.794762 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 13953
I20260926 23:33:49.794999 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-1/wal/instance
I20260926 23:33:49.795514 388 external_mini_cluster.cc:1428] Setting key f34e98cc6032e07527716f3d0602587a
I20260926 23:33:49.803843 14020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36313 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 26 23:33:49 dist-test-slave-gwfx krb5kdc[13866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:33:49.836802 14023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:49.808552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54386 (local address 127.0.97.62:36313)
0926 23:33:49.809351 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:49.809356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:49.811010 (+ 1654us) server_negotiation.cc:407] Connection header received
0926 23:33:49.814348 (+ 3338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:49.814355 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:49.814605 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:49.814816 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:49.817485 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:49.818486 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:49.821774 (+ 3288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:49.822294 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:49.822447 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:49.827914 (+ 5467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:49.827924 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:49.827929 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:49.828009 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:49.832053 (+ 4044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:49.833354 (+ 1301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:49.833360 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:49.833364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:49.833567 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:49.834158 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:49.834164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:49.834167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:49.834611 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:49.834831 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:49.836057 (+ 1226us) server_negotiation.cc:299] Negotiation successful
0926 23:33:49.836355 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":330,"threads_started":1}
I20260926 23:33:49.838371 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:49.809438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36313 (local address 127.0.0.1:54386)
0926 23:33:49.810609 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:49.810801 (+ 192us) client_negotiation.cc:174] Beginning negotiation
0926 23:33:49.813786 (+ 2985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:33:49.815371 (+ 1585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:33:49.815386 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:33:49.816223 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:33:49.817165 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:49.817195 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:49.818888 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:49.818896 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:49.821418 (+ 2522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:49.821451 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:49.822653 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:49.822665 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:49.823277 (+ 612us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:49.824358 (+ 1081us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:33:49.824401 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:33:49.827492 (+ 3091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:33:49.832375 (+ 4883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:49.832386 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:49.832419 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:49.833159 (+ 740us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:49.833753 (+ 594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:49.833758 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:49.833760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:49.834028 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:49.834861 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:33:49.834873 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:33:49.835529 (+ 656us) client_negotiation.cc:769] Sending connection context
0926 23:33:49.835909 (+ 380us) client_negotiation.cc:240] Negotiation successful
0926 23:33:49.836650 (+ 741us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"thread_start_us":523,"threads_started":1}
I20260926 23:33:49.840694 14020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35887 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:33:49 dist-test-slave-gwfx krb5kdc[13866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 23:33:49.859727 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:49.841982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35887 (local address 127.0.0.1:38480)
0926 23:33:49.842254 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:49.842277 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:33:49.842423 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:33:49.843701 (+ 1278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:33:49.843706 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:33:49.844277 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:33:49.844480 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:49.844534 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:49.846210 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:49.846215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:49.847569 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:49.847589 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:49.848660 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:49.848666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:49.848888 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:49.849643 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:33:49.849670 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:33:49.852534 (+ 2864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:33:49.856413 (+ 3879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:49.856418 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:49.856422 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:49.856960 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:49.857500 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:49.857505 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:49.857508 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:49.857701 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:49.858540 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:33:49.858544 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:33:49.858755 (+ 211us) client_negotiation.cc:769] Sending connection context
0926 23:33:49.858897 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 23:33:49.859245 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 23:33:49.859786 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:49.842283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38480 (local address 127.0.97.61:35887)
0926 23:33:49.843119 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:49.843128 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:49.843162 (+ 34us) server_negotiation.cc:407] Connection header received
0926 23:33:49.843225 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:49.843232 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:49.843502 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:49.843676 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:49.844729 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:49.846009 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:49.847769 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:49.848470 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:49.848590 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:49.852860 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:49.852881 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:49.852904 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:49.852942 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:49.856208 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:49.857143 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:49.857153 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:49.857156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:49.857318 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:49.857817 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:49.857821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:49.857823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:49.858392 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:49.858599 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:49.859084 (+ 485us) server_negotiation.cc:299] Negotiation successful
0926 23:33:49.859390 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":650,"thread_start_us":493,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260926 23:33:49.863181 14020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 } has no permanent_uuid. Determining permanent_uuid...
I20260926 23:33:49.864912 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:49.864277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33859 (local address 127.0.0.1:41700)
0926 23:33:49.864506 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:49.864672 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 23:33:49.864879 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33859
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36313,127.0.97.61:35887,127.0.97.60:33859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:33:49.865206 13971 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33859: connect: Connection refused (error 111)
I20260926 23:33:49.867350 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:49.866903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33859 (local address 127.0.0.1:41708)
0926 23:33:49.867105 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:49.867199 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 23:33:49.867599 14020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33859: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33859: connect: Connection refused (error 111)
I20260926 23:33:49.895478 13946 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35887 } attempt: 5
Sep 26 23:33:49 dist-test-slave-gwfx krb5kdc[13866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 23:33:49.911641 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:49.896377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35887 (local address 127.0.0.1:38486)
0926 23:33:49.896619 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:49.896643 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:33:49.896818 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:33:49.897205 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:33:49.897211 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:33:49.897716 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:33:49.897941 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:49.897953 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:49.899081 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:49.899085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:49.900067 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:49.900081 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:49.900893 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:49.900900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:49.901099 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:49.901817 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:33:49.901839 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:33:49.904511 (+ 2672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:33:49.908705 (+ 4194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:49.908710 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:49.908714 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:49.909339 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:49.909792 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:49.909796 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:49.909798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:49.909953 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:49.910629 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:33:49.910635 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:33:49.910831 (+ 196us) client_negotiation.cc:769] Sending connection context
0926 23:33:49.910972 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 23:33:49.911320 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 23:33:49.911697 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:49.896435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38486 (local address 127.0.97.61:35887)
0926 23:33:49.896632 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:49.896640 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:49.896723 (+ 83us) server_negotiation.cc:407] Connection header received
0926 23:33:49.896975 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:49.896980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:49.897059 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:49.897153 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:49.898118 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:49.898927 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:49.900220 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:49.900720 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:49.900849 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:49.904818 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:49.904827 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:49.904832 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:49.904884 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:49.908476 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:49.909481 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:49.909488 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:49.909491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:49.909671 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:49.910050 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:49.910055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:49.910058 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:49.910493 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:49.910678 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:49.911115 (+ 437us) server_negotiation.cc:299] Negotiation successful
0926 23:33:49.911378 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 23:33:49.913008 13946 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 } has no permanent_uuid. Determining permanent_uuid...
I20260926 23:33:49.914302 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:49.913831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33859 (local address 127.0.0.1:41722)
0926 23:33:49.914016 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:49.914123 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 23:33:49.915521 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:49.915121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33859 (local address 127.0.0.1:41726)
0926 23:33:49.915300 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:49.915388 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260926 23:33:49.915743 13946 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33859: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33859: connect: Connection refused (error 111)
I20260926 23:33:49.916958 14020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 } attempt: 1
I20260926 23:33:49.918077 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:49.917616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33859 (local address 127.0.0.1:41732)
0926 23:33:49.917807 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:49.917921 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 23:33:49.919298 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:49.918871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33859 (local address 127.0.0.1:41744)
0926 23:33:49.919039 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:49.919122 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 23:33:49.919514 14020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33859: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33859: connect: Connection refused (error 111)
I20260926 23:33:49.967010 13946 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 } attempt: 1
I20260926 23:33:49.968559 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:49.967959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33859 (local address 127.0.0.1:41752)
0926 23:33:49.968212 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:49.968330 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 23:33:49.969756 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:49.969340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33859 (local address 127.0.0.1:41758)
0926 23:33:49.969510 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:49.969612 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 23:33:49.970012 13946 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33859: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33859: connect: Connection refused (error 111)
I20260926 23:33:49.987747 14020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 } attempt: 2
I20260926 23:33:49.989382 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:49.988773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33859 (local address 127.0.0.1:41774)
0926 23:33:49.989012 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:49.989124 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 23:33:49.990592 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:49.990188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33859 (local address 127.0.0.1:41784)
0926 23:33:49.990365 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:49.990447 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 23:33:49.990854 14020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33859: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33859: connect: Connection refused (error 111)
I20260926 23:33:50.038264 13946 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 } attempt: 2
I20260926 23:33:50.039702 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:50.039126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33859 (local address 127.0.0.1:41796)
0926 23:33:50.039390 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:50.039501 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 23:33:50.040803 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:50.040409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33859 (local address 127.0.0.1:41804)
0926 23:33:50.040586 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:50.040669 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 23:33:50.041020 13946 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33859: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33859: connect: Connection refused (error 111)
I20260926 23:33:50.082093 14020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 } attempt: 3
I20260926 23:33:50.083601 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:50.083001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33859 (local address 127.0.0.1:41814)
0926 23:33:50.083281 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:50.083392 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 23:33:50.085094 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:50.084525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33859 (local address 127.0.0.1:41816)
0926 23:33:50.084834 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:50.084936 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
W20260926 23:33:50.085325 14020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33859: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33859: connect: Connection refused (error 111)
I20260926 23:33:50.147302 13946 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 } attempt: 3
I20260926 23:33:50.148876 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:50.148249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33859 (local address 127.0.0.1:41822)
0926 23:33:50.148518 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:50.148641 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 23:33:50.150259 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:50.149793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33859 (local address 127.0.0.1:41826)
0926 23:33:50.149990 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:50.150107 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 23:33:50.150511 13946 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33859: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33859: connect: Connection refused (error 111)
I20260926 23:33:50.228564 14020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 } attempt: 4
I20260926 23:33:50.229952 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:50.229415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33859 (local address 127.0.0.1:41832)
0926 23:33:50.229641 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:50.229741 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 23:33:50.231328 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:50.230894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33859 (local address 127.0.0.1:41836)
0926 23:33:50.231054 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:50.231141 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 23:33:50.231616 14020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33859: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33859: connect: Connection refused (error 111)
W20260926 23:33:50.276624 14028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:33:50.277071 14028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:33:50.277205 14028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:33:50.285710 14028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:33:50.285887 14028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:33:50.285979 14028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:33:50.286062 14028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:33:50.286123 14028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:33:50.298033 14028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36313,127.0.97.61:35887,127.0.97.60:33859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:33:50.299921 14028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:33:50.301698 14028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:33:50.317330 14036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:33:50.317373 14033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:33:50.317337 14034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:33:50.318084 14028 server_base.cc:1034] running on GCE node
I20260926 23:33:50.319434 14028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:33:50.321677 14028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:33:50.323091 14028 hybrid_clock.cc:648] HybridClock initialized: now 1790465630323005 us; error 95 us; skew 500 ppm
I20260926 23:33:50.323740 14028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:33:50 dist-test-slave-gwfx krb5kdc[13866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:33:50.327759 13946 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 } attempt: 4
I20260926 23:33:50.328661 14028 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260926 23:33:50.329138 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:50.328531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33859 (local address 127.0.0.1:41846)
0926 23:33:50.328799 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:50.328922 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260926 23:33:50.330613 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:50.330044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33859 (local address 127.0.0.1:41854)
0926 23:33:50.330245 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:50.330441 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260926 23:33:50.331075 13946 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33859: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33859: connect: Connection refused (error 111)
I20260926 23:33:50.331579 14028 webserver.cc:460] Webserver started at http://127.0.97.60:46179/ using document root <none> and password file <none>
I20260926 23:33:50.332556 14028 fs_manager.cc:359] Metadata directory not provided
I20260926 23:33:50.332712 14028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:33:50.333105 14028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:33:50.337213 14028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-2/data/instance:
uuid: "cf067451e222442db438e7956a007b7f"
format_stamp: "Formatted at 2026-09-26 23:33:50 on dist-test-slave-gwfx"
server_key: "4dc8352402dc43ece2cb6561d0034e31"
server_key_iv: "5f717e2b15a8425b2d24a39a99d87b85"
server_key_version: "encryptionkey@0"
I20260926 23:33:50.338317 14028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-2/wal/instance:
uuid: "cf067451e222442db438e7956a007b7f"
format_stamp: "Formatted at 2026-09-26 23:33:50 on dist-test-slave-gwfx"
server_key: "4dc8352402dc43ece2cb6561d0034e31"
server_key_iv: "5f717e2b15a8425b2d24a39a99d87b85"
server_key_version: "encryptionkey@0"
I20260926 23:33:50.345281 14028 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 23:33:50.349970 14043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:33:50.351955 14028 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 23:33:50.352202 14028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-2/wal
uuid: "cf067451e222442db438e7956a007b7f"
format_stamp: "Formatted at 2026-09-26 23:33:50 on dist-test-slave-gwfx"
server_key: "4dc8352402dc43ece2cb6561d0034e31"
server_key_iv: "5f717e2b15a8425b2d24a39a99d87b85"
server_key_version: "encryptionkey@0"
I20260926 23:33:50.352469 14028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:33:50.372028 14028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:33:50.377662 14028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:33:50.377998 14028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:33:50.399422 14028 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:33859
I20260926 23:33:50.399467 14094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:33859 every 8 connection(s)
I20260926 23:33:50.401217 14028 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-2/data/info.pb
I20260926 23:33:50.405632 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 14028
I20260926 23:33:50.405860 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790465400694755-388-0/minicluster-data/master-2/wal/instance
I20260926 23:33:50.406351 14095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:33:50.406509 388 external_mini_cluster.cc:1428] Setting key 67e21f0e28f669c6c8e14f4bfa29641b
I20260926 23:33:50.418980 14095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36313 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:33:50 dist-test-slave-gwfx krb5kdc[13866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 23:33:50 dist-test-slave-gwfx krb5kdc[13866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:33:50.435272 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:50.408783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41868 (local address 127.0.97.60:33859)
0926 23:33:50.409728 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:50.409742 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:50.409804 (+ 62us) server_negotiation.cc:407] Connection header received
0926 23:33:50.412006 (+ 2202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:50.412039 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:50.412617 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:50.413085 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:50.415473 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:50.417578 (+ 2105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:50.420040 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:50.421033 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:50.421250 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:50.426823 (+ 5573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:50.426835 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:50.426850 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:50.426897 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:50.430135 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:50.431143 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:50.431152 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:50.431161 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:50.431369 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:50.431805 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:50.431810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:50.431812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:50.432491 (+ 679us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:50.432835 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:50.433360 (+ 525us) server_negotiation.cc:299] Negotiation successful
0926 23:33:50.433900 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":422,"threads_started":1}
I20260926 23:33:50.440925 14100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:50.423313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36313 (local address 127.0.0.1:54402)
0926 23:33:50.424494 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:50.424556 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 23:33:50.424810 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:33:50.425339 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:33:50.425348 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:33:50.426184 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:33:50.426517 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:50.426528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:50.427902 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:50.427907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:50.429255 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:50.429271 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:50.430083 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:50.430090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:50.430262 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:50.430985 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:33:50.431005 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:33:50.433350 (+ 2345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:33:50.437321 (+ 3971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:50.437334 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:50.437342 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:50.438017 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:50.438513 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:50.438518 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:50.438522 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:50.438675 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:50.439491 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:33:50.439502 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:33:50.439799 (+ 297us) client_negotiation.cc:769] Sending connection context
0926 23:33:50.440047 (+ 248us) client_negotiation.cc:240] Negotiation successful
0926 23:33:50.440461 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"thread_start_us":653,"threads_started":1}
I20260926 23:33:50.441025 14099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:50.423313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54402 (local address 127.0.97.62:36313)
0926 23:33:50.424060 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:50.424067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:50.424690 (+ 623us) server_negotiation.cc:407] Connection header received
0926 23:33:50.424970 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:50.424977 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:50.425085 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:50.425225 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:50.426740 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:50.427712 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:50.429418 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:50.429871 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:50.429991 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:50.433595 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:50.433602 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:50.433606 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:50.433653 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:50.437054 (+ 3401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:50.438156 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:50.438161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:50.438164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:50.438387 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:50.438890 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:50.438896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:50.438899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:50.439333 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:50.439496 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:50.440434 (+ 938us) server_negotiation.cc:299] Negotiation successful
0926 23:33:50.440640 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":343,"threads_started":1}
I20260926 23:33:50.443334 14095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35887 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:33:50 dist-test-slave-gwfx krb5kdc[13866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 26 23:33:50 dist-test-slave-gwfx krb5kdc[13866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 23:33:50.458935 14101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:50.439737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54410 (local address 127.0.97.62:36313)
0926 23:33:50.440558 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:50.440564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:50.440589 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:33:50.440641 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:50.440646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:50.440723 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:50.440842 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:50.442443 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:50.443618 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:50.445240 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:50.445700 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:50.445821 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:50.451055 (+ 5234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:50.451062 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:50.451066 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:50.451111 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:50.455283 (+ 4172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:50.456333 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:50.456339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:50.456341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:50.456497 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:50.456958 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:50.456964 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:50.456967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:50.457452 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:50.457813 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:50.458071 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 23:33:50.458368 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":392,"threads_started":1}
I20260926 23:33:50.460386 14100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:50.444157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35887 (local address 127.0.0.1:38496)
0926 23:33:50.444336 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:50.444355 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:33:50.444445 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:33:50.445258 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:33:50.445263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:33:50.445679 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:33:50.445853 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:50.445861 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:50.447148 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:50.447152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:50.448140 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:50.448154 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:50.448966 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:50.448981 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:50.449217 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:50.450117 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:33:50.450149 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:33:50.452904 (+ 2755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:33:50.456857 (+ 3953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:50.456865 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:50.456869 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:50.457706 (+ 837us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:50.458267 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:50.458273 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:50.458277 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:50.458517 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:50.459292 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:33:50.459298 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:33:50.459470 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 23:33:50.459613 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 23:33:50.459990 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 23:33:50.460492 14102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:50.444323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38496 (local address 127.0.97.61:35887)
0926 23:33:50.444903 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:50.444908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:50.444924 (+ 16us) server_negotiation.cc:407] Connection header received
0926 23:33:50.444983 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:50.444987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:50.445051 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:50.445155 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:50.446005 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:50.446993 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:50.448291 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:50.448754 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:50.448898 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:50.453173 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:50.453180 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:50.453184 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:50.453237 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:50.456652 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:50.457875 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:50.457882 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:50.457886 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:50.458134 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:50.458677 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:50.458717 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:50.458720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:50.459127 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:50.459303 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:50.459761 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 23:33:50.460038 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":337,"threads_started":1}
I20260926 23:33:50.461584 14095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:33:50 dist-test-slave-gwfx krb5kdc[13866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 23:33:50 dist-test-slave-gwfx krb5kdc[13866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 23:33:50.478143 14100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:50.462663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33859 (local address 127.0.0.1:41882)
0926 23:33:50.462964 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:50.462987 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:33:50.463115 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:33:50.463486 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:33:50.463490 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:33:50.463987 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:33:50.464142 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:50.464151 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:50.465085 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:50.465091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:50.465976 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:50.465987 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:50.466740 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:50.466746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:50.466927 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:50.467711 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:33:50.467730 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:33:50.470243 (+ 2513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:33:50.474631 (+ 4388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:50.474640 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:50.474646 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:50.475315 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:50.475814 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:50.475821 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:50.475826 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:50.476029 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:50.476896 (+ 867us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:33:50.476901 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:33:50.477090 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 23:33:50.477267 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 23:33:50.477688 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192}
I20260926 23:33:50.479259 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:50.462661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41882 (local address 127.0.97.60:33859)
0926 23:33:50.462857 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:50.462861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:50.463079 (+ 218us) server_negotiation.cc:407] Connection header received
0926 23:33:50.463223 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:50.463227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:50.463345 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:50.463434 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:50.464273 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:50.464968 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:50.466103 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:50.466545 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:50.466649 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:50.470487 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:50.470494 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:50.470497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:50.470533 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:50.474232 (+ 3699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:50.475484 (+ 1252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:50.475491 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:50.475495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:50.475664 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:50.476188 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:50.476193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:50.476196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:50.476671 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:50.476841 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:50.478553 (+ 1712us) server_negotiation.cc:299] Negotiation successful
0926 23:33:50.478892 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 23:33:50.480381 14102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:50.461221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38508 (local address 127.0.97.61:35887)
0926 23:33:50.461391 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:50.461395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:50.461409 (+ 14us) server_negotiation.cc:407] Connection header received
0926 23:33:50.461460 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:50.461465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:50.461542 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:50.461655 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:50.463835 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:50.464635 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:50.466053 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:50.466502 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:50.466621 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:50.471933 (+ 5312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:50.471939 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:50.471942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:50.471988 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:50.476081 (+ 4093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:50.477214 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:50.477221 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:50.477225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:50.477435 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:50.477882 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:50.477888 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:50.477892 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:50.478301 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:50.478598 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:50.479309 (+ 711us) server_negotiation.cc:299] Negotiation successful
0926 23:33:50.479930 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":71}
I20260926 23:33:50.486958 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:33:50.488131 14095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf067451e222442db438e7956a007b7f: Bootstrap starting.
I20260926 23:33:50.493670 14095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf067451e222442db438e7956a007b7f: Neither blocks nor log segments found. Creating new log.
I20260926 23:33:50.496431 14095 log.cc:824] T 00000000000000000000000000000000 P cf067451e222442db438e7956a007b7f: Log is configured to *not* fsync() on all Append() calls
I20260926 23:33:50.511142 14095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf067451e222442db438e7956a007b7f: No bootstrap required, opened a new log
I20260926 23:33:50.518051 14102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:50.494370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38524 (local address 127.0.97.61:35887)
0926 23:33:50.494615 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:50.494622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:50.497183 (+ 2561us) server_negotiation.cc:407] Connection header received
0926 23:33:50.497412 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:50.497418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:50.497530 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:50.497658 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:50.500878 (+ 3220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:50.502369 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:50.505096 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:50.505650 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:50.505810 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:50.510040 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:50.510049 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:50.510054 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:50.510117 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:50.514191 (+ 4074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:50.515304 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:50.515312 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:50.515316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:50.515511 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:50.515956 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:50.515962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:50.515965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:50.516397 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:50.516664 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:50.517337 (+ 673us) server_negotiation.cc:299] Negotiation successful
0926 23:33:50.517632 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 23:33:50.522358 14101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:50.494492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.0.97.62:36313)
0926 23:33:50.495103 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:50.495111 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:50.501676 (+ 6565us) server_negotiation.cc:407] Connection header received
0926 23:33:50.502268 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:50.502276 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:50.502403 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:50.502507 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:50.505242 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:50.506382 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:50.508901 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:50.509612 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:50.509726 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:50.514725 (+ 4999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:50.514734 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:50.514741 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:50.514796 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:50.518775 (+ 3979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:50.519887 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:50.519894 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:50.519898 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:50.520084 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:50.520577 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:50.520583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:50.520587 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:50.521011 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:50.521249 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:50.521838 (+ 589us) server_negotiation.cc:299] Negotiation successful
0926 23:33:50.522067 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
I20260926 23:33:50.523528 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:50.497959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41898 (local address 127.0.97.60:33859)
0926 23:33:50.499045 (+ 1086us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:50.499052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:50.503287 (+ 4235us) server_negotiation.cc:407] Connection header received
0926 23:33:50.504677 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:50.504688 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:50.504808 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:50.504904 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:50.507858 (+ 2954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:50.509036 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:50.511122 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:50.512239 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:50.512386 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:50.515970 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:50.515978 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:50.515984 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:50.516036 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:50.520195 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:50.521163 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:50.521169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:50.521173 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:50.521368 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:50.521776 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:50.521781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:50.521785 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:50.522192 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:50.522481 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:50.522844 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 23:33:50.523130 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":908}
I20260926 23:33:50.525519 14095 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf067451e222442db438e7956a007b7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b48d8fb2104fe2b82b3ea1565d2820" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36313 } } peers { permanent_uuid: "5697e87853b4414db9727f49a1e4d2bf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35887 } } peers { permanent_uuid: "cf067451e222442db438e7956a007b7f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 } }
I20260926 23:33:50.525946 14095 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf067451e222442db438e7956a007b7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:33:50.526082 14095 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf067451e222442db438e7956a007b7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf067451e222442db438e7956a007b7f, State: Initialized, Role: FOLLOWER
I20260926 23:33:50.526885 14095 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf067451e222442db438e7956a007b7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b48d8fb2104fe2b82b3ea1565d2820" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36313 } } peers { permanent_uuid: "5697e87853b4414db9727f49a1e4d2bf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35887 } } peers { permanent_uuid: "cf067451e222442db438e7956a007b7f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 } }
I20260926 23:33:50.528076 14111 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf067451e222442db438e7956a007b7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b48d8fb2104fe2b82b3ea1565d2820" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36313 } } peers { permanent_uuid: "5697e87853b4414db9727f49a1e4d2bf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35887 } } peers { permanent_uuid: "cf067451e222442db438e7956a007b7f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 } } }
I20260926 23:33:50.528455 14111 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf067451e222442db438e7956a007b7f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:33:50.529567 14095 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf067451e222442db438e7956a007b7f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:33:50.530880 14020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 } attempt: 5
Sep 26 23:33:50 dist-test-slave-gwfx krb5kdc[13866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260926 23:33:50.545907 14122 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cf067451e222442db438e7956a007b7f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 23:33:50.546080 14122 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 23:33:50.550264 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:50.531558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33859 (local address 127.0.0.1:41904)
0926 23:33:50.531821 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:50.531845 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:33:50.531979 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:33:50.532477 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:33:50.532483 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:33:50.533065 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:33:50.533323 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:50.533338 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:50.535241 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:50.535248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:50.536686 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:50.536704 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:50.537853 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:50.537860 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:50.538103 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:50.538907 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:33:50.538933 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:33:50.542072 (+ 3139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:33:50.546939 (+ 4867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:50.546946 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:50.546951 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:50.547572 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:50.548157 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:50.548163 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:50.548167 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:50.548372 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:50.549077 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:33:50.549083 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:33:50.549279 (+ 196us) client_negotiation.cc:769] Sending connection context
0926 23:33:50.549449 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 23:33:50.549814 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 23:33:50.550264 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:50.531863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41904 (local address 127.0.97.60:33859)
0926 23:33:50.532062 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:50.532072 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:50.532095 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:33:50.532171 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:50.532176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:50.532278 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:50.532424 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:50.533531 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:50.535033 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:50.536894 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:50.537680 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:50.537829 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:50.542330 (+ 4501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:50.542339 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:50.542346 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:50.542404 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:50.546741 (+ 4337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:50.547799 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:50.547805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:50.547809 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:50.548016 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:50.548498 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:50.548507 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:50.548509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:50.548948 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:50.549100 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:50.549570 (+ 470us) server_negotiation.cc:299] Negotiation successful
0926 23:33:50.549856 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 23:33:50.538983 14028 master_runner.cc:186] Error getting master registration for 127.0.97.62:36313: OK, instance_id { permanent_uuid: "58b48d8fb2104fe2b82b3ea1565d2820" instance_seqno: 1790465629167406 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 23:33:50.551261 14028 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 23:33:50.557210 14020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5697e87853b4414db9727f49a1e4d2bf: Bootstrap starting.
I20260926 23:33:50.561836 14020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5697e87853b4414db9727f49a1e4d2bf: Neither blocks nor log segments found. Creating new log.
I20260926 23:33:50.563354 14020 log.cc:824] T 00000000000000000000000000000000 P 5697e87853b4414db9727f49a1e4d2bf: Log is configured to *not* fsync() on all Append() calls
I20260926 23:33:50.567740 14020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5697e87853b4414db9727f49a1e4d2bf: No bootstrap required, opened a new log
I20260926 23:33:50.580533 14020 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5697e87853b4414db9727f49a1e4d2bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b48d8fb2104fe2b82b3ea1565d2820" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36313 } } peers { permanent_uuid: "5697e87853b4414db9727f49a1e4d2bf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35887 } } peers { permanent_uuid: "cf067451e222442db438e7956a007b7f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 } }
I20260926 23:33:50.581104 14020 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5697e87853b4414db9727f49a1e4d2bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:33:50.581249 14020 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5697e87853b4414db9727f49a1e4d2bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5697e87853b4414db9727f49a1e4d2bf, State: Initialized, Role: FOLLOWER
I20260926 23:33:50.582124 14020 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5697e87853b4414db9727f49a1e4d2bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b48d8fb2104fe2b82b3ea1565d2820" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36313 } } peers { permanent_uuid: "5697e87853b4414db9727f49a1e4d2bf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35887 } } peers { permanent_uuid: "cf067451e222442db438e7956a007b7f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 } }
I20260926 23:33:50.583343 14125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5697e87853b4414db9727f49a1e4d2bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b48d8fb2104fe2b82b3ea1565d2820" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36313 } } peers { permanent_uuid: "5697e87853b4414db9727f49a1e4d2bf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35887 } } peers { permanent_uuid: "cf067451e222442db438e7956a007b7f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 } } }
I20260926 23:33:50.583701 14125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5697e87853b4414db9727f49a1e4d2bf [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:33:50.585067 14020 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5697e87853b4414db9727f49a1e4d2bf [sys.catalog]: configured and running, proceeding with master startup.
W20260926 23:33:50.597381 14136 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5697e87853b4414db9727f49a1e4d2bf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 23:33:50.597570 14136 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 23:33:50.594163 13953 master_runner.cc:186] Error getting master registration for 127.0.97.62:36313: OK, instance_id { permanent_uuid: "58b48d8fb2104fe2b82b3ea1565d2820" instance_seqno: 1790465629167406 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 23:33:50.607434 13953 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 23:33:50.608336 13946 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 } attempt: 5
Sep 26 23:33:50 dist-test-slave-gwfx krb5kdc[13866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 23:33:50.630252 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:50.609267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41912 (local address 127.0.97.60:33859)
0926 23:33:50.609993 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:50.609999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:50.610021 (+ 22us) server_negotiation.cc:407] Connection header received
0926 23:33:50.610103 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:50.610109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:50.610211 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:50.610327 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:50.612043 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:50.613315 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:50.615307 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:50.616027 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:50.616203 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:50.622108 (+ 5905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:50.622116 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:50.622121 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:50.622271 (+ 150us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:50.626458 (+ 4187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:50.627590 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:50.627597 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:50.627601 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:50.627810 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:50.628355 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:50.628361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:50.628365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:50.628821 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:50.629011 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:50.629496 (+ 485us) server_negotiation.cc:299] Negotiation successful
0926 23:33:50.629789 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574}
I20260926 23:33:50.630300 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:50.609181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33859 (local address 127.0.0.1:41912)
0926 23:33:50.609557 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:50.609584 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 23:33:50.609727 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:33:50.610419 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:33:50.610425 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:33:50.611356 (+ 931us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:33:50.611833 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:50.611852 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:50.613513 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:50.613519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:50.615076 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:50.615108 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:50.616230 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:50.616237 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:50.616470 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:50.617355 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:33:50.617384 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:33:50.621849 (+ 4465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:33:50.626628 (+ 4779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:50.626635 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:50.626640 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:50.627410 (+ 770us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:50.627949 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:50.627955 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:50.627960 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:50.628198 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:50.628960 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:33:50.628965 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:33:50.629178 (+ 213us) client_negotiation.cc:769] Sending connection context
0926 23:33:50.629356 (+ 178us) client_negotiation.cc:240] Negotiation successful
0926 23:33:50.629731 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207}
W20260926 23:33:50.630700 14049 tablet.cc:2369] T 00000000000000000000000000000000 P cf067451e222442db438e7956a007b7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:33:50.637971 13946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820: Bootstrap starting.
I20260926 23:33:50.645233 13946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820: Neither blocks nor log segments found. Creating new log.
I20260926 23:33:50.647146 13946 log.cc:824] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820: Log is configured to *not* fsync() on all Append() calls
I20260926 23:33:50.651841 13946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820: No bootstrap required, opened a new log
I20260926 23:33:50.665517 13946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b48d8fb2104fe2b82b3ea1565d2820" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36313 } } peers { permanent_uuid: "5697e87853b4414db9727f49a1e4d2bf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35887 } } peers { permanent_uuid: "cf067451e222442db438e7956a007b7f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 } }
I20260926 23:33:50.666092 13946 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:33:50.666239 13946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58b48d8fb2104fe2b82b3ea1565d2820, State: Initialized, Role: FOLLOWER
I20260926 23:33:50.667151 13946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b48d8fb2104fe2b82b3ea1565d2820" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36313 } } peers { permanent_uuid: "5697e87853b4414db9727f49a1e4d2bf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35887 } } peers { permanent_uuid: "cf067451e222442db438e7956a007b7f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 } }
I20260926 23:33:50.668360 14139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b48d8fb2104fe2b82b3ea1565d2820" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36313 } } peers { permanent_uuid: "5697e87853b4414db9727f49a1e4d2bf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35887 } } peers { permanent_uuid: "cf067451e222442db438e7956a007b7f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 } } }
I20260926 23:33:50.668885 14139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:33:50.670217 13946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:33:50.688102 13879 master_runner.cc:263] No leader master found from master 58b48d8fb2104fe2b82b3ea1565d2820
I20260926 23:33:50.688294 13879 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 23:33:50.697955 14150 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 23:33:50.698285 14150 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260926 23:33:50.768705 13974 tablet.cc:2369] T 00000000000000000000000000000000 P 5697e87853b4414db9727f49a1e4d2bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:33:50.791905 14139 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:33:50.792300 14139 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b48d8fb2104fe2b82b3ea1565d2820" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36313 } } peers { permanent_uuid: "5697e87853b4414db9727f49a1e4d2bf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35887 } } peers { permanent_uuid: "cf067451e222442db438e7956a007b7f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 } }
I20260926 23:33:50.794430 14139 leader_election.cc:288] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5697e87853b4414db9727f49a1e4d2bf (127.0.97.61:35887), cf067451e222442db438e7956a007b7f (127.0.97.60:33859)
I20260926 23:33:50.796403 13995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "58b48d8fb2104fe2b82b3ea1565d2820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5697e87853b4414db9727f49a1e4d2bf" is_pre_election: true
I20260926 23:33:50.797194 13995 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5697e87853b4414db9727f49a1e4d2bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58b48d8fb2104fe2b82b3ea1565d2820 in term 0.
I20260926 23:33:50.797986 14070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "58b48d8fb2104fe2b82b3ea1565d2820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf067451e222442db438e7956a007b7f" is_pre_election: true
I20260926 23:33:50.798763 14070 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cf067451e222442db438e7956a007b7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58b48d8fb2104fe2b82b3ea1565d2820 in term 0.
I20260926 23:33:50.799997 13897 leader_election.cc:302] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5697e87853b4414db9727f49a1e4d2bf, 58b48d8fb2104fe2b82b3ea1565d2820; no voters:
I20260926 23:33:50.801067 14139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:33:50.801242 14139 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:33:50.801344 14139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:33:50.805279 14139 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b48d8fb2104fe2b82b3ea1565d2820" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36313 } } peers { permanent_uuid: "5697e87853b4414db9727f49a1e4d2bf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35887 } } peers { permanent_uuid: "cf067451e222442db438e7956a007b7f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 } }
I20260926 23:33:50.808027 13995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "58b48d8fb2104fe2b82b3ea1565d2820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5697e87853b4414db9727f49a1e4d2bf"
I20260926 23:33:50.808460 13995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5697e87853b4414db9727f49a1e4d2bf [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:33:50.812979 13995 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5697e87853b4414db9727f49a1e4d2bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58b48d8fb2104fe2b82b3ea1565d2820 in term 1.
I20260926 23:33:50.813766 13897 leader_election.cc:302] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5697e87853b4414db9727f49a1e4d2bf, 58b48d8fb2104fe2b82b3ea1565d2820; no voters:
I20260926 23:33:50.819248 14139 leader_election.cc:288] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820 [CANDIDATE]: Term 1 election: Requested vote from peers 5697e87853b4414db9727f49a1e4d2bf (127.0.97.61:35887), cf067451e222442db438e7956a007b7f (127.0.97.60:33859)
I20260926 23:33:50.820207 14139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:33:50.820789 14139 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820 [term 1 LEADER]: Becoming Leader. State: Replica: 58b48d8fb2104fe2b82b3ea1565d2820, State: Running, Role: LEADER
I20260926 23:33:50.821930 14139 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b48d8fb2104fe2b82b3ea1565d2820" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36313 } } peers { permanent_uuid: "5697e87853b4414db9727f49a1e4d2bf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35887 } } peers { permanent_uuid: "cf067451e222442db438e7956a007b7f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 } }
I20260926 23:33:50.833313 14070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "58b48d8fb2104fe2b82b3ea1565d2820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf067451e222442db438e7956a007b7f"
I20260926 23:33:50.833650 14070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf067451e222442db438e7956a007b7f [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:33:50.838513 14070 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cf067451e222442db438e7956a007b7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58b48d8fb2104fe2b82b3ea1565d2820 in term 1.
I20260926 23:33:50.857591 14152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58b48d8fb2104fe2b82b3ea1565d2820. Latest consensus state: current_term: 1 leader_uuid: "58b48d8fb2104fe2b82b3ea1565d2820" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b48d8fb2104fe2b82b3ea1565d2820" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36313 } } peers { permanent_uuid: "5697e87853b4414db9727f49a1e4d2bf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35887 } } peers { permanent_uuid: "cf067451e222442db438e7956a007b7f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 } } }
I20260926 23:33:50.858376 14152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820 [sys.catalog]: This master's current role is: LEADER
I20260926 23:33:50.875420 14154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:33:50.879413 14154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:33:50.906040 14070 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cf067451e222442db438e7956a007b7f [term 1 FOLLOWER]: Refusing update from remote peer 58b48d8fb2104fe2b82b3ea1565d2820: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260926 23:33:50.907058 13900 tablet.cc:2369] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:33:50.908785 14152 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf067451e222442db438e7956a007b7f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:33:50.917373 13995 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5697e87853b4414db9727f49a1e4d2bf [term 1 FOLLOWER]: Refusing update from remote peer 58b48d8fb2104fe2b82b3ea1565d2820: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:33:50.918702 14152 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5697e87853b4414db9727f49a1e4d2bf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:33:50.954587 14125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5697e87853b4414db9727f49a1e4d2bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58b48d8fb2104fe2b82b3ea1565d2820. Latest consensus state: current_term: 1 leader_uuid: "58b48d8fb2104fe2b82b3ea1565d2820" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b48d8fb2104fe2b82b3ea1565d2820" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36313 } } peers { permanent_uuid: "5697e87853b4414db9727f49a1e4d2bf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35887 } } peers { permanent_uuid: "cf067451e222442db438e7956a007b7f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 } } }
I20260926 23:33:50.955111 14125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5697e87853b4414db9727f49a1e4d2bf [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:33:50.959084 14155 mvcc.cc:205] Tried to move back new op lower bound from 7333747224176062464 to 7333747223863586816. Current Snapshot: MvccSnapshot[applied={T|T < 7333747224176062464}]
I20260926 23:33:50.964511 14111 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf067451e222442db438e7956a007b7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58b48d8fb2104fe2b82b3ea1565d2820. Latest consensus state: current_term: 1 leader_uuid: "58b48d8fb2104fe2b82b3ea1565d2820" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b48d8fb2104fe2b82b3ea1565d2820" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36313 } } peers { permanent_uuid: "5697e87853b4414db9727f49a1e4d2bf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35887 } } peers { permanent_uuid: "cf067451e222442db438e7956a007b7f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 } } }
I20260926 23:33:50.965004 14111 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf067451e222442db438e7956a007b7f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:33:50.973400 14111 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf067451e222442db438e7956a007b7f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "58b48d8fb2104fe2b82b3ea1565d2820" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b48d8fb2104fe2b82b3ea1565d2820" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36313 } } peers { permanent_uuid: "5697e87853b4414db9727f49a1e4d2bf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35887 } } peers { permanent_uuid: "cf067451e222442db438e7956a007b7f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 } } }
I20260926 23:33:50.973783 14111 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf067451e222442db438e7956a007b7f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:33:50.973913 14156 mvcc.cc:205] Tried to move back new op lower bound from 7333747224176062464 to 7333747223863586816. Current Snapshot: MvccSnapshot[applied={T|T < 7333747224176062464}]
I20260926 23:33:50.988176 14154 catalog_manager.cc:1347] Generated new cluster ID: 959dcf94ef0a46db9f4e2beec292696b
I20260926 23:33:50.988353 14154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:33:50.991461 14125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5697e87853b4414db9727f49a1e4d2bf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "58b48d8fb2104fe2b82b3ea1565d2820" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b48d8fb2104fe2b82b3ea1565d2820" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36313 } } peers { permanent_uuid: "5697e87853b4414db9727f49a1e4d2bf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35887 } } peers { permanent_uuid: "cf067451e222442db438e7956a007b7f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 } } }
I20260926 23:33:50.991825 14125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5697e87853b4414db9727f49a1e4d2bf [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:33:50.991879 14152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "58b48d8fb2104fe2b82b3ea1565d2820" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b48d8fb2104fe2b82b3ea1565d2820" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36313 } } peers { permanent_uuid: "5697e87853b4414db9727f49a1e4d2bf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35887 } } peers { permanent_uuid: "cf067451e222442db438e7956a007b7f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 } } }
I20260926 23:33:50.992187 14152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820 [sys.catalog]: This master's current role is: LEADER
I20260926 23:33:50.992475 14152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "58b48d8fb2104fe2b82b3ea1565d2820" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b48d8fb2104fe2b82b3ea1565d2820" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36313 } } peers { permanent_uuid: "5697e87853b4414db9727f49a1e4d2bf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35887 } } peers { permanent_uuid: "cf067451e222442db438e7956a007b7f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33859 } } }
I20260926 23:33:50.992681 14152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820 [sys.catalog]: This master's current role is: LEADER
I20260926 23:33:51.018040 14154 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:33:51.020537 14154 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:33:51.038341 14154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58b48d8fb2104fe2b82b3ea1565d2820: Generated new TSK 0
I20260926 23:33:51.040032 14154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:33:51.548693 14122 catalog_manager.cc:1259] Loaded cluster ID: 959dcf94ef0a46db9f4e2beec292696b
I20260926 23:33:51.548827 14122 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cf067451e222442db438e7956a007b7f: loading cluster ID for follower catalog manager: success
I20260926 23:33:51.551971 14122 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cf067451e222442db438e7956a007b7f: acquiring CA information for follower catalog manager: success
I20260926 23:33:51.554292 14122 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cf067451e222442db438e7956a007b7f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 23:33:51.600451 14136 catalog_manager.cc:1259] Loaded cluster ID: 959dcf94ef0a46db9f4e2beec292696b
I20260926 23:33:51.600593 14136 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5697e87853b4414db9727f49a1e4d2bf: loading cluster ID for follower catalog manager: success
I20260926 23:33:51.603992 14136 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5697e87853b4414db9727f49a1e4d2bf: acquiring CA information for follower catalog manager: success
I20260926 23:33:51.606632 14136 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5697e87853b4414db9727f49a1e4d2bf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 23:33:51.664513 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 13879
I20260926 23:33:51.666855 13941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:33:51.790155 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 13879
I20260926 23:33:51.817777 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 13953
I20260926 23:33:51.819397 14015 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:33:51.943459 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 13953
I20260926 23:33:51.964875 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 14028
I20260926 23:33:51.966503 14090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:33:52.089494 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 14028
2026-09-26T23:33:52Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4774 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:33:52 dist-test-slave-gwfx krb5kdc[14175](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:33:52 dist-test-slave-gwfx krb5kdc[14175](info): set up 2 sockets
Sep 26 23:33:52 dist-test-slave-gwfx krb5kdc[14175](info): commencing operation
krb5kdc: starting...
W20260926 23:33:53.143397 388 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.002s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:33:53 dist-test-slave-gwfx krb5kdc[14175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:33:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:33:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:33:53.415731 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37145
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:33:53.798627 14188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:33:53.799036 14188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:33:53.799175 14188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:33:53.806774 14188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:33:53.806918 14188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:33:53.806991 14188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:33:53.807066 14188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:33:53.807127 14188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:33:53.817489 14188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:33:53.819053 14188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:33:53.820660 14188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:33:53.835230 14193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:33:53.835695 14194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:33:53.835739 14196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:33:53.837113 14188 server_base.cc:1034] running on GCE node
I20260926 23:33:53.838272 14188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:33:53.840337 14188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:33:53.841604 14188 hybrid_clock.cc:648] HybridClock initialized: now 1790465633841543 us; error 59 us; skew 500 ppm
I20260926 23:33:53.842057 14188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:33:53 dist-test-slave-gwfx krb5kdc[14175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:33:53.845839 14188 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:33:53.847750 14188 webserver.cc:460] Webserver started at http://127.0.97.62:45489/ using document root <none> and password file <none>
I20260926 23:33:53.848585 14188 fs_manager.cc:359] Metadata directory not provided
I20260926 23:33:53.848696 14188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:33:53.849010 14188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:33:53.852759 14188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "f8a3377183f24d92a538a3845845e9d9"
format_stamp: "Formatted at 2026-09-26 23:33:53 on dist-test-slave-gwfx"
server_key: "1a48a4ce8593446cfc01d276b5b09bba"
server_key_iv: "bdaeb75b5fb536d805bc7f7b4974298b"
server_key_version: "encryptionkey@0"
I20260926 23:33:53.853646 14188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "f8a3377183f24d92a538a3845845e9d9"
format_stamp: "Formatted at 2026-09-26 23:33:53 on dist-test-slave-gwfx"
server_key: "1a48a4ce8593446cfc01d276b5b09bba"
server_key_iv: "bdaeb75b5fb536d805bc7f7b4974298b"
server_key_version: "encryptionkey@0"
I20260926 23:33:53.860303 14188 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 23:33:53.864861 14203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:33:53.866906 14188 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 23:33:53.867133 14188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "f8a3377183f24d92a538a3845845e9d9"
format_stamp: "Formatted at 2026-09-26 23:33:53 on dist-test-slave-gwfx"
server_key: "1a48a4ce8593446cfc01d276b5b09bba"
server_key_iv: "bdaeb75b5fb536d805bc7f7b4974298b"
server_key_version: "encryptionkey@0"
I20260926 23:33:53.867384 14188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:33:53.883323 14188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:33:53.888586 14188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:33:53.888882 14188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:33:53.909945 14188 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37145
I20260926 23:33:53.909981 14254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37145 every 8 connection(s)
I20260926 23:33:53.911724 14188 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:33:53.914551 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 14188
I20260926 23:33:53.914841 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:33:53.915719 388 external_mini_cluster.cc:1428] Setting key 30628ee4afb96e46d62bf85c9f9ab190
I20260926 23:33:53.917480 14255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 23:33:53 dist-test-slave-gwfx krb5kdc[14175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:33:53.934448 14255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8a3377183f24d92a538a3845845e9d9: Bootstrap starting.
I20260926 23:33:53.939049 14255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8a3377183f24d92a538a3845845e9d9: Neither blocks nor log segments found. Creating new log.
I20260926 23:33:53.940662 14255 log.cc:824] T 00000000000000000000000000000000 P f8a3377183f24d92a538a3845845e9d9: Log is configured to *not* fsync() on all Append() calls
I20260926 23:33:53.943948 14258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:53.917989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40448 (local address 127.0.97.62:37145)
0926 23:33:53.918877 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:53.918891 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:53.918940 (+ 49us) server_negotiation.cc:407] Connection header received
0926 23:33:53.920891 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:53.920922 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:53.921539 (+ 617us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:53.921988 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:53.924316 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:53.925973 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:53.928048 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:53.928910 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:53.929091 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:53.934905 (+ 5814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:53.934915 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:53.934936 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:53.934995 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:53.938454 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:53.939660 (+ 1206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:53.939671 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:53.939696 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:53.939949 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:53.940467 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:53.940471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:53.940473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:53.941236 (+ 763us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:53.941636 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:53.942130 (+ 494us) server_negotiation.cc:299] Negotiation successful
0926 23:33:53.942633 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":373,"threads_started":1}
I20260926 23:33:53.944741 14255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8a3377183f24d92a538a3845845e9d9: No bootstrap required, opened a new log
I20260926 23:33:53.948751 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:33:53.954213 14255 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8a3377183f24d92a538a3845845e9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8a3377183f24d92a538a3845845e9d9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37145 } }
I20260926 23:33:53.954780 14255 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8a3377183f24d92a538a3845845e9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:33:53.954895 14255 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8a3377183f24d92a538a3845845e9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8a3377183f24d92a538a3845845e9d9, State: Initialized, Role: FOLLOWER
I20260926 23:33:53.955602 14255 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8a3377183f24d92a538a3845845e9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8a3377183f24d92a538a3845845e9d9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37145 } }
I20260926 23:33:53.955889 14255 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8a3377183f24d92a538a3845845e9d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:33:53.956037 14255 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8a3377183f24d92a538a3845845e9d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:33:53.956225 14255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8a3377183f24d92a538a3845845e9d9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:33:53.958873 14255 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8a3377183f24d92a538a3845845e9d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8a3377183f24d92a538a3845845e9d9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37145 } }
I20260926 23:33:53.959551 14255 leader_election.cc:302] T 00000000000000000000000000000000 P f8a3377183f24d92a538a3845845e9d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8a3377183f24d92a538a3845845e9d9; no voters:
I20260926 23:33:53.960256 14255 leader_election.cc:288] T 00000000000000000000000000000000 P f8a3377183f24d92a538a3845845e9d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:33:53.960430 14260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8a3377183f24d92a538a3845845e9d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:33:53.961120 14260 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8a3377183f24d92a538a3845845e9d9 [term 1 LEADER]: Becoming Leader. State: Replica: f8a3377183f24d92a538a3845845e9d9, State: Running, Role: LEADER
I20260926 23:33:53.961781 14260 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8a3377183f24d92a538a3845845e9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8a3377183f24d92a538a3845845e9d9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37145 } }
I20260926 23:33:53.962662 14255 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8a3377183f24d92a538a3845845e9d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:33:53.964864 14261 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8a3377183f24d92a538a3845845e9d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8a3377183f24d92a538a3845845e9d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8a3377183f24d92a538a3845845e9d9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37145 } } }
I20260926 23:33:53.965505 14261 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8a3377183f24d92a538a3845845e9d9 [sys.catalog]: This master's current role is: LEADER
I20260926 23:33:53.965979 14262 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8a3377183f24d92a538a3845845e9d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8a3377183f24d92a538a3845845e9d9. Latest consensus state: current_term: 1 leader_uuid: "f8a3377183f24d92a538a3845845e9d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8a3377183f24d92a538a3845845e9d9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37145 } } }
I20260926 23:33:53.966384 14262 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8a3377183f24d92a538a3845845e9d9 [sys.catalog]: This master's current role is: LEADER
I20260926 23:33:53.972648 14269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:33:53.978246 14269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:33:53.990197 14269 catalog_manager.cc:1347] Generated new cluster ID: adc0d712b226497fae8b3fc931884cea
I20260926 23:33:53.990381 14269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:33:54.002321 14269 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:33:54.004774 14269 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:33:54.020953 14269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8a3377183f24d92a538a3845845e9d9: Generated new TSK 0
I20260926 23:33:54.022657 14269 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:33:54.063442 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 14188
I20260926 23:33:54.065057 14250 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:33:54.170830 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 14188
2026-09-26T23:33:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2076 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:33:54 dist-test-slave-gwfx krb5kdc[14283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:33:54 dist-test-slave-gwfx krb5kdc[14283](info): set up 2 sockets
Sep 26 23:33:54 dist-test-slave-gwfx krb5kdc[14283](info): commencing operation
krb5kdc: starting...
W20260926 23:33:55.215413 388 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.002s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:33:55 dist-test-slave-gwfx krb5kdc[14283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:33:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:33:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:33:55.495102 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37995
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:33:55.888643 14296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:33:55.889057 14296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:33:55.889189 14296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:33:55.897336 14296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:33:55.897490 14296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:33:55.897564 14296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:33:55.897644 14296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:33:55.897706 14296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:33:55.908972 14296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:33:55.910863 14296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:33:55.912488 14296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:33:55.926868 14301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:33:55.926923 14304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:33:55.926987 14302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:33:55.927652 14296 server_base.cc:1034] running on GCE node
I20260926 23:33:55.928859 14296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:33:55.930863 14296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:33:55.932114 14296 hybrid_clock.cc:648] HybridClock initialized: now 1790465635932053 us; error 56 us; skew 500 ppm
I20260926 23:33:55.932538 14296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:33:55 dist-test-slave-gwfx krb5kdc[14283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:33:55.936614 14296 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:33:55.938670 14296 webserver.cc:460] Webserver started at http://127.0.97.62:39459/ using document root <none> and password file <none>
I20260926 23:33:55.939870 14296 fs_manager.cc:359] Metadata directory not provided
I20260926 23:33:55.940047 14296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:33:55.940471 14296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:33:55.944764 14296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "8255fe64fe7f4c0ca60dc6aa921e1bc5"
format_stamp: "Formatted at 2026-09-26 23:33:55 on dist-test-slave-gwfx"
server_key: "e866be79e2b94da75fd8e01ad40010f1"
server_key_iv: "8f125005c3e916e784eb75b31cadb717"
server_key_version: "encryptionkey@0"
I20260926 23:33:55.946148 14296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "8255fe64fe7f4c0ca60dc6aa921e1bc5"
format_stamp: "Formatted at 2026-09-26 23:33:55 on dist-test-slave-gwfx"
server_key: "e866be79e2b94da75fd8e01ad40010f1"
server_key_iv: "8f125005c3e916e784eb75b31cadb717"
server_key_version: "encryptionkey@0"
I20260926 23:33:55.954593 14296 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.010s sys 0.001s
I20260926 23:33:55.959865 14311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:33:55.962069 14296 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:33:55.962335 14296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "8255fe64fe7f4c0ca60dc6aa921e1bc5"
format_stamp: "Formatted at 2026-09-26 23:33:55 on dist-test-slave-gwfx"
server_key: "e866be79e2b94da75fd8e01ad40010f1"
server_key_iv: "8f125005c3e916e784eb75b31cadb717"
server_key_version: "encryptionkey@0"
I20260926 23:33:55.962601 14296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:33:55.985870 14296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:33:55.992367 14296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:33:55.992753 14296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:33:56.016683 14296 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37995
I20260926 23:33:56.016719 14362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37995 every 8 connection(s)
I20260926 23:33:56.018500 14296 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:33:56.023682 14363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:33:56.024657 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 14296
I20260926 23:33:56.024906 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:33:56.025863 388 external_mini_cluster.cc:1428] Setting key c24c9453c893678d75f2ca30fe2a3adb
I20260926 23:33:56.042543 14363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5: Bootstrap starting.
Sep 26 23:33:56 dist-test-slave-gwfx krb5kdc[14283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:33:56.047506 14363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5: Neither blocks nor log segments found. Creating new log.
I20260926 23:33:56.048894 14363 log.cc:824] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5: Log is configured to *not* fsync() on all Append() calls
I20260926 23:33:56.053458 14363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5: No bootstrap required, opened a new log
I20260926 23:33:56.056123 14366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:56.028581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38652 (local address 127.0.97.62:37995)
0926 23:33:56.029571 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:56.029588 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:56.029651 (+ 63us) server_negotiation.cc:407] Connection header received
0926 23:33:56.031711 (+ 2060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:56.031752 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:56.032378 (+ 626us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:56.032854 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:56.034893 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:56.036517 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:56.038980 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:56.040116 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:56.040334 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:56.046304 (+ 5970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:56.046317 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:56.046341 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:56.046409 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:56.050340 (+ 3931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:56.051721 (+ 1381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:56.051731 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:56.051740 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:56.051956 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:56.052489 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:56.052506 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:56.052509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:56.053251 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:56.053556 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:56.054171 (+ 615us) server_negotiation.cc:299] Negotiation successful
0926 23:33:56.054766 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":437,"threads_started":1}
I20260926 23:33:56.063814 14363 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8255fe64fe7f4c0ca60dc6aa921e1bc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37995 } }
I20260926 23:33:56.064383 14363 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:33:56.064503 14363 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8255fe64fe7f4c0ca60dc6aa921e1bc5, State: Initialized, Role: FOLLOWER
I20260926 23:33:56.065271 14363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8255fe64fe7f4c0ca60dc6aa921e1bc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37995 } }
I20260926 23:33:56.065559 14363 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:33:56.065810 14363 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:33:56.066027 14363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:33:56.069036 14363 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8255fe64fe7f4c0ca60dc6aa921e1bc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37995 } }
I20260926 23:33:56.069773 14363 leader_election.cc:302] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8255fe64fe7f4c0ca60dc6aa921e1bc5; no voters:
I20260926 23:33:56.070521 14363 leader_election.cc:288] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:33:56.070789 14368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:33:56.072152 14368 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [term 1 LEADER]: Becoming Leader. State: Replica: 8255fe64fe7f4c0ca60dc6aa921e1bc5, State: Running, Role: LEADER
I20260926 23:33:56.072875 14368 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8255fe64fe7f4c0ca60dc6aa921e1bc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37995 } }
I20260926 23:33:56.073493 14363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:33:56.076426 14370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8255fe64fe7f4c0ca60dc6aa921e1bc5. Latest consensus state: current_term: 1 leader_uuid: "8255fe64fe7f4c0ca60dc6aa921e1bc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8255fe64fe7f4c0ca60dc6aa921e1bc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37995 } } }
I20260926 23:33:56.076514 14369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8255fe64fe7f4c0ca60dc6aa921e1bc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8255fe64fe7f4c0ca60dc6aa921e1bc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37995 } } }
I20260926 23:33:56.076825 14370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [sys.catalog]: This master's current role is: LEADER
I20260926 23:33:56.076890 14369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [sys.catalog]: This master's current role is: LEADER
I20260926 23:33:56.077759 14375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:33:56.092392 14375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:33:56.106462 14375 catalog_manager.cc:1347] Generated new cluster ID: 74c8e7b904974b18a48bd8f4729db3bb
I20260926 23:33:56.106611 14375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:33:56.149344 14375 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:33:56.151058 14375 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:33:56.163686 14375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5: Generated new TSK 0
I20260926 23:33:56.165277 14375 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:33:56.250876 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37995
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:33:56.676254 14389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:33:56.676662 14389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:33:56.676813 14389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:33:56.685114 14389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:33:56.685312 14389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:33:56.685484 14389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:33:56.697032 14389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:33:56.698918 14389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:33:56.700734 14389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:33:56.717374 14394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:33:56.717398 14397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:33:56.717420 14395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:33:56.724094 14389 server_base.cc:1034] running on GCE node
I20260926 23:33:56.725148 14389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:33:56.726627 14389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:33:56.727931 14389 hybrid_clock.cc:648] HybridClock initialized: now 1790465636727883 us; error 49 us; skew 500 ppm
I20260926 23:33:56.728350 14389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:33:56 dist-test-slave-gwfx krb5kdc[14283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:33:56.732784 14389 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:33:56.735062 14389 webserver.cc:460] Webserver started at http://127.0.97.1:38349/ using document root <none> and password file <none>
I20260926 23:33:56.736027 14389 fs_manager.cc:359] Metadata directory not provided
I20260926 23:33:56.736186 14389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:33:56.736588 14389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:33:56.740512 14389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/data/instance:
uuid: "be4b5734244b4b418e21dc6e65bb9bbb"
format_stamp: "Formatted at 2026-09-26 23:33:56 on dist-test-slave-gwfx"
server_key: "a12e27122c093abc0b0e4cdbfb3ae8f8"
server_key_iv: "edb1145c0d529d437085a9da2d2e7d7f"
server_key_version: "encryptionkey@0"
I20260926 23:33:56.741524 14389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/wal/instance:
uuid: "be4b5734244b4b418e21dc6e65bb9bbb"
format_stamp: "Formatted at 2026-09-26 23:33:56 on dist-test-slave-gwfx"
server_key: "a12e27122c093abc0b0e4cdbfb3ae8f8"
server_key_iv: "edb1145c0d529d437085a9da2d2e7d7f"
server_key_version: "encryptionkey@0"
I20260926 23:33:56.748375 14389 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 23:33:56.752732 14404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:33:56.754570 14389 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 23:33:56.754803 14389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "be4b5734244b4b418e21dc6e65bb9bbb"
format_stamp: "Formatted at 2026-09-26 23:33:56 on dist-test-slave-gwfx"
server_key: "a12e27122c093abc0b0e4cdbfb3ae8f8"
server_key_iv: "edb1145c0d529d437085a9da2d2e7d7f"
server_key_version: "encryptionkey@0"
I20260926 23:33:56.755043 14389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:33:56.769871 14389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:33:56.775249 14389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:33:56.775625 14389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:33:56.776845 14389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:33:56.778574 14389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:33:56.778664 14389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:33:56.778867 14389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:33:56.778944 14389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:33:56.814828 14389 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40569
I20260926 23:33:56.814901 14516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40569 every 8 connection(s)
I20260926 23:33:56.816543 14389 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:33:56.825610 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 14389
I20260926 23:33:56.825871 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:33:56.826612 388 external_mini_cluster.cc:1428] Setting key 8b040d3806231096212466f1d110c2d2
Sep 26 23:33:56 dist-test-slave-gwfx krb5kdc[14283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:33:56.842447 14519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:56.820151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56549 (local address 127.0.97.62:37995)
0926 23:33:56.820776 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:56.820782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:56.821969 (+ 1187us) server_negotiation.cc:407] Connection header received
0926 23:33:56.824334 (+ 2365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:56.824341 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:56.824446 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:56.824556 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:56.826634 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:56.827609 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:56.829980 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:56.830412 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:56.830543 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:56.835094 (+ 4551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:56.835102 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:56.835107 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:56.835157 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:56.838364 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:56.839622 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:56.839628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:56.839631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:56.839803 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:56.840378 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:56.840383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:56.840386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:56.840744 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:56.840877 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:56.841888 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0926 23:33:56.842136 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":327,"threads_started":1}
I20260926 23:33:56.844058 14520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:56.820739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.1:56549)
0926 23:33:56.821729 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:56.821792 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 23:33:56.823951 (+ 2159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:33:56.824870 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:33:56.824881 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:33:56.825576 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:33:56.826401 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:56.826424 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:56.827799 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:56.827803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:56.829796 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:56.829820 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:56.830727 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:56.830737 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:56.831036 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:56.831867 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:33:56.831896 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:33:56.834743 (+ 2847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:33:56.838598 (+ 3855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:56.838611 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:56.838639 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:56.839417 (+ 778us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:56.839924 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:56.839930 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:56.839933 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:56.840248 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:56.840868 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:33:56.840877 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:33:56.841405 (+ 528us) client_negotiation.cc:769] Sending connection context
0926 23:33:56.841753 (+ 348us) client_negotiation.cc:240] Negotiation successful
0926 23:33:56.842424 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":387,"threads_started":1}
I20260926 23:33:56.846108 14517 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37995
I20260926 23:33:56.846508 14517 heartbeater.cc:464] Registering TS with master...
I20260926 23:33:56.847532 14517 heartbeater.cc:511] Master 127.0.97.62:37995 requested a full tablet report, sending...
I20260926 23:33:56.850395 14328 ts_manager.cc:194] Registered new tserver with Master: be4b5734244b4b418e21dc6e65bb9bbb (127.0.97.1:40569)
I20260926 23:33:56.852710 14328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56549
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:33:56.894936 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37995
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:33:57.295831 14523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:33:57.296254 14523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:33:57.296406 14523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:33:57.304648 14523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:33:57.304831 14523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:33:57.304996 14523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:33:57.316543 14523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:33:57.318627 14523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:33:57.320554 14523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:33:57.336934 14528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:33:57.336887 14529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:33:57.336863 14531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:33:57.338033 14523 server_base.cc:1034] running on GCE node
I20260926 23:33:57.338960 14523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:33:57.340270 14523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:33:57.341507 14523 hybrid_clock.cc:648] HybridClock initialized: now 1790465637341453 us; error 60 us; skew 500 ppm
I20260926 23:33:57.341899 14523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:33:57 dist-test-slave-gwfx krb5kdc[14283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:33:57.345688 14523 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:33:57.347536 14523 webserver.cc:460] Webserver started at http://127.0.97.2:37053/ using document root <none> and password file <none>
I20260926 23:33:57.348421 14523 fs_manager.cc:359] Metadata directory not provided
I20260926 23:33:57.348546 14523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:33:57.348891 14523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:33:57.352571 14523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/data/instance:
uuid: "089dde0b70c247128b9290f7b46e474e"
format_stamp: "Formatted at 2026-09-26 23:33:57 on dist-test-slave-gwfx"
server_key: "56ba41fb43d5d957ff4794d82d9b133d"
server_key_iv: "d312e17fb53e303e073b1923be780858"
server_key_version: "encryptionkey@0"
I20260926 23:33:57.353521 14523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/wal/instance:
uuid: "089dde0b70c247128b9290f7b46e474e"
format_stamp: "Formatted at 2026-09-26 23:33:57 on dist-test-slave-gwfx"
server_key: "56ba41fb43d5d957ff4794d82d9b133d"
server_key_iv: "d312e17fb53e303e073b1923be780858"
server_key_version: "encryptionkey@0"
I20260926 23:33:57.360608 14523 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260926 23:33:57.365118 14538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:33:57.366936 14523 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:33:57.367213 14523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "089dde0b70c247128b9290f7b46e474e"
format_stamp: "Formatted at 2026-09-26 23:33:57 on dist-test-slave-gwfx"
server_key: "56ba41fb43d5d957ff4794d82d9b133d"
server_key_iv: "d312e17fb53e303e073b1923be780858"
server_key_version: "encryptionkey@0"
I20260926 23:33:57.367448 14523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:33:57.390328 14523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:33:57.395828 14523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:33:57.396152 14523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:33:57.397378 14523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:33:57.399166 14523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:33:57.399266 14523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:33:57.399376 14523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:33:57.399421 14523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:33:57.434474 14523 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38991
I20260926 23:33:57.434597 14650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38991 every 8 connection(s)
I20260926 23:33:57.436178 14523 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:33:57.445111 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 14523
I20260926 23:33:57.445304 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:33:57.445817 388 external_mini_cluster.cc:1428] Setting key 7c906bd169fff37dd56dbef207b13917
Sep 26 23:33:57 dist-test-slave-gwfx krb5kdc[14283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:33:57.461243 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:57.439226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43683 (local address 127.0.97.62:37995)
0926 23:33:57.439726 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:57.439733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:57.440878 (+ 1145us) server_negotiation.cc:407] Connection header received
0926 23:33:57.443164 (+ 2286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:57.443171 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:57.443285 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:57.443399 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:57.445453 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:57.446313 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:57.448418 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:57.448915 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:57.449037 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:57.453715 (+ 4678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:57.453723 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:57.453727 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:57.453773 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:57.456824 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:57.457930 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:57.457936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:57.457940 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:57.458143 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:57.458730 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:57.458735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:57.458738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:57.459153 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:57.459342 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:57.460449 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0926 23:33:57.460795 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":244,"threads_started":1}
I20260926 23:33:57.462538 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:57.439810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.2:43683)
0926 23:33:57.440656 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:57.440709 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 23:33:57.442813 (+ 2104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:33:57.443714 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:33:57.443725 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:33:57.444405 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:33:57.445169 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:57.445192 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:57.446501 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:57.446506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:57.448262 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:57.448278 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:57.449188 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:57.449198 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:57.449479 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:57.450326 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:33:57.450360 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:33:57.453307 (+ 2947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:33:57.457055 (+ 3748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:57.457066 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:57.457087 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:57.457747 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:57.458309 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:57.458313 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:57.458316 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:57.458579 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:57.459404 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:33:57.459416 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:33:57.459920 (+ 504us) client_negotiation.cc:769] Sending connection context
0926 23:33:57.460293 (+ 373us) client_negotiation.cc:240] Negotiation successful
0926 23:33:57.460969 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":333,"threads_started":1}
I20260926 23:33:57.464695 14651 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37995
I20260926 23:33:57.465072 14651 heartbeater.cc:464] Registering TS with master...
I20260926 23:33:57.466013 14651 heartbeater.cc:511] Master 127.0.97.62:37995 requested a full tablet report, sending...
I20260926 23:33:57.468118 14328 ts_manager.cc:194] Registered new tserver with Master: 089dde0b70c247128b9290f7b46e474e (127.0.97.2:38991)
I20260926 23:33:57.469187 14328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43683
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:33:57.506778 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37995
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:33:57.857901 14517 heartbeater.cc:503] Master 127.0.97.62:37995 was elected leader, sending a full tablet report...
W20260926 23:33:57.902096 14657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:33:57.902508 14657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:33:57.902650 14657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:33:57.910892 14657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:33:57.911075 14657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:33:57.911250 14657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:33:57.922432 14657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:33:57.924268 14657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:33:57.926127 14657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:33:57.941620 14662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:33:57.941545 14663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:33:57.941578 14665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:33:57.943023 14657 server_base.cc:1034] running on GCE node
I20260926 23:33:57.943951 14657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:33:57.945189 14657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:33:57.946408 14657 hybrid_clock.cc:648] HybridClock initialized: now 1790465637946333 us; error 65 us; skew 500 ppm
I20260926 23:33:57.946866 14657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:33:57 dist-test-slave-gwfx krb5kdc[14283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:33:57.950592 14657 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:33:57.952589 14657 webserver.cc:460] Webserver started at http://127.0.97.3:44293/ using document root <none> and password file <none>
I20260926 23:33:57.953490 14657 fs_manager.cc:359] Metadata directory not provided
I20260926 23:33:57.953635 14657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:33:57.954007 14657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:33:57.957913 14657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/data/instance:
uuid: "df32c007eff84cacb8eade269503bf10"
format_stamp: "Formatted at 2026-09-26 23:33:57 on dist-test-slave-gwfx"
server_key: "777ce6dacc53d96475b6e28ff1d9a4c9"
server_key_iv: "89849c912d646464ac21536d4add928e"
server_key_version: "encryptionkey@0"
I20260926 23:33:57.959074 14657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/wal/instance:
uuid: "df32c007eff84cacb8eade269503bf10"
format_stamp: "Formatted at 2026-09-26 23:33:57 on dist-test-slave-gwfx"
server_key: "777ce6dacc53d96475b6e28ff1d9a4c9"
server_key_iv: "89849c912d646464ac21536d4add928e"
server_key_version: "encryptionkey@0"
I20260926 23:33:57.966054 14657 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260926 23:33:57.971019 14672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:33:57.972858 14657 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260926 23:33:57.973047 14657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "df32c007eff84cacb8eade269503bf10"
format_stamp: "Formatted at 2026-09-26 23:33:57 on dist-test-slave-gwfx"
server_key: "777ce6dacc53d96475b6e28ff1d9a4c9"
server_key_iv: "89849c912d646464ac21536d4add928e"
server_key_version: "encryptionkey@0"
I20260926 23:33:57.973381 14657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:33:57.989336 14657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:33:57.995277 14657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:33:57.995656 14657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:33:57.996958 14657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:33:57.999019 14657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:33:57.999132 14657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:33:57.999282 14657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:33:57.999346 14657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:33:58.036562 14657 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40287
I20260926 23:33:58.036586 14784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40287 every 8 connection(s)
I20260926 23:33:58.038259 14657 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:33:58.048704 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 14657
I20260926 23:33:58.048898 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:33:58.049448 388 external_mini_cluster.cc:1428] Setting key 5d56ccf0e679f34e5f9cc8a5dbf38ee3
Sep 26 23:33:58 dist-test-slave-gwfx krb5kdc[14283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:33:58.062860 14787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:58.041779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57265 (local address 127.0.97.62:37995)
0926 23:33:58.042408 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:58.042415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:58.043675 (+ 1260us) server_negotiation.cc:407] Connection header received
0926 23:33:58.045884 (+ 2209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:58.045890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:58.045988 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:58.046130 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:58.048245 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.049241 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.051375 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.051881 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.052003 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:58.056241 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:58.056247 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:58.056250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:58.056295 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:58.059089 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:58.060001 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:58.060006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:58.060009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:58.060161 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:58.060664 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:58.060668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:58.060670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:58.060995 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:58.061151 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:58.062087 (+ 936us) server_negotiation.cc:299] Negotiation successful
0926 23:33:58.062388 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":285,"threads_started":1}
I20260926 23:33:58.064142 14788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:58.042488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.3:57265)
0926 23:33:58.043454 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:58.043509 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 23:33:58.045509 (+ 2000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:33:58.046391 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:33:58.046402 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:33:58.047173 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:33:58.047937 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:58.047959 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.049440 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.049445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:58.051226 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:58.051243 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.052140 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.052149 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:58.052419 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:58.053203 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:33:58.053231 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:33:58.055884 (+ 2653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:33:58.059291 (+ 3407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:58.059300 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:58.059320 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:58.059849 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:58.060288 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:33:58.060293 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:33:58.060296 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:33:58.060532 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:33:58.061125 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:33:58.061134 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:33:58.061640 (+ 506us) client_negotiation.cc:769] Sending connection context
0926 23:33:58.061931 (+ 291us) client_negotiation.cc:240] Negotiation successful
0926 23:33:58.062633 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":372,"threads_started":1}
I20260926 23:33:58.065920 14785 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37995
I20260926 23:33:58.066318 14785 heartbeater.cc:464] Registering TS with master...
I20260926 23:33:58.067286 14785 heartbeater.cc:511] Master 127.0.97.62:37995 requested a full tablet report, sending...
I20260926 23:33:58.069223 14328 ts_manager.cc:194] Registered new tserver with Master: df32c007eff84cacb8eade269503bf10 (127.0.97.3:40287)
I20260926 23:33:58.070212 14328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57265
I20260926 23:33:58.078984 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:33:58.100467 14787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:58.084197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33282 (local address 127.0.97.62:37995)
0926 23:33:58.084464 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:58.084471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:58.084767 (+ 296us) server_negotiation.cc:407] Connection header received
0926 23:33:58.084982 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:58.084988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:58.085109 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:58.085280 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:58.087122 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.088136 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.090372 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.091054 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.091178 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:58.093895 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:58.093906 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:58.093910 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:58.093962 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:58.097003 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:58.097910 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:58.097915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:58.097918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:58.098093 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:58.098512 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:58.098517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:58.098520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:58.099056 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:58.099290 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:58.099705 (+ 415us) server_negotiation.cc:299] Negotiation successful
0926 23:33:58.100088 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 23:33:58.106138 14328 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:33:58.114321 14328 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:33:58.143854 14800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:58.131794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38991 (local address 127.0.0.1:43800)
0926 23:33:58.132825 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:58.132842 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 23:33:58.132973 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:33:58.134089 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:33:58.134110 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:33:58.134146 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:33:58.134711 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:58.134731 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.137039 (+ 2308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.137046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:58.139766 (+ 2720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:58.139789 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.142561 (+ 2772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.142569 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:58.142842 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:58.142918 (+ 76us) client_negotiation.cc:769] Sending connection context
0926 23:33:58.143060 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 23:33:58.143330 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"thread_start_us":1266,"threads_started":1}
I20260926 23:33:58.144560 14798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:58.131019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40569 (local address 127.0.0.1:47848)
0926 23:33:58.132672 (+ 1653us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:58.132726 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 23:33:58.132957 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:33:58.134396 (+ 1439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:33:58.134401 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:33:58.134427 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:33:58.134959 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:58.134973 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.138587 (+ 3614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.138594 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:58.141133 (+ 2539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:58.141154 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.143622 (+ 2468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.143629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:58.143863 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:58.143904 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 23:33:58.143968 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 23:33:58.144133 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1286,"thread_start_us":2110,"threads_started":1}
I20260926 23:33:58.145210 14801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:58.132072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43800 (local address 127.0.97.2:38991)
0926 23:33:58.132977 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:58.132985 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:58.133017 (+ 32us) server_negotiation.cc:407] Connection header received
0926 23:33:58.133239 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:58.133246 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:58.133515 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:58.133801 (+ 286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:33:58.134955 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.136832 (+ 1877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.140042 (+ 3210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.141985 (+ 1943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.142143 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:58.142422 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:58.144574 (+ 2152us) server_negotiation.cc:299] Negotiation successful
0926 23:33:58.144821 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":606,"threads_started":1}
I20260926 23:33:58.146207 14799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:58.131272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47848 (local address 127.0.97.1:40569)
0926 23:33:58.132518 (+ 1246us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:58.132530 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:58.133806 (+ 1276us) server_negotiation.cc:407] Connection header received
0926 23:33:58.133933 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:58.133940 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:58.134164 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:58.136292 (+ 2128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:33:58.136612 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.138384 (+ 1772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.141450 (+ 3066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.143338 (+ 1888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.145198 (+ 1860us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:58.145529 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:58.145620 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 23:33:58.145832 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1032,"thread_start_us":830,"threads_started":1}
I20260926 23:33:58.150364 14452 tablet_service.cc:1467] Processing CreateTablet for tablet 68ba3e13cf3a42c68413db7c0f366abd (DEFAULT_TABLE table=test-table [id=15ae46a99afe44afb186b61ec717be96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:33:58.150370 14586 tablet_service.cc:1467] Processing CreateTablet for tablet 68ba3e13cf3a42c68413db7c0f366abd (DEFAULT_TABLE table=test-table [id=15ae46a99afe44afb186b61ec717be96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:33:58.152047 14586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68ba3e13cf3a42c68413db7c0f366abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:33:58.152038 14452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68ba3e13cf3a42c68413db7c0f366abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:33:58.157027 14800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:58.133982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40287 (local address 127.0.0.1:41646)
0926 23:33:58.144190 (+ 10208us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:58.144210 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:33:58.144341 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:33:58.149230 (+ 4889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:33:58.149236 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:33:58.149271 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:33:58.149480 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:58.149492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.151611 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.151618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:58.153635 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:58.153689 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.156084 (+ 2395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.156092 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:58.156307 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:58.156349 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 23:33:58.156465 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 23:33:58.156670 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10093,"thread_start_us":12698,"threads_started":1}
I20260926 23:33:58.157145 14803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:58.134657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41646 (local address 127.0.97.3:40287)
0926 23:33:58.147740 (+ 13083us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:58.147750 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:58.148571 (+ 821us) server_negotiation.cc:407] Connection header received
0926 23:33:58.148696 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:58.148704 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:58.148993 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:58.149219 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:33:58.149655 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.151419 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.153922 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.155867 (+ 1945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.156025 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:58.156315 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:58.156525 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 23:33:58.156736 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12636,"thread_start_us":13117,"threads_started":1}
I20260926 23:33:58.163057 14720 tablet_service.cc:1467] Processing CreateTablet for tablet 68ba3e13cf3a42c68413db7c0f366abd (DEFAULT_TABLE table=test-table [id=15ae46a99afe44afb186b61ec717be96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:33:58.165048 14720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68ba3e13cf3a42c68413db7c0f366abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:33:58.167233 14804 tablet_bootstrap.cc:492] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e: Bootstrap starting.
I20260926 23:33:58.168200 14805 tablet_bootstrap.cc:492] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb: Bootstrap starting.
I20260926 23:33:58.171856 14805 tablet_bootstrap.cc:654] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb: Neither blocks nor log segments found. Creating new log.
I20260926 23:33:58.171857 14804 tablet_bootstrap.cc:654] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e: Neither blocks nor log segments found. Creating new log.
I20260926 23:33:58.173527 14805 log.cc:824] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb: Log is configured to *not* fsync() on all Append() calls
I20260926 23:33:58.173537 14804 log.cc:824] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e: Log is configured to *not* fsync() on all Append() calls
I20260926 23:33:58.177245 14804 tablet_bootstrap.cc:492] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e: No bootstrap required, opened a new log
I20260926 23:33:58.177470 14805 tablet_bootstrap.cc:492] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb: No bootstrap required, opened a new log
I20260926 23:33:58.177670 14804 ts_tablet_manager.cc:1397] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260926 23:33:58.177798 14805 ts_tablet_manager.cc:1397] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260926 23:33:58.186672 14804 raft_consensus.cc:348] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "df32c007eff84cacb8eade269503bf10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40287 } } peers { permanent_uuid: "089dde0b70c247128b9290f7b46e474e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38991 } }
I20260926 23:33:58.187407 14804 raft_consensus.cc:374] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:33:58.187522 14804 raft_consensus.cc:729] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 089dde0b70c247128b9290f7b46e474e, State: Initialized, Role: FOLLOWER
I20260926 23:33:58.188053 14808 tablet_bootstrap.cc:492] T 68ba3e13cf3a42c68413db7c0f366abd P df32c007eff84cacb8eade269503bf10: Bootstrap starting.
I20260926 23:33:58.188506 14804 consensus_queue.cc:260] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "df32c007eff84cacb8eade269503bf10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40287 } } peers { permanent_uuid: "089dde0b70c247128b9290f7b46e474e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38991 } }
I20260926 23:33:58.189956 14651 heartbeater.cc:503] Master 127.0.97.62:37995 was elected leader, sending a full tablet report...
I20260926 23:33:58.190155 14805 raft_consensus.cc:348] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "df32c007eff84cacb8eade269503bf10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40287 } } peers { permanent_uuid: "089dde0b70c247128b9290f7b46e474e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38991 } }
I20260926 23:33:58.190639 14805 raft_consensus.cc:374] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:33:58.190798 14805 raft_consensus.cc:729] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be4b5734244b4b418e21dc6e65bb9bbb, State: Initialized, Role: FOLLOWER
I20260926 23:33:58.191552 14805 consensus_queue.cc:260] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "df32c007eff84cacb8eade269503bf10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40287 } } peers { permanent_uuid: "089dde0b70c247128b9290f7b46e474e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38991 } }
I20260926 23:33:58.192116 14804 ts_tablet_manager.cc:1428] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260926 23:33:58.193099 14808 tablet_bootstrap.cc:654] T 68ba3e13cf3a42c68413db7c0f366abd P df32c007eff84cacb8eade269503bf10: Neither blocks nor log segments found. Creating new log.
I20260926 23:33:58.194054 14805 ts_tablet_manager.cc:1428] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb: Time spent starting tablet: real 0.016s user 0.016s sys 0.001s
I20260926 23:33:58.194723 14808 log.cc:824] T 68ba3e13cf3a42c68413db7c0f366abd P df32c007eff84cacb8eade269503bf10: Log is configured to *not* fsync() on all Append() calls
I20260926 23:33:58.196532 14809 raft_consensus.cc:482] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:33:58.196898 14809 raft_consensus.cc:504] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "df32c007eff84cacb8eade269503bf10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40287 } } peers { permanent_uuid: "089dde0b70c247128b9290f7b46e474e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38991 } }
I20260926 23:33:58.199486 14809 leader_election.cc:288] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be4b5734244b4b418e21dc6e65bb9bbb (127.0.97.1:40569), df32c007eff84cacb8eade269503bf10 (127.0.97.3:40287)
I20260926 23:33:58.200599 14808 tablet_bootstrap.cc:492] T 68ba3e13cf3a42c68413db7c0f366abd P df32c007eff84cacb8eade269503bf10: No bootstrap required, opened a new log
I20260926 23:33:58.201031 14808 ts_tablet_manager.cc:1397] T 68ba3e13cf3a42c68413db7c0f366abd P df32c007eff84cacb8eade269503bf10: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260926 23:33:58.210490 14799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:58.200348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37519 (local address 127.0.97.1:40569)
0926 23:33:58.200610 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:58.200617 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:58.201193 (+ 576us) server_negotiation.cc:407] Connection header received
0926 23:33:58.201868 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:58.201876 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:58.201999 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:58.202118 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:33:58.203133 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.204464 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.207235 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.208493 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.208634 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:58.208752 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:58.210160 (+ 1408us) server_negotiation.cc:299] Negotiation successful
0926 23:33:58.210290 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 23:33:58.210793 14812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:58.199772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40569 (local address 127.0.97.2:37519)
0926 23:33:58.201071 (+ 1299us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:58.201093 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:33:58.201612 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:33:58.202380 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:33:58.202385 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:33:58.202417 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:33:58.202920 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:58.202937 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.204635 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.204641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:58.207032 (+ 2391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:58.207053 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.208968 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.208976 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:58.209205 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:58.209251 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 23:33:58.209347 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 23:33:58.209505 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1122,"thread_start_us":915,"threads_started":1}
I20260926 23:33:58.211941 14472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ba3e13cf3a42c68413db7c0f366abd" candidate_uuid: "089dde0b70c247128b9290f7b46e474e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" is_pre_election: true
I20260926 23:33:58.212560 14472 raft_consensus.cc:2454] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 089dde0b70c247128b9290f7b46e474e in term 0.
I20260926 23:33:58.212491 14808 raft_consensus.cc:348] T 68ba3e13cf3a42c68413db7c0f366abd P df32c007eff84cacb8eade269503bf10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "df32c007eff84cacb8eade269503bf10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40287 } } peers { permanent_uuid: "089dde0b70c247128b9290f7b46e474e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38991 } }
I20260926 23:33:58.213124 14808 raft_consensus.cc:374] T 68ba3e13cf3a42c68413db7c0f366abd P df32c007eff84cacb8eade269503bf10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:33:58.213255 14808 raft_consensus.cc:729] T 68ba3e13cf3a42c68413db7c0f366abd P df32c007eff84cacb8eade269503bf10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df32c007eff84cacb8eade269503bf10, State: Initialized, Role: FOLLOWER
I20260926 23:33:58.213395 14540 leader_election.cc:302] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 089dde0b70c247128b9290f7b46e474e, be4b5734244b4b418e21dc6e65bb9bbb; no voters:
I20260926 23:33:58.214273 14808 consensus_queue.cc:260] T 68ba3e13cf3a42c68413db7c0f366abd P df32c007eff84cacb8eade269503bf10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "df32c007eff84cacb8eade269503bf10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40287 } } peers { permanent_uuid: "089dde0b70c247128b9290f7b46e474e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38991 } }
I20260926 23:33:58.214828 14809 raft_consensus.cc:2781] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:33:58.215039 14809 raft_consensus.cc:482] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:33:58.215164 14809 raft_consensus.cc:3037] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:33:58.219218 14809 raft_consensus.cc:504] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "df32c007eff84cacb8eade269503bf10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40287 } } peers { permanent_uuid: "089dde0b70c247128b9290f7b46e474e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38991 } }
I20260926 23:33:58.219765 14785 heartbeater.cc:503] Master 127.0.97.62:37995 was elected leader, sending a full tablet report...
I20260926 23:33:58.220168 14809 leader_election.cc:288] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e [CANDIDATE]: Term 1 election: Requested vote from peers be4b5734244b4b418e21dc6e65bb9bbb (127.0.97.1:40569), df32c007eff84cacb8eade269503bf10 (127.0.97.3:40287)
I20260926 23:33:58.220904 14808 ts_tablet_manager.cc:1428] T 68ba3e13cf3a42c68413db7c0f366abd P df32c007eff84cacb8eade269503bf10: Time spent starting tablet: real 0.020s user 0.013s sys 0.007s
I20260926 23:33:58.221577 14803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:58.201387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40613 (local address 127.0.97.3:40287)
0926 23:33:58.201660 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:58.201668 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:58.207953 (+ 6285us) server_negotiation.cc:407] Connection header received
0926 23:33:58.208256 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:58.208263 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:58.208360 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:58.208424 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:33:58.209331 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.210533 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.213144 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.214414 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.214533 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:58.214648 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:58.221072 (+ 6424us) server_negotiation.cc:299] Negotiation successful
0926 23:33:58.221280 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 23:33:58.221598 14813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:58.201216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40287 (local address 127.0.97.2:40613)
0926 23:33:58.207832 (+ 6616us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:58.207856 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:33:58.208082 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:33:58.208655 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:33:58.208660 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:33:58.208689 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:33:58.209123 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:58.209134 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.210858 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.210864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:58.212893 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:33:58.212913 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.220585 (+ 7672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.220593 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:33:58.220849 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:58.220895 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 23:33:58.221014 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 23:33:58.221207 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6452,"thread_start_us":6182,"threads_started":1}
I20260926 23:33:58.221609 14472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ba3e13cf3a42c68413db7c0f366abd" candidate_uuid: "089dde0b70c247128b9290f7b46e474e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be4b5734244b4b418e21dc6e65bb9bbb"
I20260926 23:33:58.221886 14472 raft_consensus.cc:3037] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:33:58.223189 14739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ba3e13cf3a42c68413db7c0f366abd" candidate_uuid: "089dde0b70c247128b9290f7b46e474e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df32c007eff84cacb8eade269503bf10"
I20260926 23:33:58.223155 14740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ba3e13cf3a42c68413db7c0f366abd" candidate_uuid: "089dde0b70c247128b9290f7b46e474e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df32c007eff84cacb8eade269503bf10" is_pre_election: true
I20260926 23:33:58.223840 14740 raft_consensus.cc:2454] T 68ba3e13cf3a42c68413db7c0f366abd P df32c007eff84cacb8eade269503bf10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 089dde0b70c247128b9290f7b46e474e in term 0.
I20260926 23:33:58.225730 14472 raft_consensus.cc:2454] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 089dde0b70c247128b9290f7b46e474e in term 1.
I20260926 23:33:58.226249 14540 leader_election.cc:302] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 089dde0b70c247128b9290f7b46e474e, be4b5734244b4b418e21dc6e65bb9bbb; no voters: df32c007eff84cacb8eade269503bf10
I20260926 23:33:58.226639 14809 raft_consensus.cc:2781] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:33:58.227286 14809 raft_consensus.cc:686] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e [term 1 LEADER]: Becoming Leader. State: Replica: 089dde0b70c247128b9290f7b46e474e, State: Running, Role: LEADER
I20260926 23:33:58.227855 14809 consensus_queue.cc:237] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "df32c007eff84cacb8eade269503bf10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40287 } } peers { permanent_uuid: "089dde0b70c247128b9290f7b46e474e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38991 } }
I20260926 23:33:58.232649 14328 catalog_manager.cc:5495] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e reported cstate change: term changed from 0 to 1, leader changed from <none> to 089dde0b70c247128b9290f7b46e474e (127.0.97.2). New cstate: current_term: 1 leader_uuid: "089dde0b70c247128b9290f7b46e474e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df32c007eff84cacb8eade269503bf10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "089dde0b70c247128b9290f7b46e474e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38991 } health_report { overall_health: HEALTHY } } }
I20260926 23:33:58.273264 14801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:58.267165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43816 (local address 127.0.97.2:38991)
0926 23:33:58.267384 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:58.267390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:58.267409 (+ 19us) server_negotiation.cc:407] Connection header received
0926 23:33:58.267623 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:58.267628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:58.267742 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:58.267886 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:33:58.268374 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.269138 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.270728 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.271168 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.271299 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:58.271880 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:33:58.272600 (+ 720us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:33:58.272731 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:58.272840 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 23:33:58.272927 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 23:33:58.283885 14739 raft_consensus.cc:3037] T 68ba3e13cf3a42c68413db7c0f366abd P df32c007eff84cacb8eade269503bf10 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:33:58.284164 14472 raft_consensus.cc:1261] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb [term 1 FOLLOWER]: Refusing update from remote peer 089dde0b70c247128b9290f7b46e474e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:33:58.285167 14809 consensus_queue.cc:1035] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e [LEADER]: Connected to new peer: Peer: permanent_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:33:58.286945 14739 raft_consensus.cc:1261] T 68ba3e13cf3a42c68413db7c0f366abd P df32c007eff84cacb8eade269503bf10 [term 1 FOLLOWER]: Refusing update from remote peer 089dde0b70c247128b9290f7b46e474e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:33:58.287901 14809 consensus_queue.cc:1035] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df32c007eff84cacb8eade269503bf10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 23:33:58.290340 14786 tablet.cc:2369] T 68ba3e13cf3a42c68413db7c0f366abd P df32c007eff84cacb8eade269503bf10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:33:58.298497 14818 mvcc.cc:205] Tried to move back new op lower bound from 7333747254398640128 to 7333747254185615360. Current Snapshot: MvccSnapshot[applied={T|T < 7333747254398640128}]
I20260926 23:33:58.325706 14327 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:33:58.326241 14327 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:33:58.331710 14327 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5: Sending DeleteTablet for 3 replicas of tablet 68ba3e13cf3a42c68413db7c0f366abd
I20260926 23:33:58.333478 14586 tablet_service.cc:1514] Processing DeleteTablet for tablet 68ba3e13cf3a42c68413db7c0f366abd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:33:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43800
I20260926 23:33:58.333727 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 14389
I20260926 23:33:58.333877 14452 tablet_service.cc:1514] Processing DeleteTablet for tablet 68ba3e13cf3a42c68413db7c0f366abd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:33:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47848
I20260926 23:33:58.334757 14720 tablet_service.cc:1514] Processing DeleteTablet for tablet 68ba3e13cf3a42c68413db7c0f366abd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:33:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41646
I20260926 23:33:58.349193 14829 tablet_replica.cc:330] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e: stopping tablet replica
I20260926 23:33:58.350028 14829 raft_consensus.cc:2229] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:33:58.350980 14829 raft_consensus.cc:2258] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:33:58.351994 14831 tablet_replica.cc:330] T 68ba3e13cf3a42c68413db7c0f366abd P df32c007eff84cacb8eade269503bf10: stopping tablet replica
I20260926 23:33:58.352633 14831 raft_consensus.cc:2229] T 68ba3e13cf3a42c68413db7c0f366abd P df32c007eff84cacb8eade269503bf10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:33:58.353336 14831 raft_consensus.cc:2258] T 68ba3e13cf3a42c68413db7c0f366abd P df32c007eff84cacb8eade269503bf10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:33:58.354143 14829 ts_tablet_manager.cc:1905] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:33:58.354260 14828 tablet_replica.cc:330] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb: stopping tablet replica
I20260926 23:33:58.354856 14828 raft_consensus.cc:2229] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:33:58.356514 14831 ts_tablet_manager.cc:1905] T 68ba3e13cf3a42c68413db7c0f366abd P df32c007eff84cacb8eade269503bf10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:33:58.357143 14828 raft_consensus.cc:2258] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 26 23:33:58 dist-test-slave-gwfx krb5kdc[14283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:33:58.362893 14829 ts_tablet_manager.cc:1918] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:33:58.363245 14829 log.cc:1196] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/wal/wals/68ba3e13cf3a42c68413db7c0f366abd
I20260926 23:33:58.363972 14829 ts_tablet_manager.cc:1939] T 68ba3e13cf3a42c68413db7c0f366abd P 089dde0b70c247128b9290f7b46e474e: Deleting consensus metadata
I20260926 23:33:58.364674 14831 ts_tablet_manager.cc:1918] T 68ba3e13cf3a42c68413db7c0f366abd P df32c007eff84cacb8eade269503bf10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:33:58.364871 14831 log.cc:1196] T 68ba3e13cf3a42c68413db7c0f366abd P df32c007eff84cacb8eade269503bf10: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/wal/wals/68ba3e13cf3a42c68413db7c0f366abd
I20260926 23:33:58.365413 14831 ts_tablet_manager.cc:1939] T 68ba3e13cf3a42c68413db7c0f366abd P df32c007eff84cacb8eade269503bf10: Deleting consensus metadata
I20260926 23:33:58.366638 14314 catalog_manager.cc:4841] TS 089dde0b70c247128b9290f7b46e474e (127.0.97.2:38991): tablet 68ba3e13cf3a42c68413db7c0f366abd (table test-table [id=15ae46a99afe44afb186b61ec717be96]) successfully deleted
I20260926 23:33:58.369230 14828 ts_tablet_manager.cc:1905] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:33:58.369271 14313 catalog_manager.cc:4841] TS df32c007eff84cacb8eade269503bf10 (127.0.97.3:40287): tablet 68ba3e13cf3a42c68413db7c0f366abd (table test-table [id=15ae46a99afe44afb186b61ec717be96]) successfully deleted
I20260926 23:33:58.369717 14799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:58.335081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47864 (local address 127.0.97.1:40569)
0926 23:33:58.335294 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:58.335302 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:58.346216 (+ 10914us) server_negotiation.cc:407] Connection header received
0926 23:33:58.346903 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:58.346912 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:58.347059 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:58.347153 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:58.349463 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.350796 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.352932 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.353636 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.353777 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:58.360339 (+ 6562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:58.360364 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:58.360388 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:58.360449 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:58.365059 (+ 4610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:58.366277 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:58.366288 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:58.366293 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:58.366481 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:58.367484 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:58.367489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:58.367500 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:58.368055 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:58.368479 (+ 424us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:58.368936 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 23:33:58.369237 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 23:33:58.371456 14512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:33:58.470666 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 14389
W20260926 23:33:58.489780 14313 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:40569 (error 108)
I20260926 23:33:58.490131 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 14523
I20260926 23:33:58.493026 14800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:58.492223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40569 (local address 127.0.0.1:47866)
0926 23:33:58.492470 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:58.492677 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 23:33:58.495003 14313 catalog_manager.cc:4584] TS be4b5734244b4b418e21dc6e65bb9bbb (127.0.97.1:40569): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 68ba3e13cf3a42c68413db7c0f366abd: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40569: connect: Connection refused (error 111)
Sep 26 23:33:58 dist-test-slave-gwfx krb5kdc[14283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:33:58.519476 14801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:58.491500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43826 (local address 127.0.97.2:38991)
0926 23:33:58.491742 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:58.491749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:58.491778 (+ 29us) server_negotiation.cc:407] Connection header received
0926 23:33:58.492671 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:58.492678 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:58.492794 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:58.492893 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:58.494870 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.496589 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.498818 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.499543 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.499679 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:58.511176 (+ 11497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:58.511208 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:58.511236 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:58.511311 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:58.515764 (+ 4453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:58.516905 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:58.516917 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:58.516921 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:58.517161 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:58.517643 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:58.517648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:58.517652 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:58.518100 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:58.518452 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:58.518760 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 23:33:58.519025 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260926 23:33:58.520588 14646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:33:58.549045 14800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:58.548457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40569 (local address 127.0.0.1:47876)
0926 23:33:58.548751 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:58.548861 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260926 23:33:58.550246 14800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:58.549848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40569 (local address 127.0.0.1:47878)
0926 23:33:58.549985 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:58.550063 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 23:33:58.611308 14800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:58.610641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40569 (local address 127.0.0.1:47884)
0926 23:33:58.610922 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:58.611125 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260926 23:33:58.612548 14800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:58.612086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40569 (local address 127.0.0.1:47896)
0926 23:33:58.612291 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:58.612373 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 23:33:58.650563 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 14523
I20260926 23:33:58.674226 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 14657
Sep 26 23:33:58 dist-test-slave-gwfx krb5kdc[14283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:33:58.691936 14803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:58.675195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41656 (local address 127.0.97.3:40287)
0926 23:33:58.675408 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 23:33:58.675415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:33:58.675497 (+ 82us) server_negotiation.cc:407] Connection header received
0926 23:33:58.675723 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:33:58.675729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:33:58.675824 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:33:58.675916 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:33:58.677275 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.678187 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.679714 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:33:58.680178 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:33:58.680327 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:33:58.685217 (+ 4890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:33:58.685239 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:33:58.685259 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:33:58.685310 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:33:58.688655 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:58.689558 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:58.689568 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:58.689571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:58.689737 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:33:58.690129 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:33:58.690133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:33:58.690135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:33:58.690588 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:33:58.690959 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0926 23:33:58.691290 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 23:33:58.691585 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 23:33:58.692749 14780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:33:58.693362 14800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:58.692750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40569 (local address 127.0.0.1:47900)
0926 23:33:58.692992 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:58.693141 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 23:33:58.694730 14800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:33:58.694175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40569 (local address 127.0.0.1:47904)
0926 23:33:58.694395 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 23:33:58.694519 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260926 23:33:58.792065 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 14657
I20260926 23:33:58.808354 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 14296
I20260926 23:33:58.809854 14358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:33:58.938810 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 14296
I20260926 23:33:58.961082 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37995
--webserver_interface=127.0.97.62
--webserver_port=39459
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:33:59.376430 14837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:33:59.376868 14837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:33:59.377019 14837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:33:59.385569 14837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:33:59.385761 14837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:33:59.385849 14837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:33:59.385931 14837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:33:59.385998 14837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:33:59.398787 14837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=39459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:33:59.400786 14837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:33:59.402968 14837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:33:59.433506 14843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:33:59.433506 14846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:33:59.433517 14844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:33:59.434607 14837 server_base.cc:1034] running on GCE node
I20260926 23:33:59.435815 14837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:33:59.438133 14837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:33:59.439424 14837 hybrid_clock.cc:648] HybridClock initialized: now 1790465639439348 us; error 73 us; skew 500 ppm
I20260926 23:33:59.439946 14837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:33:59 dist-test-slave-gwfx krb5kdc[14283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:33:59.448946 14837 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:33:59.451231 14837 webserver.cc:460] Webserver started at http://127.0.97.62:39459/ using document root <none> and password file <none>
I20260926 23:33:59.452214 14837 fs_manager.cc:359] Metadata directory not provided
I20260926 23:33:59.452374 14837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:33:59.460299 14837 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260926 23:33:59.464396 14853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:33:59.466583 14837 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260926 23:33:59.466926 14837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "8255fe64fe7f4c0ca60dc6aa921e1bc5"
format_stamp: "Formatted at 2026-09-26 23:33:55 on dist-test-slave-gwfx"
server_key: "e866be79e2b94da75fd8e01ad40010f1"
server_key_iv: "8f125005c3e916e784eb75b31cadb717"
server_key_version: "encryptionkey@0"
I20260926 23:33:59.468348 14837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:33:59.514731 14837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:33:59.520452 14837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:33:59.520792 14837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:33:59.543231 14837 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37995
I20260926 23:33:59.543256 14904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37995 every 8 connection(s)
I20260926 23:33:59.545162 14837 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:33:59.553575 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 14837
I20260926 23:33:59.554119 14905 sys_catalog.cc:263] Verifying existing consensus state
I20260926 23:33:59.554558 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:40569
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37995
--webserver_port=38349
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:33:59.558580 14905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5: Bootstrap starting.
I20260926 23:33:59.589395 14905 log.cc:824] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5: Log is configured to *not* fsync() on all Append() calls
I20260926 23:33:59.609092 14905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 23:33:59.609949 14905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5: Bootstrap complete.
I20260926 23:33:59.625077 14905 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8255fe64fe7f4c0ca60dc6aa921e1bc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37995 } }
I20260926 23:33:59.626546 14905 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8255fe64fe7f4c0ca60dc6aa921e1bc5, State: Initialized, Role: FOLLOWER
I20260926 23:33:59.627493 14905 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8255fe64fe7f4c0ca60dc6aa921e1bc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37995 } }
I20260926 23:33:59.627838 14905 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:33:59.627982 14905 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:33:59.628197 14905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [term 1 FOLLOWER]: Advancing to term 2
I20260926 23:33:59.635762 14905 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8255fe64fe7f4c0ca60dc6aa921e1bc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37995 } }
I20260926 23:33:59.636649 14905 leader_election.cc:302] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8255fe64fe7f4c0ca60dc6aa921e1bc5; no voters:
I20260926 23:33:59.637457 14905 leader_election.cc:288] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 23:33:59.637639 14909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 23:33:59.639026 14909 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [term 2 LEADER]: Becoming Leader. State: Replica: 8255fe64fe7f4c0ca60dc6aa921e1bc5, State: Running, Role: LEADER
I20260926 23:33:59.639690 14909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8255fe64fe7f4c0ca60dc6aa921e1bc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37995 } }
I20260926 23:33:59.640156 14905 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:33:59.642517 14910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8255fe64fe7f4c0ca60dc6aa921e1bc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8255fe64fe7f4c0ca60dc6aa921e1bc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37995 } } }
I20260926 23:33:59.643013 14910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [sys.catalog]: This master's current role is: LEADER
I20260926 23:33:59.646390 14914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:33:59.646196 14911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8255fe64fe7f4c0ca60dc6aa921e1bc5. Latest consensus state: current_term: 2 leader_uuid: "8255fe64fe7f4c0ca60dc6aa921e1bc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8255fe64fe7f4c0ca60dc6aa921e1bc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37995 } } }
I20260926 23:33:59.646730 14911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [sys.catalog]: This master's current role is: LEADER
I20260926 23:33:59.656262 14914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:33:59.658099 14914 catalog_manager.cc:1259] Loaded cluster ID: 74c8e7b904974b18a48bd8f4729db3bb
I20260926 23:33:59.658349 14914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:33:59.662065 14914 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 23:33:59.681932 14914 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 23:33:59.703393 14914 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:33:59.705019 14914 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5: Loaded TSK: 0
I20260926 23:33:59.706497 14914 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 23:34:00.026229 14907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:34:00.026712 14907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:34:00.026880 14907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:34:00.036242 14907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:34:00.036448 14907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:34:00.036620 14907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:34:00.049436 14907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:40569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38349
--tserver_master_addrs=127.0.97.62:37995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:34:00.051525 14907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:34:00.053311 14907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:34:00.071372 14933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:34:00.071424 14935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:34:00.071367 14932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:34:00.073684 14907 server_base.cc:1034] running on GCE node
I20260926 23:34:00.074484 14907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:34:00.075793 14907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:34:00.077023 14907 hybrid_clock.cc:648] HybridClock initialized: now 1790465640076969 us; error 93 us; skew 500 ppm
I20260926 23:34:00.077451 14907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:34:00 dist-test-slave-gwfx krb5kdc[14283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:34:00.081243 14907 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:34:00.083273 14907 webserver.cc:460] Webserver started at http://127.0.97.1:38349/ using document root <none> and password file <none>
I20260926 23:34:00.084244 14907 fs_manager.cc:359] Metadata directory not provided
I20260926 23:34:00.084404 14907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:34:00.092156 14907 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260926 23:34:00.097046 14942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:00.099776 14907 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:34:00.100031 14907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "be4b5734244b4b418e21dc6e65bb9bbb"
format_stamp: "Formatted at 2026-09-26 23:33:56 on dist-test-slave-gwfx"
server_key: "a12e27122c093abc0b0e4cdbfb3ae8f8"
server_key_iv: "edb1145c0d529d437085a9da2d2e7d7f"
server_key_version: "encryptionkey@0"
I20260926 23:34:00.101342 14907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:34:00.120609 14907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:34:00.126482 14907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:34:00.126880 14907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:34:00.128293 14907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:34:00.130628 14949 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260926 23:34:00.138253 14907 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 23:34:00.138465 14907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.001s sys 0.000s
I20260926 23:34:00.138633 14907 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 23:34:00.143071 14907 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 23:34:00.143216 14907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.005s sys 0.000s
I20260926 23:34:00.143700 14949 tablet_bootstrap.cc:492] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb: Bootstrap starting.
I20260926 23:34:00.156356 14949 log.cc:824] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb: Log is configured to *not* fsync() on all Append() calls
I20260926 23:34:00.181967 14949 tablet_bootstrap.cc:492] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 23:34:00.182885 14949 tablet_bootstrap.cc:492] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb: Bootstrap complete.
I20260926 23:34:00.183658 14949 ts_tablet_manager.cc:1397] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb: Time spent bootstrapping tablet: real 0.040s user 0.029s sys 0.008s
I20260926 23:34:00.184468 14907 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40569
I20260926 23:34:00.184486 15056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40569 every 8 connection(s)
I20260926 23:34:00.186328 14907 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:34:00.191851 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 14907
I20260926 23:34:00.192655 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:38991
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37995
--webserver_port=37053
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:34:00.194146 14949 raft_consensus.cc:348] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "df32c007eff84cacb8eade269503bf10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40287 } } peers { permanent_uuid: "089dde0b70c247128b9290f7b46e474e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38991 } }
I20260926 23:34:00.195683 14949 raft_consensus.cc:729] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: be4b5734244b4b418e21dc6e65bb9bbb, State: Initialized, Role: FOLLOWER
I20260926 23:34:00.196693 14949 consensus_queue.cc:260] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "df32c007eff84cacb8eade269503bf10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40287 } } peers { permanent_uuid: "089dde0b70c247128b9290f7b46e474e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38991 } }
I20260926 23:34:00.198745 14949 ts_tablet_manager.cc:1428] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb: Time spent starting tablet: real 0.015s user 0.007s sys 0.009s
Sep 26 23:34:00 dist-test-slave-gwfx krb5kdc[14283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:34:00.220275 15060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:00.190263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.1:54681)
0926 23:34:00.191402 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:00.191463 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 23:34:00.193421 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:34:00.195996 (+ 2575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:34:00.196010 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:34:00.197125 (+ 1115us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:34:00.198258 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:00.198296 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:00.200793 (+ 2497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:00.200799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:00.202863 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:00.202890 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:00.204711 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:00.204718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:00.204968 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:00.205903 (+ 935us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:34:00.205934 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:34:00.209276 (+ 3342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:34:00.214428 (+ 5152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:34:00.214440 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:34:00.214457 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:34:00.215262 (+ 805us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:34:00.215982 (+ 720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:34:00.215987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:34:00.215990 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:34:00.216306 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:34:00.217526 (+ 1220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:34:00.217535 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:34:00.217941 (+ 406us) client_negotiation.cc:769] Sending connection context
0926 23:34:00.218248 (+ 307us) client_negotiation.cc:240] Negotiation successful
0926 23:34:00.218842 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"thread_start_us":501,"threads_started":1}
I20260926 23:34:00.220549 15059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:00.190144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54681 (local address 127.0.97.62:37995)
0926 23:34:00.190880 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:00.190904 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:00.192176 (+ 1272us) server_negotiation.cc:407] Connection header received
0926 23:34:00.194332 (+ 2156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:00.194356 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:00.195026 (+ 670us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:00.195608 (+ 582us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:34:00.198627 (+ 3019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:00.200513 (+ 1886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:00.203133 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:00.204428 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:00.204683 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:00.209748 (+ 5065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:34:00.209763 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:34:00.209790 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:34:00.209877 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:34:00.214138 (+ 4261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:00.215484 (+ 1346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:00.215495 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:00.215505 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:00.215815 (+ 310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:00.216454 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:00.216459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:00.216463 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:00.217338 (+ 875us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:34:00.217577 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:00.218640 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0926 23:34:00.219249 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":283,"threads_started":1}
I20260926 23:34:00.224483 15057 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37995
I20260926 23:34:00.224908 15057 heartbeater.cc:464] Registering TS with master...
I20260926 23:34:00.225814 15057 heartbeater.cc:511] Master 127.0.97.62:37995 requested a full tablet report, sending...
I20260926 23:34:00.230310 14870 ts_manager.cc:194] Registered new tserver with Master: be4b5734244b4b418e21dc6e65bb9bbb (127.0.97.1:40569)
I20260926 23:34:00.236932 14870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54681
I20260926 23:34:00.242122 15057 heartbeater.cc:503] Master 127.0.97.62:37995 was elected leader, sending a full tablet report...
I20260926 23:34:00.248628 15063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:00.235932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40569 (local address 127.0.0.1:47920)
0926 23:34:00.238984 (+ 3052us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:00.239039 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 23:34:00.239261 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:34:00.240973 (+ 1712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:34:00.240982 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:34:00.241026 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:34:00.241454 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:00.241473 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:00.243713 (+ 2240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:00.243721 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:00.245673 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:00.245690 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:00.247637 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:00.247644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:00.247822 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:00.247874 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 23:34:00.248005 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 23:34:00.248256 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2636,"thread_start_us":2499,"threads_started":1}
I20260926 23:34:00.248713 15064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:00.236225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47920 (local address 127.0.97.1:40569)
0926 23:34:00.240048 (+ 3823us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:00.240057 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:00.240096 (+ 39us) server_negotiation.cc:407] Connection header received
0926 23:34:00.240203 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:00.240216 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:00.240669 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:00.240898 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:34:00.241710 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:00.243539 (+ 1829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:00.245848 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:00.247391 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:00.247539 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:00.247802 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:00.248066 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 23:34:00.248245 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":3582,"thread_start_us":3349,"threads_started":1}
I20260926 23:34:00.250566 14992 tablet_service.cc:1514] Processing DeleteTablet for tablet 68ba3e13cf3a42c68413db7c0f366abd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:33:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47920
I20260926 23:34:00.253065 15065 tablet_replica.cc:330] stopping tablet replica
I20260926 23:34:00.253521 15065 raft_consensus.cc:2229] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:34:00.253969 15065 raft_consensus.cc:2258] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:34:00.255541 15065 ts_tablet_manager.cc:1905] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:34:00.264917 15065 ts_tablet_manager.cc:1918] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:34:00.265132 15065 log.cc:1196] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/wal/wals/68ba3e13cf3a42c68413db7c0f366abd
I20260926 23:34:00.265779 15065 ts_tablet_manager.cc:1939] T 68ba3e13cf3a42c68413db7c0f366abd P be4b5734244b4b418e21dc6e65bb9bbb: Deleting consensus metadata
I20260926 23:34:00.268911 14855 catalog_manager.cc:4841] TS be4b5734244b4b418e21dc6e65bb9bbb (127.0.97.1:40569): tablet 68ba3e13cf3a42c68413db7c0f366abd (table test-table [id=15ae46a99afe44afb186b61ec717be96]) successfully deleted
W20260926 23:34:00.630301 15062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:34:00.630713 15062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:34:00.630851 15062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:34:00.639292 15062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:34:00.639462 15062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:34:00.639598 15062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:34:00.651253 15062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:38991
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=37053
--tserver_master_addrs=127.0.97.62:37995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:34:00.653000 15062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:34:00.654596 15062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:34:00.671489 15070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:34:00.671571 15073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:34:00.671531 15071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:34:00.673295 15062 server_base.cc:1034] running on GCE node
I20260926 23:34:00.674219 15062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:34:00.675567 15062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:34:00.676826 15062 hybrid_clock.cc:648] HybridClock initialized: now 1790465640676755 us; error 55 us; skew 500 ppm
I20260926 23:34:00.677261 15062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:34:00 dist-test-slave-gwfx krb5kdc[14283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:34:00.681110 15062 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:34:00.683116 15062 webserver.cc:460] Webserver started at http://127.0.97.2:37053/ using document root <none> and password file <none>
I20260926 23:34:00.684012 15062 fs_manager.cc:359] Metadata directory not provided
I20260926 23:34:00.684150 15062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:34:00.691394 15062 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 23:34:00.695226 15080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:00.697304 15062 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 23:34:00.697589 15062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "089dde0b70c247128b9290f7b46e474e"
format_stamp: "Formatted at 2026-09-26 23:33:57 on dist-test-slave-gwfx"
server_key: "56ba41fb43d5d957ff4794d82d9b133d"
server_key_iv: "d312e17fb53e303e073b1923be780858"
server_key_version: "encryptionkey@0"
I20260926 23:34:00.699040 15062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:34:00.736855 15062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:34:00.742748 15062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:34:00.743206 15062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:34:00.744550 15062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:34:00.746358 15062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:34:00.746476 15062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:00.746610 15062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:34:00.746712 15062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:00.783437 15062 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38991
I20260926 23:34:00.783485 15192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38991 every 8 connection(s)
I20260926 23:34:00.785455 15062 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:34:00.789862 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 15062
I20260926 23:34:00.790553 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:40287
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37995
--webserver_port=44293
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:34:00 dist-test-slave-gwfx krb5kdc[14283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:34:00.812934 15195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:00.788975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36369 (local address 127.0.97.62:37995)
0926 23:34:00.789536 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:00.789543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:00.791115 (+ 1572us) server_negotiation.cc:407] Connection header received
0926 23:34:00.793372 (+ 2257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:00.793379 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:00.793489 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:00.793653 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:34:00.795765 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:00.796789 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:00.798734 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:00.799181 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:00.799291 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:00.804924 (+ 5633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:34:00.804932 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:34:00.804935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:34:00.804988 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:34:00.808141 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:00.809369 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:00.809376 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:00.809379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:00.809588 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:00.810259 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:00.810264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:00.810266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:00.810664 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:34:00.810861 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:00.812306 (+ 1445us) server_negotiation.cc:299] Negotiation successful
0926 23:34:00.812588 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":242,"threads_started":1}
I20260926 23:34:00.814981 15196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:00.789744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.2:36369)
0926 23:34:00.790857 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:00.790928 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 23:34:00.793017 (+ 2089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:34:00.793970 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:34:00.793981 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:34:00.794678 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:34:00.795488 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:00.795544 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:00.796992 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:00.796997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:00.798479 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:00.798496 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:00.799480 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:00.799488 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:00.799867 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:00.800771 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:34:00.800806 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:34:00.804505 (+ 3699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:34:00.808425 (+ 3920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:34:00.808436 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:34:00.808457 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:34:00.809114 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:34:00.809774 (+ 660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:34:00.809780 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:34:00.809784 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:34:00.810093 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:34:00.810888 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:34:00.810900 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:34:00.811632 (+ 732us) client_negotiation.cc:769] Sending connection context
0926 23:34:00.812088 (+ 456us) client_negotiation.cc:240] Negotiation successful
0926 23:34:00.812928 (+ 840us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":437,"threads_started":1}
I20260926 23:34:00.817346 15193 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37995
I20260926 23:34:00.817893 15193 heartbeater.cc:464] Registering TS with master...
I20260926 23:34:00.819254 15193 heartbeater.cc:511] Master 127.0.97.62:37995 requested a full tablet report, sending...
I20260926 23:34:00.821835 14870 ts_manager.cc:194] Registered new tserver with Master: 089dde0b70c247128b9290f7b46e474e (127.0.97.2:38991)
I20260926 23:34:00.823254 14870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36369
W20260926 23:34:01.206954 15197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:34:01.207616 15197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:34:01.207757 15197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:34:01.216526 15197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:34:01.216707 15197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:34:01.216840 15197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:34:01.228161 15197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:40287
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=44293
--tserver_master_addrs=127.0.97.62:37995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:34:01.229888 15197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:34:01.231549 15197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:34:01.247185 15202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:34:01.247215 15203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:34:01.248140 15205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:34:01.249632 15197 server_base.cc:1034] running on GCE node
I20260926 23:34:01.250415 15197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:34:01.251679 15197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:34:01.252918 15197 hybrid_clock.cc:648] HybridClock initialized: now 1790465641252851 us; error 53 us; skew 500 ppm
I20260926 23:34:01.253278 15197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:34:01 dist-test-slave-gwfx krb5kdc[14283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:34:01.257126 15197 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:34:01.258996 15197 webserver.cc:460] Webserver started at http://127.0.97.3:44293/ using document root <none> and password file <none>
I20260926 23:34:01.259938 15197 fs_manager.cc:359] Metadata directory not provided
I20260926 23:34:01.260085 15197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:34:01.267056 15197 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.003s
I20260926 23:34:01.270578 15212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:01.272667 15197 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260926 23:34:01.272908 15197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "df32c007eff84cacb8eade269503bf10"
format_stamp: "Formatted at 2026-09-26 23:33:57 on dist-test-slave-gwfx"
server_key: "777ce6dacc53d96475b6e28ff1d9a4c9"
server_key_iv: "89849c912d646464ac21536d4add928e"
server_key_version: "encryptionkey@0"
I20260926 23:34:01.274253 15197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:34:01.308532 15197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:34:01.314885 15197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:34:01.315236 15197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:34:01.316653 15197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:34:01.318465 15197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:34:01.318586 15197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:01.318742 15197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:34:01.318819 15197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:01.356498 15197 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40287
I20260926 23:34:01.356542 15324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40287 every 8 connection(s)
I20260926 23:34:01.358250 15197 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:34:01.364292 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 15197
Sep 26 23:34:01 dist-test-slave-gwfx krb5kdc[14283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:34:01.383060 15330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:01.365328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33284 (local address 127.0.97.62:37995)
0926 23:34:01.367262 (+ 1934us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:01.367267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:01.367364 (+ 97us) server_negotiation.cc:407] Connection header received
0926 23:34:01.367431 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:01.367435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:01.367530 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:01.367654 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:34:01.369478 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:01.370369 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:01.372134 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:01.372543 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:01.372634 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:01.375951 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:34:01.375959 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:34:01.375963 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:34:01.376015 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:34:01.379516 (+ 3501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:01.380624 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:01.380632 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:01.380635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:01.380810 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:01.381308 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:01.381313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:01.381315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:01.381713 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:34:01.382025 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:01.382330 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 23:34:01.382594 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1720,"thread_start_us":1551,"threads_started":1}
I20260926 23:34:01.384174 15327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:01.361342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33885 (local address 127.0.97.62:37995)
0926 23:34:01.361943 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:01.361951 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:01.363093 (+ 1142us) server_negotiation.cc:407] Connection header received
0926 23:34:01.365386 (+ 2293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:01.365393 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:01.365529 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:01.366805 (+ 1276us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:34:01.368213 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:01.369068 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:01.371322 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:01.371772 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:01.371873 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:01.376800 (+ 4927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:34:01.376805 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:34:01.376809 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:34:01.376859 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:34:01.380319 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:01.381468 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:01.381474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:01.381478 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:01.381620 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:01.382119 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:01.382122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:01.382124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:01.382386 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:34:01.382518 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:01.383623 (+ 1105us) server_negotiation.cc:299] Negotiation successful
0926 23:34:01.383854 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":276,"threads_started":1}
I20260926 23:34:01.386198 15328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:01.361983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.3:33885)
0926 23:34:01.362889 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:01.362936 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 23:34:01.364983 (+ 2047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:34:01.366091 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:34:01.366107 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:34:01.367060 (+ 953us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:34:01.367974 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:01.368011 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:01.369271 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:01.369277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:01.371140 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:01.371162 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:01.372037 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:01.372046 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:01.372324 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:01.373337 (+ 1013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:34:01.373375 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:34:01.376456 (+ 3081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:34:01.380503 (+ 4047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:34:01.380513 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:34:01.380533 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:34:01.381312 (+ 779us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:34:01.381744 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:34:01.381750 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:34:01.381754 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:34:01.381988 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:34:01.382521 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:34:01.382531 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:34:01.383102 (+ 571us) client_negotiation.cc:769] Sending connection context
0926 23:34:01.383499 (+ 397us) client_negotiation.cc:240] Negotiation successful
0926 23:34:01.384643 (+ 1144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":318,"threads_started":1}
I20260926 23:34:01.388080 15325 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37995
I20260926 23:34:01.388482 15325 heartbeater.cc:464] Registering TS with master...
I20260926 23:34:01.389410 15325 heartbeater.cc:511] Master 127.0.97.62:37995 requested a full tablet report, sending...
I20260926 23:34:01.391450 14870 ts_manager.cc:194] Registered new tserver with Master: df32c007eff84cacb8eade269503bf10 (127.0.97.3:40287)
I20260926 23:34:01.392418 14870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33885
I20260926 23:34:01.398885 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:34:01.407953 15327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:01.400007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33298 (local address 127.0.97.62:37995)
0926 23:34:01.400239 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:01.400246 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:01.400784 (+ 538us) server_negotiation.cc:407] Connection header received
0926 23:34:01.401017 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:01.401023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:01.401263 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:01.401415 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:34:01.401879 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:01.402786 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:01.404870 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:01.405426 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:01.405564 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:01.406302 (+ 738us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:34:01.407182 (+ 880us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:34:01.407312 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:01.407501 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 23:34:01.407603 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 23:34:01.409312 14870 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:33298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:34:01.418638 14870 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:34:01.449126 14992 tablet_service.cc:1467] Processing CreateTablet for tablet 84b58489f5294e10b665e772b7b2dfd9 (DEFAULT_TABLE table=test-table [id=3c8cff34147c46ba9d896225929d14ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:34:01.450322 14992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84b58489f5294e10b665e772b7b2dfd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:34:01.456997 15337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:01.432971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41664 (local address 127.0.97.3:40287)
0926 23:34:01.447616 (+ 14645us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:01.447626 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:01.447827 (+ 201us) server_negotiation.cc:407] Connection header received
0926 23:34:01.447996 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:01.448007 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:01.448407 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:01.449298 (+ 891us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:34:01.449645 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:01.451366 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:01.453911 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:01.455464 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:01.455624 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:01.456036 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:01.456374 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 23:34:01.456643 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14452,"thread_start_us":14226,"threads_started":1}
I20260926 23:34:01.456997 15338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:01.432501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40287 (local address 127.0.0.1:41664)
0926 23:34:01.445570 (+ 13069us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:01.445617 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 23:34:01.445918 (+ 301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:34:01.448687 (+ 2769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:34:01.448696 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:34:01.448728 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:34:01.449372 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:01.449395 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:01.451621 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:01.451629 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:01.453309 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:01.453328 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:01.455715 (+ 2387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:01.455722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:01.456028 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:01.456146 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 23:34:01.456256 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 23:34:01.456446 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12816,"mutex_wait_us":3,"thread_start_us":14612,"threads_started":1}
I20260926 23:34:01.457979 15336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:01.433038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38991 (local address 127.0.0.1:43842)
0926 23:34:01.447474 (+ 14436us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:01.447501 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 23:34:01.447792 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:34:01.449032 (+ 1240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:34:01.449037 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:34:01.449063 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:34:01.449948 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:01.449964 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:01.452013 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:01.452019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:01.454534 (+ 2515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:01.454554 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:01.457008 (+ 2454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:01.457017 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:01.457218 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:01.457254 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 23:34:01.457369 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 23:34:01.457541 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14274,"mutex_wait_us":33,"thread_start_us":13028,"threads_started":1}
I20260926 23:34:01.458462 15339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:01.433229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43842 (local address 127.0.97.2:38991)
0926 23:34:01.446972 (+ 13743us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:01.446985 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:01.447691 (+ 706us) server_negotiation.cc:407] Connection header received
0926 23:34:01.448088 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:01.448097 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:01.448407 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:01.448668 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:34:01.450192 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:01.451796 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:01.454825 (+ 3029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:01.456419 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:01.456576 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:01.456794 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:01.457877 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0926 23:34:01.458116 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13551,"thread_start_us":13356,"threads_started":1}
I20260926 23:34:01.461457 15340 tablet_bootstrap.cc:492] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb: Bootstrap starting.
I20260926 23:34:01.463760 15340 tablet_bootstrap.cc:654] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb: Neither blocks nor log segments found. Creating new log.
I20260926 23:34:01.464133 15128 tablet_service.cc:1467] Processing CreateTablet for tablet 84b58489f5294e10b665e772b7b2dfd9 (DEFAULT_TABLE table=test-table [id=3c8cff34147c46ba9d896225929d14ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:34:01.464136 15260 tablet_service.cc:1467] Processing CreateTablet for tablet 84b58489f5294e10b665e772b7b2dfd9 (DEFAULT_TABLE table=test-table [id=3c8cff34147c46ba9d896225929d14ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:34:01.466199 15128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84b58489f5294e10b665e772b7b2dfd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:34:01.466184 15260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84b58489f5294e10b665e772b7b2dfd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:34:01.467376 15340 tablet_bootstrap.cc:492] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb: No bootstrap required, opened a new log
I20260926 23:34:01.467597 15340 ts_tablet_manager.cc:1397] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 23:34:01.468662 15340 raft_consensus.cc:348] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df32c007eff84cacb8eade269503bf10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40287 } } peers { permanent_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "089dde0b70c247128b9290f7b46e474e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38991 } }
I20260926 23:34:01.468972 15340 raft_consensus.cc:374] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:34:01.469142 15340 raft_consensus.cc:729] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be4b5734244b4b418e21dc6e65bb9bbb, State: Initialized, Role: FOLLOWER
I20260926 23:34:01.469573 15340 consensus_queue.cc:260] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df32c007eff84cacb8eade269503bf10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40287 } } peers { permanent_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "089dde0b70c247128b9290f7b46e474e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38991 } }
I20260926 23:34:01.473459 15340 ts_tablet_manager.cc:1428] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260926 23:34:01.486526 15343 tablet_bootstrap.cc:492] T 84b58489f5294e10b665e772b7b2dfd9 P 089dde0b70c247128b9290f7b46e474e: Bootstrap starting.
I20260926 23:34:01.486526 15344 tablet_bootstrap.cc:492] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10: Bootstrap starting.
I20260926 23:34:01.492285 15343 tablet_bootstrap.cc:654] T 84b58489f5294e10b665e772b7b2dfd9 P 089dde0b70c247128b9290f7b46e474e: Neither blocks nor log segments found. Creating new log.
I20260926 23:34:01.492343 15344 tablet_bootstrap.cc:654] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10: Neither blocks nor log segments found. Creating new log.
I20260926 23:34:01.494414 15344 log.cc:824] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10: Log is configured to *not* fsync() on all Append() calls
I20260926 23:34:01.494414 15343 log.cc:824] T 84b58489f5294e10b665e772b7b2dfd9 P 089dde0b70c247128b9290f7b46e474e: Log is configured to *not* fsync() on all Append() calls
I20260926 23:34:01.499047 15344 tablet_bootstrap.cc:492] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10: No bootstrap required, opened a new log
I20260926 23:34:01.499045 15343 tablet_bootstrap.cc:492] T 84b58489f5294e10b665e772b7b2dfd9 P 089dde0b70c247128b9290f7b46e474e: No bootstrap required, opened a new log
I20260926 23:34:01.499492 15344 ts_tablet_manager.cc:1397] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20260926 23:34:01.499493 15343 ts_tablet_manager.cc:1397] T 84b58489f5294e10b665e772b7b2dfd9 P 089dde0b70c247128b9290f7b46e474e: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.003s
I20260926 23:34:01.511684 15344 raft_consensus.cc:348] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df32c007eff84cacb8eade269503bf10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40287 } } peers { permanent_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "089dde0b70c247128b9290f7b46e474e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38991 } }
I20260926 23:34:01.511684 15343 raft_consensus.cc:348] T 84b58489f5294e10b665e772b7b2dfd9 P 089dde0b70c247128b9290f7b46e474e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df32c007eff84cacb8eade269503bf10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40287 } } peers { permanent_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "089dde0b70c247128b9290f7b46e474e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38991 } }
I20260926 23:34:01.512326 15344 raft_consensus.cc:374] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:34:01.512327 15343 raft_consensus.cc:374] T 84b58489f5294e10b665e772b7b2dfd9 P 089dde0b70c247128b9290f7b46e474e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:34:01.512452 15343 raft_consensus.cc:729] T 84b58489f5294e10b665e772b7b2dfd9 P 089dde0b70c247128b9290f7b46e474e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 089dde0b70c247128b9290f7b46e474e, State: Initialized, Role: FOLLOWER
I20260926 23:34:01.512447 15344 raft_consensus.cc:729] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df32c007eff84cacb8eade269503bf10, State: Initialized, Role: FOLLOWER
I20260926 23:34:01.513311 15344 consensus_queue.cc:260] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df32c007eff84cacb8eade269503bf10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40287 } } peers { permanent_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "089dde0b70c247128b9290f7b46e474e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38991 } }
I20260926 23:34:01.513315 15343 consensus_queue.cc:260] T 84b58489f5294e10b665e772b7b2dfd9 P 089dde0b70c247128b9290f7b46e474e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df32c007eff84cacb8eade269503bf10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40287 } } peers { permanent_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "089dde0b70c247128b9290f7b46e474e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38991 } }
I20260926 23:34:01.514861 15325 heartbeater.cc:503] Master 127.0.97.62:37995 was elected leader, sending a full tablet report...
I20260926 23:34:01.515268 15193 heartbeater.cc:503] Master 127.0.97.62:37995 was elected leader, sending a full tablet report...
I20260926 23:34:01.515776 15344 ts_tablet_manager.cc:1428] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260926 23:34:01.515776 15343 ts_tablet_manager.cc:1428] T 84b58489f5294e10b665e772b7b2dfd9 P 089dde0b70c247128b9290f7b46e474e: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
W20260926 23:34:01.538002 15194 tablet.cc:2369] T 84b58489f5294e10b665e772b7b2dfd9 P 089dde0b70c247128b9290f7b46e474e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:34:01.610205 15326 tablet.cc:2369] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:34:01.651072 15348 raft_consensus.cc:482] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:34:01.651472 15348 raft_consensus.cc:504] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df32c007eff84cacb8eade269503bf10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40287 } } peers { permanent_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "089dde0b70c247128b9290f7b46e474e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38991 } }
I20260926 23:34:01.653738 15348 leader_election.cc:288] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be4b5734244b4b418e21dc6e65bb9bbb (127.0.97.1:40569), 089dde0b70c247128b9290f7b46e474e (127.0.97.2:38991)
I20260926 23:34:01.660883 15339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:01.654342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47095 (local address 127.0.97.2:38991)
0926 23:34:01.654591 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:01.654599 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:01.655340 (+ 741us) server_negotiation.cc:407] Connection header received
0926 23:34:01.655506 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:01.655511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:01.655589 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:01.655672 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:34:01.656227 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:01.656971 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:01.658847 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:01.659728 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:01.659870 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:01.659989 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:01.660380 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 23:34:01.660554 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 23:34:01.660910 15328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:01.654023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40569 (local address 127.0.97.3:53161)
0926 23:34:01.654382 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:01.654405 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:34:01.654586 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:34:01.655510 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:34:01.655515 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:34:01.655544 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:34:01.655787 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:01.655801 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:01.657009 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:01.657015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:01.658556 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:01.658575 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:01.659923 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:01.659931 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:01.660182 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:01.660234 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 23:34:01.660327 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 23:34:01.660544 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":68}
I20260926 23:34:01.660924 15349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:01.654254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53161 (local address 127.0.97.1:40569)
0926 23:34:01.655181 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:01.655187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:01.655208 (+ 21us) server_negotiation.cc:407] Connection header received
0926 23:34:01.655288 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:01.655292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:01.655362 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:01.655470 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:34:01.655957 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:01.656856 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:01.658789 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:01.659751 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:01.659877 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:01.660007 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:01.660388 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 23:34:01.660559 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":601,"threads_started":1}
I20260926 23:34:01.661000 15350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:01.654351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38991 (local address 127.0.97.3:47095)
0926 23:34:01.655214 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:01.655229 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:34:01.655362 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:34:01.655708 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:34:01.655716 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:34:01.655742 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:34:01.656062 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:01.656071 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:01.657138 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:01.657142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:01.658650 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:01.658664 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:01.659959 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:01.659965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:01.660181 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:01.660234 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 23:34:01.660326 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 23:34:01.660556 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"mutex_wait_us":49,"thread_start_us":555,"threads_started":1}
I20260926 23:34:01.662249 15148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b58489f5294e10b665e772b7b2dfd9" candidate_uuid: "df32c007eff84cacb8eade269503bf10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "089dde0b70c247128b9290f7b46e474e" is_pre_election: true
I20260926 23:34:01.662245 15012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b58489f5294e10b665e772b7b2dfd9" candidate_uuid: "df32c007eff84cacb8eade269503bf10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" is_pre_election: true
I20260926 23:34:01.662880 15148 raft_consensus.cc:2454] T 84b58489f5294e10b665e772b7b2dfd9 P 089dde0b70c247128b9290f7b46e474e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df32c007eff84cacb8eade269503bf10 in term 0.
I20260926 23:34:01.662942 15012 raft_consensus.cc:2454] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df32c007eff84cacb8eade269503bf10 in term 0.
I20260926 23:34:01.663894 15215 leader_election.cc:302] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 089dde0b70c247128b9290f7b46e474e, df32c007eff84cacb8eade269503bf10; no voters:
I20260926 23:34:01.664474 15348 raft_consensus.cc:2781] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:34:01.664674 15348 raft_consensus.cc:482] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:34:01.664788 15348 raft_consensus.cc:3037] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:34:01.668081 15348 raft_consensus.cc:504] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df32c007eff84cacb8eade269503bf10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40287 } } peers { permanent_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "089dde0b70c247128b9290f7b46e474e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38991 } }
I20260926 23:34:01.668942 15348 leader_election.cc:288] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10 [CANDIDATE]: Term 1 election: Requested vote from peers be4b5734244b4b418e21dc6e65bb9bbb (127.0.97.1:40569), 089dde0b70c247128b9290f7b46e474e (127.0.97.2:38991)
I20260926 23:34:01.669665 15148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b58489f5294e10b665e772b7b2dfd9" candidate_uuid: "df32c007eff84cacb8eade269503bf10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "089dde0b70c247128b9290f7b46e474e"
I20260926 23:34:01.669679 15012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b58489f5294e10b665e772b7b2dfd9" candidate_uuid: "df32c007eff84cacb8eade269503bf10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be4b5734244b4b418e21dc6e65bb9bbb"
I20260926 23:34:01.669925 15012 raft_consensus.cc:3037] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:34:01.669915 15148 raft_consensus.cc:3037] T 84b58489f5294e10b665e772b7b2dfd9 P 089dde0b70c247128b9290f7b46e474e [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:34:01.672680 15012 raft_consensus.cc:2454] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df32c007eff84cacb8eade269503bf10 in term 1.
I20260926 23:34:01.672681 15148 raft_consensus.cc:2454] T 84b58489f5294e10b665e772b7b2dfd9 P 089dde0b70c247128b9290f7b46e474e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df32c007eff84cacb8eade269503bf10 in term 1.
I20260926 23:34:01.673390 15214 leader_election.cc:302] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 089dde0b70c247128b9290f7b46e474e, be4b5734244b4b418e21dc6e65bb9bbb, df32c007eff84cacb8eade269503bf10; no voters:
I20260926 23:34:01.673877 15348 raft_consensus.cc:2781] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:34:01.674537 15348 raft_consensus.cc:686] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10 [term 1 LEADER]: Becoming Leader. State: Replica: df32c007eff84cacb8eade269503bf10, State: Running, Role: LEADER
I20260926 23:34:01.675200 15348 consensus_queue.cc:237] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df32c007eff84cacb8eade269503bf10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40287 } } peers { permanent_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "089dde0b70c247128b9290f7b46e474e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38991 } }
I20260926 23:34:01.681510 14869 catalog_manager.cc:5495] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10 reported cstate change: term changed from 0 to 1, leader changed from <none> to df32c007eff84cacb8eade269503bf10 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "df32c007eff84cacb8eade269503bf10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df32c007eff84cacb8eade269503bf10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "089dde0b70c247128b9290f7b46e474e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38991 } health_report { overall_health: UNKNOWN } } }
W20260926 23:34:01.689818 15058 tablet.cc:2369] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:34:01.712559 15337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:01.706232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41668 (local address 127.0.97.3:40287)
0926 23:34:01.706479 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:01.706486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:01.706511 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:34:01.706798 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:01.706803 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:01.706904 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:01.707155 (+ 251us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:34:01.707609 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:01.708403 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:01.709912 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:01.710410 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:01.710542 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:01.711207 (+ 665us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:34:01.711897 (+ 690us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:34:01.712059 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:01.712148 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 23:34:01.712255 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 23:34:01.723541 15012 raft_consensus.cc:1261] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [term 1 FOLLOWER]: Refusing update from remote peer df32c007eff84cacb8eade269503bf10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:34:01.723544 15148 raft_consensus.cc:1261] T 84b58489f5294e10b665e772b7b2dfd9 P 089dde0b70c247128b9290f7b46e474e [term 1 FOLLOWER]: Refusing update from remote peer df32c007eff84cacb8eade269503bf10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:34:01.724730 15351 consensus_queue.cc:1035] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:34:01.725198 15348 consensus_queue.cc:1035] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "089dde0b70c247128b9290f7b46e474e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:34:01.737597 15354 mvcc.cc:205] Tried to move back new op lower bound from 7333747268486356992 to 7333747268306456576. Current Snapshot: MvccSnapshot[applied={T|T < 7333747268486356992}]
I20260926 23:34:01.757269 14869 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:33298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:34:01.757695 14869 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:33298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:34:01.762660 14869 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5: Sending DeleteTablet for 3 replicas of tablet 84b58489f5294e10b665e772b7b2dfd9
I20260926 23:34:01.764179 14992 tablet_service.cc:1514] Processing DeleteTablet for tablet 84b58489f5294e10b665e772b7b2dfd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:34:01 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47920
I20260926 23:34:01.764312 15260 tablet_service.cc:1514] Processing DeleteTablet for tablet 84b58489f5294e10b665e772b7b2dfd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:34:01 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41664
I20260926 23:34:01.764312 15128 tablet_service.cc:1514] Processing DeleteTablet for tablet 84b58489f5294e10b665e772b7b2dfd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:34:01 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43842
I20260926 23:34:01.766058 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 14907
I20260926 23:34:01.780809 15365 tablet_replica.cc:330] T 84b58489f5294e10b665e772b7b2dfd9 P 089dde0b70c247128b9290f7b46e474e: stopping tablet replica
I20260926 23:34:01.790810 15349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:01.767979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47930 (local address 127.0.97.1:40569)
0926 23:34:01.768262 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:01.768269 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:01.768292 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:34:01.768374 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:01.768381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:01.768483 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:01.768618 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:34:01.770932 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:01.772306 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:01.774461 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:01.775386 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:01.775547 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:01.780702 (+ 5155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:34:01.780732 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:34:01.780758 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:34:01.780820 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:34:01.785618 (+ 4798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:01.786914 (+ 1296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:01.786926 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:01.786931 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:01.787160 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:01.787679 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:01.787686 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:01.787689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:01.788232 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:34:01.788577 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:01.790031 (+ 1454us) server_negotiation.cc:299] Negotiation successful
0926 23:34:01.790333 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 23:34:01.792303 15052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:34:01.792572 15365 raft_consensus.cc:2229] T 84b58489f5294e10b665e772b7b2dfd9 P 089dde0b70c247128b9290f7b46e474e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:34:01.792858 15366 tablet_replica.cc:330] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10: stopping tablet replica
I20260926 23:34:01.793601 15365 raft_consensus.cc:2258] T 84b58489f5294e10b665e772b7b2dfd9 P 089dde0b70c247128b9290f7b46e474e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:34:01.793608 15366 raft_consensus.cc:2229] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:34:01.794456 15366 raft_consensus.cc:2258] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:34:01.796872 15365 ts_tablet_manager.cc:1905] T 84b58489f5294e10b665e772b7b2dfd9 P 089dde0b70c247128b9290f7b46e474e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:34:01.800890 15366 ts_tablet_manager.cc:1905] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:34:01.807217 15365 ts_tablet_manager.cc:1918] T 84b58489f5294e10b665e772b7b2dfd9 P 089dde0b70c247128b9290f7b46e474e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:34:01.807468 15365 log.cc:1196] T 84b58489f5294e10b665e772b7b2dfd9 P 089dde0b70c247128b9290f7b46e474e: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/wal/wals/84b58489f5294e10b665e772b7b2dfd9
I20260926 23:34:01.808173 15365 ts_tablet_manager.cc:1939] T 84b58489f5294e10b665e772b7b2dfd9 P 089dde0b70c247128b9290f7b46e474e: Deleting consensus metadata
I20260926 23:34:01.810726 14856 catalog_manager.cc:4841] TS 089dde0b70c247128b9290f7b46e474e (127.0.97.2:38991): tablet 84b58489f5294e10b665e772b7b2dfd9 (table test-table [id=3c8cff34147c46ba9d896225929d14ab]) successfully deleted
I20260926 23:34:01.814970 15366 ts_tablet_manager.cc:1918] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:34:01.815217 15366 log.cc:1196] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/wal/wals/84b58489f5294e10b665e772b7b2dfd9
I20260926 23:34:01.815861 15366 ts_tablet_manager.cc:1939] T 84b58489f5294e10b665e772b7b2dfd9 P df32c007eff84cacb8eade269503bf10: Deleting consensus metadata
I20260926 23:34:01.818444 14855 catalog_manager.cc:4841] TS df32c007eff84cacb8eade269503bf10 (127.0.97.3:40287): tablet 84b58489f5294e10b665e772b7b2dfd9 (table test-table [id=3c8cff34147c46ba9d896225929d14ab]) successfully deleted
I20260926 23:34:01.907220 15364 tablet_replica.cc:330] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb: stopping tablet replica
I20260926 23:34:01.907533 15364 raft_consensus.cc:2229] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:34:01.907933 15364 raft_consensus.cc:2258] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:34:01.908043 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 14907
W20260926 23:34:01.927290 14855 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:40569 (error 108)
I20260926 23:34:01.927408 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 15062
I20260926 23:34:01.929935 15336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:01.929263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40569 (local address 127.0.0.1:47940)
0926 23:34:01.929554 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:01.929712 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
W20260926 23:34:01.930294 14855 catalog_manager.cc:4584] TS be4b5734244b4b418e21dc6e65bb9bbb (127.0.97.1:40569): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 84b58489f5294e10b665e772b7b2dfd9: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40569: connect: Connection refused (error 111)
I20260926 23:34:01.943248 15339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:01.928611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43848 (local address 127.0.97.2:38991)
0926 23:34:01.928848 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:01.928855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:01.928879 (+ 24us) server_negotiation.cc:407] Connection header received
0926 23:34:01.929069 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:01.929075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:01.929183 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:01.929283 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:34:01.930909 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:01.931731 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:01.933155 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:01.933588 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:01.933712 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:01.936740 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:34:01.936760 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:34:01.936782 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:34:01.936826 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:34:01.939979 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:01.940834 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:01.940843 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:01.940846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:01.940986 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:01.941395 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:01.941400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:01.941403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:01.941875 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:34:01.942200 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:01.942510 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 23:34:01.942899 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 23:34:01.944201 15188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:34:01.984329 15336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:01.983780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40569 (local address 127.0.0.1:47942)
0926 23:34:01.984013 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:01.984135 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 23:34:01.985510 15336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:01.985117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40569 (local address 127.0.0.1:47946)
0926 23:34:01.985276 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:01.985356 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 23:34:02.046372 15336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:02.045738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40569 (local address 127.0.0.1:47962)
0926 23:34:02.045995 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:02.046101 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 23:34:02.047736 15336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:02.047296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40569 (local address 127.0.0.1:47968)
0926 23:34:02.047474 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:02.047568 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 23:34:02.101613 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 15062
I20260926 23:34:02.128896 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 15197
I20260926 23:34:02.129333 15336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:02.128648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40569 (local address 127.0.0.1:47978)
0926 23:34:02.128914 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:02.129042 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 23:34:02.131458 15336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:02.130671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40569 (local address 127.0.0.1:47984)
0926 23:34:02.131132 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:02.131256 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341}
I20260926 23:34:02.146962 15337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:02.130154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41676 (local address 127.0.97.3:40287)
0926 23:34:02.130374 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:02.130380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:02.130406 (+ 26us) server_negotiation.cc:407] Connection header received
0926 23:34:02.130555 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:02.130561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:02.130663 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:02.130808 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:34:02.132713 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:02.133645 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:02.135219 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:02.135739 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:02.135890 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:02.139132 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:34:02.139156 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:34:02.139178 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:34:02.139242 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:34:02.143288 (+ 4046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:02.144319 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:02.144329 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:02.144332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:02.144561 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:02.145004 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:02.145009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:02.145011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:02.145485 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:34:02.145870 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:02.146186 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 23:34:02.146473 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 23:34:02.147806 15320 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:34:02.293531 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 15197
I20260926 23:34:02.304993 15336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:02.304342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40569 (local address 127.0.0.1:47994)
0926 23:34:02.304612 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:02.304748 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260926 23:34:02.306226 15336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:02.305806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40569 (local address 127.0.0.1:48006)
0926 23:34:02.305969 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:02.306058 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 23:34:02.313748 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 14837
I20260926 23:34:02.315205 14900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:34:02.446980 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 14837
I20260926 23:34:02.470997 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37995
--webserver_interface=127.0.97.62
--webserver_port=39459
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:34:02.883670 15372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:34:02.884117 15372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:34:02.884265 15372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:34:02.892668 15372 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:34:02.892849 15372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:34:02.892936 15372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:34:02.893018 15372 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:34:02.893082 15372 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:34:02.904774 15372 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=39459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:34:02.906572 15372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:34:02.908272 15372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:34:02.922541 15377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:34:02.922504 15380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:34:02.922477 15378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:34:02.923447 15372 server_base.cc:1034] running on GCE node
I20260926 23:34:02.924489 15372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:34:02.926445 15372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:34:02.927719 15372 hybrid_clock.cc:648] HybridClock initialized: now 1790465642927640 us; error 75 us; skew 500 ppm
I20260926 23:34:02.928139 15372 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:34:02 dist-test-slave-gwfx krb5kdc[14283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:34:02.932303 15372 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:34:02.934320 15372 webserver.cc:460] Webserver started at http://127.0.97.62:39459/ using document root <none> and password file <none>
I20260926 23:34:02.935288 15372 fs_manager.cc:359] Metadata directory not provided
I20260926 23:34:02.935425 15372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:34:02.943073 15372 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 23:34:02.949306 15387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:02.951419 15372 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 23:34:02.951645 15372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "8255fe64fe7f4c0ca60dc6aa921e1bc5"
format_stamp: "Formatted at 2026-09-26 23:33:55 on dist-test-slave-gwfx"
server_key: "e866be79e2b94da75fd8e01ad40010f1"
server_key_iv: "8f125005c3e916e784eb75b31cadb717"
server_key_version: "encryptionkey@0"
I20260926 23:34:02.952907 15372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:34:02.976823 15372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:34:02.983021 15372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:34:02.983366 15372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:34:03.005065 15372 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37995
I20260926 23:34:03.005123 15438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37995 every 8 connection(s)
I20260926 23:34:03.007256 15372 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:34:03.012693 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 15372
I20260926 23:34:03.013797 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:40569
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37995
--webserver_port=38349
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:34:03.018576 15439 sys_catalog.cc:263] Verifying existing consensus state
I20260926 23:34:03.024746 15439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5: Bootstrap starting.
I20260926 23:34:03.050503 15439 log.cc:824] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5: Log is configured to *not* fsync() on all Append() calls
I20260926 23:34:03.070102 15439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 23:34:03.070837 15439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5: Bootstrap complete.
I20260926 23:34:03.081766 15439 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8255fe64fe7f4c0ca60dc6aa921e1bc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37995 } }
I20260926 23:34:03.083039 15439 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8255fe64fe7f4c0ca60dc6aa921e1bc5, State: Initialized, Role: FOLLOWER
I20260926 23:34:03.083937 15439 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8255fe64fe7f4c0ca60dc6aa921e1bc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37995 } }
I20260926 23:34:03.084230 15439 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:34:03.084364 15439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:34:03.084549 15439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [term 2 FOLLOWER]: Advancing to term 3
I20260926 23:34:03.089725 15439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8255fe64fe7f4c0ca60dc6aa921e1bc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37995 } }
I20260926 23:34:03.090793 15439 leader_election.cc:302] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8255fe64fe7f4c0ca60dc6aa921e1bc5; no voters:
I20260926 23:34:03.091734 15439 leader_election.cc:288] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 23:34:03.091922 15443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 23:34:03.093350 15443 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [term 3 LEADER]: Becoming Leader. State: Replica: 8255fe64fe7f4c0ca60dc6aa921e1bc5, State: Running, Role: LEADER
I20260926 23:34:03.094250 15443 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8255fe64fe7f4c0ca60dc6aa921e1bc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37995 } }
I20260926 23:34:03.094871 15439 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:34:03.097699 15445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8255fe64fe7f4c0ca60dc6aa921e1bc5. Latest consensus state: current_term: 3 leader_uuid: "8255fe64fe7f4c0ca60dc6aa921e1bc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8255fe64fe7f4c0ca60dc6aa921e1bc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37995 } } }
I20260926 23:34:03.097735 15444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8255fe64fe7f4c0ca60dc6aa921e1bc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8255fe64fe7f4c0ca60dc6aa921e1bc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37995 } } }
I20260926 23:34:03.098300 15445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [sys.catalog]: This master's current role is: LEADER
I20260926 23:34:03.098306 15444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8255fe64fe7f4c0ca60dc6aa921e1bc5 [sys.catalog]: This master's current role is: LEADER
I20260926 23:34:03.108249 15450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:34:03.128079 15450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:34:03.129941 15450 catalog_manager.cc:1259] Loaded cluster ID: 74c8e7b904974b18a48bd8f4729db3bb
I20260926 23:34:03.130136 15450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 23:34:03.132622 15450 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fbf09f509ad google::LogMessage::Fail() at ??:0
@ 0x7fbf09f5388a google::LogMessage::SendToLog() at ??:0
@ 0x7fbf09f504de google::LogMessage::Flush() at ??:0
@ 0x7fbf09f51a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbf1f983512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fbf1f980976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fbf1fa08610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fbf1fa08439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fbf0cb9c345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbf0cb9f91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fbf0cb9f799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fbf0cb7aeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbf0fe546db start_thread at ??:0
@ 0x7fbf08dea61f clone at ??:0
W20260926 23:34:03.463064 15441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:34:03.463485 15441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:34:03.463641 15441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:34:03.472126 15441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:34:03.472306 15441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:34:03.472476 15441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:34:03.484196 15441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:40569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38349
--tserver_master_addrs=127.0.97.62:37995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:34:03.485997 15441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:34:03.487804 15441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:34:03.505173 15483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:34:03.505120 15480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:34:03.506261 15481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:34:03.507150 15441 server_base.cc:1034] running on GCE node
I20260926 23:34:03.508169 15441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:34:03.509662 15441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:34:03.511004 15441 hybrid_clock.cc:648] HybridClock initialized: now 1790465643510918 us; error 68 us; skew 500 ppm
I20260926 23:34:03.511508 15441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:34:03 dist-test-slave-gwfx krb5kdc[14283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:34:03.517109 15441 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:34:03.519774 15441 webserver.cc:460] Webserver started at http://127.0.97.1:38349/ using document root <none> and password file <none>
I20260926 23:34:03.521068 15441 fs_manager.cc:359] Metadata directory not provided
I20260926 23:34:03.521262 15441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:34:03.530530 15441 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.002s sys 0.004s
I20260926 23:34:03.535368 15490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:03.537575 15441 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s
I20260926 23:34:03.537837 15441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "be4b5734244b4b418e21dc6e65bb9bbb"
format_stamp: "Formatted at 2026-09-26 23:33:56 on dist-test-slave-gwfx"
server_key: "a12e27122c093abc0b0e4cdbfb3ae8f8"
server_key_iv: "edb1145c0d529d437085a9da2d2e7d7f"
server_key_version: "encryptionkey@0"
I20260926 23:34:03.539534 15441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:34:03.565881 15441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:34:03.571811 15441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:34:03.572117 15441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:34:03.573484 15441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:34:03.576383 15497 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260926 23:34:03.583774 15441 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 23:34:03.583943 15441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.001s sys 0.000s
I20260926 23:34:03.584087 15441 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 23:34:03.587932 15441 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 23:34:03.588037 15441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.003s sys 0.000s
I20260926 23:34:03.588503 15497 tablet_bootstrap.cc:492] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb: Bootstrap starting.
I20260926 23:34:03.619738 15497 log.cc:824] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb: Log is configured to *not* fsync() on all Append() calls
I20260926 23:34:03.631508 15441 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40569
I20260926 23:34:03.631597 15604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40569 every 8 connection(s)
I20260926 23:34:03.633029 15441 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:34:03.635676 15497 tablet_bootstrap.cc:492] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 23:34:03.636273 15497 tablet_bootstrap.cc:492] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb: Bootstrap complete.
I20260926 23:34:03.637781 15497 ts_tablet_manager.cc:1397] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb: Time spent bootstrapping tablet: real 0.050s user 0.026s sys 0.019s
I20260926 23:34:03.639536 15607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:03.636759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.1:53247)
0926 23:34:03.637747 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:03.638039 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":398,"threads_started":1}
I20260926 23:34:03.639839 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 15441
I20260926 23:34:03.640487 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:38991
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37995
--webserver_port=37053
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:34:03.643702 15605 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37995 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37995: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
I20260926 23:34:03.645176 15607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:03.644601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.1:43985)
0926 23:34:03.644839 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:03.644968 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 23:34:03.647101 15607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:03.645894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.1:59621)
0926 23:34:03.646662 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:03.646805 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":436,"threads_started":1}
W20260926 23:34:03.647332 15605 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 23:34:03.647034 15497 raft_consensus.cc:348] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df32c007eff84cacb8eade269503bf10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40287 } } peers { permanent_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "089dde0b70c247128b9290f7b46e474e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38991 } }
I20260926 23:34:03.648296 15497 raft_consensus.cc:729] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: be4b5734244b4b418e21dc6e65bb9bbb, State: Initialized, Role: FOLLOWER
I20260926 23:34:03.649115 15497 consensus_queue.cc:260] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df32c007eff84cacb8eade269503bf10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40287 } } peers { permanent_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "089dde0b70c247128b9290f7b46e474e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38991 } }
I20260926 23:34:03.651216 15607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:03.650734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.1:34421)
0926 23:34:03.650918 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:03.651018 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 23:34:03.651211 15497 ts_tablet_manager.cc:1428] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb: Time spent starting tablet: real 0.013s user 0.015s sys 0.001s
I20260926 23:34:03.652365 15607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:03.651961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.1:41743)
0926 23:34:03.652122 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:03.652211 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 23:34:04.043303 15609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:34:04.043702 15609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:34:04.043813 15609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:34:04.052217 15609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:34:04.052374 15609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:34:04.052506 15609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:34:04.063966 15609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:38991
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=37053
--tserver_master_addrs=127.0.97.62:37995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:34:04.065681 15609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:34:04.067308 15609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:34:04.083949 15615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:34:04.083949 15618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:34:04.083990 15616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:34:04.084992 15609 server_base.cc:1034] running on GCE node
I20260926 23:34:04.085769 15609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:34:04.087049 15609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:34:04.088305 15609 hybrid_clock.cc:648] HybridClock initialized: now 1790465644088230 us; error 64 us; skew 500 ppm
I20260926 23:34:04.088732 15609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:34:04 dist-test-slave-gwfx krb5kdc[14283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:34:04.092813 15609 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:34:04.094920 15609 webserver.cc:460] Webserver started at http://127.0.97.2:37053/ using document root <none> and password file <none>
I20260926 23:34:04.095877 15609 fs_manager.cc:359] Metadata directory not provided
I20260926 23:34:04.096037 15609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:34:04.103752 15609 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 23:34:04.107779 15625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:04.109875 15609 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 23:34:04.110124 15609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "089dde0b70c247128b9290f7b46e474e"
format_stamp: "Formatted at 2026-09-26 23:33:57 on dist-test-slave-gwfx"
server_key: "56ba41fb43d5d957ff4794d82d9b133d"
server_key_iv: "d312e17fb53e303e073b1923be780858"
server_key_version: "encryptionkey@0"
I20260926 23:34:04.111582 15609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:34:04.127938 15609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:34:04.133540 15609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:34:04.133860 15609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:34:04.135064 15609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:34:04.136953 15609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:34:04.137071 15609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:04.137189 15609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:34:04.137260 15609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:04.174036 15609 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38991
I20260926 23:34:04.174099 15737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38991 every 8 connection(s)
I20260926 23:34:04.175848 15609 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:34:04.182338 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:04.179749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.2:55191)
0926 23:34:04.180697 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:04.180951 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":384,"threads_started":1}
I20260926 23:34:04.182623 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 15609
W20260926 23:34:04.183341 15738 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37995 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37995: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
I20260926 23:34:04.183499 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:40287
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37995
--webserver_port=44293
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:34:04.184553 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:04.184063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.2:55987)
0926 23:34:04.184265 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:04.184372 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 23:34:04.185554 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:04.185165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.2:35037)
0926 23:34:04.185316 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:04.185410 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 23:34:04.185835 15738 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 23:34:04.579433 15741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:34:04.579835 15741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:34:04.579990 15741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:34:04.588124 15741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:34:04.588271 15741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:34:04.588435 15741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:34:04.599763 15741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:40287
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=44293
--tserver_master_addrs=127.0.97.62:37995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:34:04.601523 15741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:34:04.603413 15741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:34:04.620313 15747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:34:04.620344 15746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:34:04.620395 15749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:34:04.622228 15741 server_base.cc:1034] running on GCE node
I20260926 23:34:04.623204 15741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:34:04.624548 15741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:34:04.625790 15741 hybrid_clock.cc:648] HybridClock initialized: now 1790465644625696 us; error 102 us; skew 500 ppm
I20260926 23:34:04.626175 15741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:34:04 dist-test-slave-gwfx krb5kdc[14283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:34:04.634809 15741 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:34:04.637315 15741 webserver.cc:460] Webserver started at http://127.0.97.3:44293/ using document root <none> and password file <none>
I20260926 23:34:04.638557 15741 fs_manager.cc:359] Metadata directory not provided
I20260926 23:34:04.638808 15741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:34:04.647403 15741 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 23:34:04.651376 15756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:04.653575 15741 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 23:34:04.654729 15757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:04.653600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.1:34543)
0926 23:34:04.654210 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:04.654332 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":316,"threads_started":1}
I20260926 23:34:04.653867 15741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "df32c007eff84cacb8eade269503bf10"
format_stamp: "Formatted at 2026-09-26 23:33:57 on dist-test-slave-gwfx"
server_key: "777ce6dacc53d96475b6e28ff1d9a4c9"
server_key_iv: "89849c912d646464ac21536d4add928e"
server_key_version: "encryptionkey@0"
I20260926 23:34:04.655458 15741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:34:04.690339 15741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:34:04.695777 15741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:34:04.696086 15741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:34:04.697345 15741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:34:04.699084 15741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:34:04.699193 15741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:04.699311 15741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:34:04.699391 15741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:04.735056 15741 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40287
I20260926 23:34:04.735132 15869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40287 every 8 connection(s)
I20260926 23:34:04.736792 15741 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:34:04.743276 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:04.740761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.3:60975)
0926 23:34:04.741741 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:04.742005 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":363,"threads_started":1}
W20260926 23:34:04.744177 15870 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37995 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37995: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
I20260926 23:34:04.745255 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:04.744838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.3:40685)
0926 23:34:04.745006 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:04.745103 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 23:34:04.745591 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 15741
I20260926 23:34:04.746256 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:04.745878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.3:58823)
0926 23:34:04.746025 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:04.746113 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 23:34:04.746541 15870 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 23:34:05.188402 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:05.186985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.2:57023)
0926 23:34:05.187746 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:05.188078 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":355,"threads_started":1}
I20260926 23:34:05.208472 15875 raft_consensus.cc:482] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:34:05.208832 15875 raft_consensus.cc:504] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df32c007eff84cacb8eade269503bf10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40287 } } peers { permanent_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "089dde0b70c247128b9290f7b46e474e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38991 } }
I20260926 23:34:05.210749 15875 leader_election.cc:288] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df32c007eff84cacb8eade269503bf10 (127.0.97.3:40287), 089dde0b70c247128b9290f7b46e474e (127.0.97.2:38991)
Sep 26 23:34:05 dist-test-slave-gwfx krb5kdc[14283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
Sep 26 23:34:05 dist-test-slave-gwfx krb5kdc[14283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:34:05.250439 15878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:05.211584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59167 (local address 127.0.97.2:38991)
0926 23:34:05.223548 (+ 11964us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:05.223575 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:05.224369 (+ 794us) server_negotiation.cc:407] Connection header received
0926 23:34:05.227039 (+ 2670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:05.227050 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:05.227536 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:05.227901 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:34:05.230616 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:05.231984 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:05.234171 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:05.234930 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:05.235120 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:05.240554 (+ 5434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:34:05.240566 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:34:05.240583 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:34:05.240638 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:34:05.245252 (+ 4614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:05.246816 (+ 1564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:05.246829 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:05.246836 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:05.247092 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:05.247586 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:05.247591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:05.247593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:05.248271 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:34:05.248489 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:05.249483 (+ 994us) server_negotiation.cc:299] Negotiation successful
0926 23:34:05.249933 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11599,"thread_start_us":11364,"threads_started":1}
I20260926 23:34:05.250455 15879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:05.211834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46873 (local address 127.0.97.3:40287)
0926 23:34:05.224588 (+ 12754us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:05.224603 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:05.224658 (+ 55us) server_negotiation.cc:407] Connection header received
0926 23:34:05.227099 (+ 2441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:05.227112 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:05.227617 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:05.228011 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:34:05.230608 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:05.231997 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:05.234182 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:05.235016 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:05.235243 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:05.240146 (+ 4903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:34:05.240157 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:34:05.240177 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:34:05.240248 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:34:05.245269 (+ 5021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:05.246767 (+ 1498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:05.246779 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:05.246791 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:05.247109 (+ 318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:05.247637 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:05.247641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:05.247643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:05.248305 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:34:05.248505 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:05.249480 (+ 975us) server_negotiation.cc:299] Negotiation successful
0926 23:34:05.249951 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12540,"thread_start_us":12384,"threads_started":1}
I20260926 23:34:05.250522 15876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:05.211149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40287 (local address 127.0.97.1:46873)
0926 23:34:05.224184 (+ 13035us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:05.224212 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 23:34:05.225024 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:34:05.228140 (+ 3116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:34:05.228154 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:34:05.229310 (+ 1156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:34:05.230360 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:05.230388 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:05.232213 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:05.232219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:05.233986 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:05.234010 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:05.235270 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:05.235277 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:05.235518 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:05.236382 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:34:05.236402 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:34:05.239676 (+ 3274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:34:05.245544 (+ 5868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:34:05.245558 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:34:05.245581 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:34:05.246523 (+ 942us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:34:05.247280 (+ 757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:34:05.247286 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:34:05.247293 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:34:05.247499 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:34:05.248459 (+ 960us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:34:05.248472 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:34:05.248867 (+ 395us) client_negotiation.cc:769] Sending connection context
0926 23:34:05.249191 (+ 324us) client_negotiation.cc:240] Negotiation successful
0926 23:34:05.249979 (+ 788us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12865,"mutex_wait_us":510,"spinlock_wait_cycles":29312,"thread_start_us":12053,"threads_started":1}
I20260926 23:34:05.251307 15877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:05.211149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38991 (local address 127.0.97.1:59167)
0926 23:34:05.224049 (+ 12900us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:05.224134 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0926 23:34:05.224980 (+ 846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:34:05.228140 (+ 3160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:34:05.228155 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:34:05.229449 (+ 1294us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:34:05.230364 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:05.230392 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:05.232217 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:05.232222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:05.233987 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:05.234012 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:05.235138 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:05.235145 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:05.235450 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:05.236288 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:34:05.236322 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:34:05.240097 (+ 3775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:34:05.245544 (+ 5447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:34:05.245558 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:34:05.245581 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:34:05.246523 (+ 942us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:34:05.247238 (+ 715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:34:05.247242 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:34:05.247244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:34:05.247459 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:34:05.248453 (+ 994us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:34:05.248462 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:34:05.248868 (+ 406us) client_negotiation.cc:769] Sending connection context
0926 23:34:05.249191 (+ 323us) client_negotiation.cc:240] Negotiation successful
0926 23:34:05.250193 (+ 1002us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12626,"mutex_wait_us":500,"spinlock_wait_cycles":57088,"thread_start_us":12672,"threads_started":1}
W20260926 23:34:05.254940 15492 leader_election.cc:341] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df32c007eff84cacb8eade269503bf10 (127.0.97.3:40287): Not found: Tablet not found: 84b58489f5294e10b665e772b7b2dfd9
W20260926 23:34:05.255365 15493 leader_election.cc:341] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 089dde0b70c247128b9290f7b46e474e (127.0.97.2:38991): Not found: Tablet not found: 84b58489f5294e10b665e772b7b2dfd9
I20260926 23:34:05.255491 15493 leader_election.cc:302] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be4b5734244b4b418e21dc6e65bb9bbb; no voters: 089dde0b70c247128b9290f7b46e474e, df32c007eff84cacb8eade269503bf10
I20260926 23:34:05.255911 15875 raft_consensus.cc:2726] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 23:34:05.656723 15877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:05.656052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.1:59555)
0926 23:34:05.656386 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:05.656549 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260926 23:34:05.748320 15880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:05.747297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.3:53003)
0926 23:34:05.747902 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:05.748054 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":268,"threads_started":1}
I20260926 23:34:06.190894 15881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:06.189858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.2:55375)
0926 23:34:06.190493 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:06.190614 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":323,"threads_started":1}
I20260926 23:34:06.659368 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:06.658049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.1:38439)
0926 23:34:06.659022 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:06.659148 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":804,"thread_start_us":360,"threads_started":1}
I20260926 23:34:06.750445 15883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:06.749435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.3:45811)
0926 23:34:06.750089 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:06.750217 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":293,"threads_started":1}
I20260926 23:34:07.193264 15884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:07.192205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.2:50445)
0926 23:34:07.192897 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:07.193025 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":346,"threads_started":1}
I20260926 23:34:07.453504 15885 raft_consensus.cc:482] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:34:07.453771 15885 raft_consensus.cc:504] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df32c007eff84cacb8eade269503bf10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40287 } } peers { permanent_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "089dde0b70c247128b9290f7b46e474e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38991 } }
I20260926 23:34:07.454902 15885 leader_election.cc:288] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df32c007eff84cacb8eade269503bf10 (127.0.97.3:40287), 089dde0b70c247128b9290f7b46e474e (127.0.97.2:38991)
W20260926 23:34:07.456262 15492 leader_election.cc:341] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df32c007eff84cacb8eade269503bf10 (127.0.97.3:40287): Not found: Tablet not found: 84b58489f5294e10b665e772b7b2dfd9
W20260926 23:34:07.456593 15493 leader_election.cc:341] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 089dde0b70c247128b9290f7b46e474e (127.0.97.2:38991): Not found: Tablet not found: 84b58489f5294e10b665e772b7b2dfd9
I20260926 23:34:07.456758 15493 leader_election.cc:302] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be4b5734244b4b418e21dc6e65bb9bbb; no voters: 089dde0b70c247128b9290f7b46e474e, df32c007eff84cacb8eade269503bf10
I20260926 23:34:07.457099 15885 raft_consensus.cc:2726] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 23:34:07.662240 15886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:07.660823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.1:46625)
0926 23:34:07.661486 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:07.661936 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":317,"threads_started":1}
I20260926 23:34:07.752957 15887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:07.751783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.3:40221)
0926 23:34:07.752566 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:07.752734 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":324,"threads_started":1}
I20260926 23:34:08.195458 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:08.194464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.2:47605)
0926 23:34:08.195141 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:08.195239 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":398,"threads_started":1}
I20260926 23:34:08.664870 15889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:08.663893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.1:46295)
0926 23:34:08.664484 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:08.664604 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":283,"threads_started":1}
I20260926 23:34:08.755550 15890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:08.754404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.3:42539)
0926 23:34:08.755201 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:08.755319 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":433,"threads_started":1}
I20260926 23:34:09.197655 15891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:09.196746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.2:41445)
0926 23:34:09.197353 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:09.197467 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":292,"threads_started":1}
I20260926 23:34:09.667156 15892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:09.666171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.1:51497)
0926 23:34:09.666810 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:09.666934 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":314,"threads_started":1}
I20260926 23:34:09.757747 15893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:09.756745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.3:57647)
0926 23:34:09.757394 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:09.757529 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":351,"threads_started":1}
I20260926 23:34:10.200119 15894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:10.199073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.2:38629)
0926 23:34:10.199692 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:10.199815 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":300,"threads_started":1}
I20260926 23:34:10.669359 15895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:10.668373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.1:38461)
0926 23:34:10.668984 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:10.669112 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":358,"threads_started":1}
I20260926 23:34:10.680789 15896 raft_consensus.cc:482] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:34:10.681025 15896 raft_consensus.cc:504] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df32c007eff84cacb8eade269503bf10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40287 } } peers { permanent_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "089dde0b70c247128b9290f7b46e474e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38991 } }
I20260926 23:34:10.681820 15896 leader_election.cc:288] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df32c007eff84cacb8eade269503bf10 (127.0.97.3:40287), 089dde0b70c247128b9290f7b46e474e (127.0.97.2:38991)
W20260926 23:34:10.683112 15493 leader_election.cc:341] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 089dde0b70c247128b9290f7b46e474e (127.0.97.2:38991): Not found: Tablet not found: 84b58489f5294e10b665e772b7b2dfd9
W20260926 23:34:10.683432 15492 leader_election.cc:341] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df32c007eff84cacb8eade269503bf10 (127.0.97.3:40287): Not found: Tablet not found: 84b58489f5294e10b665e772b7b2dfd9
I20260926 23:34:10.683581 15492 leader_election.cc:302] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be4b5734244b4b418e21dc6e65bb9bbb; no voters: 089dde0b70c247128b9290f7b46e474e, df32c007eff84cacb8eade269503bf10
I20260926 23:34:10.683887 15896 raft_consensus.cc:2726] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 23:34:10.759984 15897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:10.758931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.3:41067)
0926 23:34:10.759623 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:10.759753 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":333,"threads_started":1}
I20260926 23:34:11.202653 15898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:11.201526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.2:49769)
0926 23:34:11.202264 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:11.202412 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":331,"threads_started":1}
I20260926 23:34:11.671774 15899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:11.670755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.1:56361)
0926 23:34:11.671418 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:11.671532 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":372,"threads_started":1}
I20260926 23:34:11.762560 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:11.761331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.3:40263)
0926 23:34:11.762139 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:11.762297 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":339,"threads_started":1}
I20260926 23:34:12.204917 15901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:12.203998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.2:39503)
0926 23:34:12.204571 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:12.204682 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":276,"threads_started":1}
I20260926 23:34:12.674055 15902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:12.673042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.1:36325)
0926 23:34:12.673694 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:12.673810 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":331,"threads_started":1}
I20260926 23:34:12.764753 15903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:12.763827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.3:52857)
0926 23:34:12.764404 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:12.764523 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":298,"threads_started":1}
I20260926 23:34:13.207253 15904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:13.206220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.2:35345)
0926 23:34:13.206887 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:13.207014 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":333,"threads_started":1}
I20260926 23:34:13.676611 15905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:13.675508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.1:36567)
0926 23:34:13.676241 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:13.676375 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":435,"threads_started":1}
I20260926 23:34:13.766983 15906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:13.765896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.3:48475)
0926 23:34:13.766544 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:13.766662 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":299,"threads_started":1}
I20260926 23:34:14.209671 15907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:14.208549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.2:56651)
0926 23:34:14.209262 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:14.209437 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":357,"threads_started":1}
I20260926 23:34:14.473557 15908 raft_consensus.cc:482] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:34:14.473829 15908 raft_consensus.cc:504] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df32c007eff84cacb8eade269503bf10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40287 } } peers { permanent_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "089dde0b70c247128b9290f7b46e474e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38991 } }
I20260926 23:34:14.474534 15908 leader_election.cc:288] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df32c007eff84cacb8eade269503bf10 (127.0.97.3:40287), 089dde0b70c247128b9290f7b46e474e (127.0.97.2:38991)
W20260926 23:34:14.475814 15492 leader_election.cc:341] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df32c007eff84cacb8eade269503bf10 (127.0.97.3:40287): Not found: Tablet not found: 84b58489f5294e10b665e772b7b2dfd9
W20260926 23:34:14.476058 15493 leader_election.cc:341] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 089dde0b70c247128b9290f7b46e474e (127.0.97.2:38991): Not found: Tablet not found: 84b58489f5294e10b665e772b7b2dfd9
I20260926 23:34:14.476179 15493 leader_election.cc:302] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be4b5734244b4b418e21dc6e65bb9bbb; no voters: 089dde0b70c247128b9290f7b46e474e, df32c007eff84cacb8eade269503bf10
I20260926 23:34:14.476487 15908 raft_consensus.cc:2726] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 23:34:14.679041 15909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:14.677910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.1:47625)
0926 23:34:14.678569 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:14.678744 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":303,"threads_started":1}
I20260926 23:34:14.769194 15910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:14.768277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.3:38659)
0926 23:34:14.768868 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:14.768966 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":305,"threads_started":1}
I20260926 23:34:15.212080 15911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:15.211109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.2:56905)
0926 23:34:15.211717 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:15.211861 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":306,"threads_started":1}
I20260926 23:34:15.681456 15912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:15.680317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.1:57701)
0926 23:34:15.681041 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:15.681164 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":359,"threads_started":1}
I20260926 23:34:15.771593 15913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:15.770536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.3:59157)
0926 23:34:15.771218 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:15.771343 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":310,"threads_started":1}
I20260926 23:34:16.214300 15914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:16.213272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.2:45509)
0926 23:34:16.213979 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:16.214098 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":400,"threads_started":1}
I20260926 23:34:16.683611 15915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:16.682724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.1:35259)
0926 23:34:16.683283 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:16.683394 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":281,"threads_started":1}
I20260926 23:34:16.774081 15916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:16.772999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.3:41685)
0926 23:34:16.773680 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:16.773815 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":315,"threads_started":1}
I20260926 23:34:17.216835 15917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:17.215857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.2:46323)
0926 23:34:17.216519 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:17.216614 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":380,"threads_started":1}
I20260926 23:34:17.685720 15918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:17.684744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.1:52117)
0926 23:34:17.685381 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:17.685503 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":347,"threads_started":1}
I20260926 23:34:17.776185 15919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:17.775243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.3:38955)
0926 23:34:17.775884 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:17.775985 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":361,"threads_started":1}
I20260926 23:34:18.219498 15920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:18.218366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.2:39261)
0926 23:34:18.219076 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:18.219187 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":429,"threads_started":1}
I20260926 23:34:18.688337 15921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:18.687405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.1:35029)
0926 23:34:18.687988 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:18.688107 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":268,"threads_started":1}
I20260926 23:34:18.735443 15922 raft_consensus.cc:482] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:34:18.735728 15922 raft_consensus.cc:504] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df32c007eff84cacb8eade269503bf10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40287 } } peers { permanent_uuid: "be4b5734244b4b418e21dc6e65bb9bbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "089dde0b70c247128b9290f7b46e474e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38991 } }
I20260926 23:34:18.736763 15922 leader_election.cc:288] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df32c007eff84cacb8eade269503bf10 (127.0.97.3:40287), 089dde0b70c247128b9290f7b46e474e (127.0.97.2:38991)
W20260926 23:34:18.737754 15493 leader_election.cc:341] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 089dde0b70c247128b9290f7b46e474e (127.0.97.2:38991): Not found: Tablet not found: 84b58489f5294e10b665e772b7b2dfd9
W20260926 23:34:18.737978 15492 leader_election.cc:341] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df32c007eff84cacb8eade269503bf10 (127.0.97.3:40287): Not found: Tablet not found: 84b58489f5294e10b665e772b7b2dfd9
I20260926 23:34:18.738180 15492 leader_election.cc:302] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be4b5734244b4b418e21dc6e65bb9bbb; no voters: 089dde0b70c247128b9290f7b46e474e, df32c007eff84cacb8eade269503bf10
I20260926 23:34:18.738464 15922 raft_consensus.cc:2726] T 84b58489f5294e10b665e772b7b2dfd9 P be4b5734244b4b418e21dc6e65bb9bbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 23:34:18.778359 15923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:18.777359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.3:55707)
0926 23:34:18.778012 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:18.778121 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":332,"threads_started":1}
I20260926 23:34:19.221786 15924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:19.220832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.2:37993)
0926 23:34:19.221431 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:19.221568 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":294,"threads_started":1}
I20260926 23:34:19.690739 15925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:19.689651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.1:46615)
0926 23:34:19.690314 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:19.690433 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":318,"threads_started":1}
I20260926 23:34:19.757191 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 15441
I20260926 23:34:19.781034 15927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:19.779896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37995 (local address 127.0.97.3:41769)
0926 23:34:19.780631 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:19.780778 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":338,"threads_started":1}
I20260926 23:34:19.781390 15926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:19.758817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56356 (local address 127.0.97.1:40569)
0926 23:34:19.759538 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:19.759549 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:19.759582 (+ 33us) server_negotiation.cc:407] Connection header received
0926 23:34:19.759673 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:19.759683 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:19.760028 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:19.760222 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:34:19.762237 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:19.763600 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:19.765041 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:19.765747 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:19.765894 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:19.773735 (+ 7841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:34:19.773758 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:34:19.773780 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:34:19.773897 (+ 117us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:34:19.777346 (+ 3449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:19.778404 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:19.778415 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:19.778418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:19.778602 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:19.779207 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:19.779212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:19.779214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:19.779787 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:34:19.780276 (+ 489us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:19.780693 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 23:34:19.780993 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":302,"threads_started":1}
I20260926 23:34:19.782994 15600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:34:19.891700 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 15441
I20260926 23:34:19.913270 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 15609
I20260926 23:34:19.931730 15929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:19.914408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45020 (local address 127.0.97.2:38991)
0926 23:34:19.915009 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:19.915013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:19.915034 (+ 21us) server_negotiation.cc:407] Connection header received
0926 23:34:19.915105 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:19.915109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:19.915184 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:19.915312 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:34:19.917202 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:19.918335 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:19.920395 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:19.920915 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:19.921074 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:19.924422 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:34:19.924429 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:34:19.924433 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:34:19.924485 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:34:19.927832 (+ 3347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:19.929009 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:19.929015 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:19.929018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:19.929228 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:19.929664 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:19.929668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:19.929670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:19.929994 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:34:19.930248 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:19.931077 (+ 829us) server_negotiation.cc:299] Negotiation successful
0926 23:34:19.931327 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":358,"threads_started":1}
I20260926 23:34:19.932663 15733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:34:20.027444 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 15609
I20260926 23:34:20.045166 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 15741
I20260926 23:34:20.062479 15931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:20.046164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34254 (local address 127.0.97.3:40287)
0926 23:34:20.046819 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:20.046824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:20.046845 (+ 21us) server_negotiation.cc:407] Connection header received
0926 23:34:20.046914 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:20.046918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:20.046986 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:20.047112 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:34:20.048731 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:20.049645 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:20.051315 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:20.052008 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:20.052172 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:20.055497 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:34:20.055504 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:34:20.055508 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:34:20.055557 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:34:20.058884 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:20.060005 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:20.060010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:20.060020 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:20.060245 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:20.060835 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:20.060841 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:20.060844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:20.061221 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:34:20.061579 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:20.061897 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 23:34:20.062155 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":285,"threads_started":1}
I20260926 23:34:20.063472 15865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:34:20.158780 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 15741
2026-09-26T23:34:20Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25990 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:34:20 dist-test-slave-gwfx krb5kdc[15935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:34:20 dist-test-slave-gwfx krb5kdc[15935](info): set up 2 sockets
Sep 26 23:34:20 dist-test-slave-gwfx krb5kdc[15935](info): commencing operation
krb5kdc: starting...
W20260926 23:34:21.227725 388 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.002s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:34:21 dist-test-slave-gwfx krb5kdc[15935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:34:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:34:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:34:21.507481 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46435
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:34:21.900080 15948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:34:21.900439 15948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:34:21.900573 15948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:34:21.909337 15948 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 23:34:21.909631 15948 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 23:34:21.909924 15948 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 23:34:21.910027 15948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:34:21.910110 15948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:34:21.910210 15948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:34:21.910277 15948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:34:21.910351 15948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:34:21.921258 15948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:34:21.922833 15948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:34:21.924438 15948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:34:21.939004 15953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:34:21.939076 15954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:34:21.939023 15956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:34:21.940542 15948 server_base.cc:1034] running on GCE node
I20260926 23:34:21.941483 15948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:34:21.943305 15948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:34:21.944689 15948 hybrid_clock.cc:648] HybridClock initialized: now 1790465661944554 us; error 83 us; skew 500 ppm
I20260926 23:34:21.945119 15948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:34:21 dist-test-slave-gwfx krb5kdc[15935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:34:21.949446 15948 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:34:21.951485 15948 webserver.cc:460] Webserver started at http://127.0.97.62:43993/ using document root <none> and password file <none>
I20260926 23:34:21.952358 15948 fs_manager.cc:359] Metadata directory not provided
I20260926 23:34:21.952474 15948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:34:21.952839 15948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:34:21.956665 15948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "ccac865f8548457ca50960ec81cd9c6e"
format_stamp: "Formatted at 2026-09-26 23:34:21 on dist-test-slave-gwfx"
server_key: "c3c8b301e151609dcf6eeba52363bb73"
server_key_iv: "108fcb6ebdd58e00bf72bbf50bb64563"
server_key_version: "encryptionkey@0"
I20260926 23:34:21.957587 15948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "ccac865f8548457ca50960ec81cd9c6e"
format_stamp: "Formatted at 2026-09-26 23:34:21 on dist-test-slave-gwfx"
server_key: "c3c8b301e151609dcf6eeba52363bb73"
server_key_iv: "108fcb6ebdd58e00bf72bbf50bb64563"
server_key_version: "encryptionkey@0"
I20260926 23:34:21.964305 15948 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 23:34:21.969161 15963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:21.971189 15948 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 23:34:21.971431 15948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "ccac865f8548457ca50960ec81cd9c6e"
format_stamp: "Formatted at 2026-09-26 23:34:21 on dist-test-slave-gwfx"
server_key: "c3c8b301e151609dcf6eeba52363bb73"
server_key_iv: "108fcb6ebdd58e00bf72bbf50bb64563"
server_key_version: "encryptionkey@0"
I20260926 23:34:21.971727 15948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:34:21.988373 15948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:34:21.993569 15948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:34:21.993851 15948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:34:22.015131 15948 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46435
I20260926 23:34:22.015178 16014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46435 every 8 connection(s)
I20260926 23:34:22.016673 15948 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:34:22.021582 16015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:34:22.026638 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 15948
I20260926 23:34:22.026899 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:34:22.027594 388 external_mini_cluster.cc:1428] Setting key e9e2992bcb7b4ab7e544c18f09499159
I20260926 23:34:22.036011 16015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e: Bootstrap starting.
I20260926 23:34:22.040830 16015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e: Neither blocks nor log segments found. Creating new log.
I20260926 23:34:22.042373 16015 log.cc:824] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e: Log is configured to *not* fsync() on all Append() calls
Sep 26 23:34:22 dist-test-slave-gwfx krb5kdc[15935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:34:22.046639 16015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e: No bootstrap required, opened a new log
I20260926 23:34:22.057399 16015 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccac865f8548457ca50960ec81cd9c6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46435 } }
I20260926 23:34:22.057919 16015 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:34:22.058024 16015 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccac865f8548457ca50960ec81cd9c6e, State: Initialized, Role: FOLLOWER
I20260926 23:34:22.058543 16018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:22.030145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58934 (local address 127.0.97.62:46435)
0926 23:34:22.031106 (+ 961us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:22.031118 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:22.031164 (+ 46us) server_negotiation.cc:407] Connection header received
0926 23:34:22.033726 (+ 2562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:22.033759 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:22.034272 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:22.034838 (+ 566us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:34:22.037126 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:22.038674 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:22.040979 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:22.041917 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:22.042118 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:22.047750 (+ 5632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:34:22.047761 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:34:22.047790 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:34:22.047849 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:34:22.052021 (+ 4172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:22.053282 (+ 1261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:22.053306 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:22.053327 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:22.053763 (+ 436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:22.054485 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:22.054554 (+ 69us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:22.054556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:22.055404 (+ 848us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:34:22.055712 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:22.056467 (+ 755us) server_negotiation.cc:299] Negotiation successful
0926 23:34:22.057180 (+ 713us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":456,"threads_started":1}
I20260926 23:34:22.058878 16015 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccac865f8548457ca50960ec81cd9c6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46435 } }
I20260926 23:34:22.059224 16015 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:34:22.059406 16015 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:34:22.059633 16015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:34:22.062757 16015 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccac865f8548457ca50960ec81cd9c6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46435 } }
I20260926 23:34:22.063342 16015 leader_election.cc:302] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccac865f8548457ca50960ec81cd9c6e; no voters:
I20260926 23:34:22.064029 16015 leader_election.cc:288] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:34:22.064252 16020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:34:22.065008 16020 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [term 1 LEADER]: Becoming Leader. State: Replica: ccac865f8548457ca50960ec81cd9c6e, State: Running, Role: LEADER
I20260926 23:34:22.065665 16020 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccac865f8548457ca50960ec81cd9c6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46435 } }
I20260926 23:34:22.066418 16015 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:34:22.069949 16022 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccac865f8548457ca50960ec81cd9c6e. Latest consensus state: current_term: 1 leader_uuid: "ccac865f8548457ca50960ec81cd9c6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccac865f8548457ca50960ec81cd9c6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46435 } } }
I20260926 23:34:22.070061 16021 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccac865f8548457ca50960ec81cd9c6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccac865f8548457ca50960ec81cd9c6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46435 } } }
I20260926 23:34:22.070986 16022 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [sys.catalog]: This master's current role is: LEADER
I20260926 23:34:22.071111 16021 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [sys.catalog]: This master's current role is: LEADER
I20260926 23:34:22.072525 16028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:34:22.079324 16028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:34:22.091969 16028 catalog_manager.cc:1347] Generated new cluster ID: 15657a014ee248b7a2b50e0902dcf253
I20260926 23:34:22.092094 16028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:34:22.117368 16028 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:34:22.119098 16028 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:34:22.126808 16028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e: Generated new TSK 0
I20260926 23:34:22.127767 16028 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:34:22.191887 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46435
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:34:22.584230 16041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:34:22.584606 16041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:34:22.584714 16041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:34:22.592818 16041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:34:22.592985 16041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:34:22.593114 16041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:34:22.604319 16041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:34:22.606010 16041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:34:22.607684 16041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:34:22.623574 16047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:34:22.623886 16049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:34:22.623607 16046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:34:22.624415 16041 server_base.cc:1034] running on GCE node
I20260926 23:34:22.625222 16041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:34:22.626581 16041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:34:22.627893 16041 hybrid_clock.cc:648] HybridClock initialized: now 1790465662627794 us; error 85 us; skew 500 ppm
I20260926 23:34:22.628322 16041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:34:22 dist-test-slave-gwfx krb5kdc[15935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:34:22.632416 16041 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:34:22.634384 16041 webserver.cc:460] Webserver started at http://127.0.97.1:40445/ using document root <none> and password file <none>
I20260926 23:34:22.635335 16041 fs_manager.cc:359] Metadata directory not provided
I20260926 23:34:22.635465 16041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:34:22.635852 16041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:34:22.639525 16041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/data/instance:
uuid: "ae8abc5b72a04cb59f4a9dc64d532cac"
format_stamp: "Formatted at 2026-09-26 23:34:22 on dist-test-slave-gwfx"
server_key: "00c43a93fd78032679f5511a10a0a81c"
server_key_iv: "1ad40631d1d9c466d7940c65184ea2d6"
server_key_version: "encryptionkey@0"
I20260926 23:34:22.640394 16041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ae8abc5b72a04cb59f4a9dc64d532cac"
format_stamp: "Formatted at 2026-09-26 23:34:22 on dist-test-slave-gwfx"
server_key: "00c43a93fd78032679f5511a10a0a81c"
server_key_iv: "1ad40631d1d9c466d7940c65184ea2d6"
server_key_version: "encryptionkey@0"
I20260926 23:34:22.646934 16041 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260926 23:34:22.651479 16056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:22.653287 16041 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 23:34:22.653460 16041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "ae8abc5b72a04cb59f4a9dc64d532cac"
format_stamp: "Formatted at 2026-09-26 23:34:22 on dist-test-slave-gwfx"
server_key: "00c43a93fd78032679f5511a10a0a81c"
server_key_iv: "1ad40631d1d9c466d7940c65184ea2d6"
server_key_version: "encryptionkey@0"
I20260926 23:34:22.653688 16041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:34:22.683254 16041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:34:22.688515 16041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:34:22.688824 16041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:34:22.690034 16041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:34:22.691728 16041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:34:22.691821 16041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:22.691921 16041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:34:22.691963 16041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:22.727767 16041 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36783
I20260926 23:34:22.727826 16168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36783 every 8 connection(s)
I20260926 23:34:22.729354 16041 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:34:22.730720 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 16041
I20260926 23:34:22.730942 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:34:22.731789 388 external_mini_cluster.cc:1428] Setting key 2aee10b9d752290c53df7b303a8a8236
Sep 26 23:34:22 dist-test-slave-gwfx krb5kdc[15935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:34:22.755009 16171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:22.732662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42597 (local address 127.0.97.62:46435)
0926 23:34:22.733172 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:22.733177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:22.734487 (+ 1310us) server_negotiation.cc:407] Connection header received
0926 23:34:22.737434 (+ 2947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:22.737439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:22.737523 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:22.737630 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:34:22.740142 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:22.740996 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:22.743403 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:22.743879 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:22.743994 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:22.748555 (+ 4561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:34:22.748561 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:34:22.748565 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:34:22.748608 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:34:22.751311 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:22.752230 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:22.752234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:22.752237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:22.752372 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:22.752900 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:22.752904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:22.752906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:22.753227 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:34:22.753373 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:22.754279 (+ 906us) server_negotiation.cc:299] Negotiation successful
0926 23:34:22.754569 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":262,"threads_started":1}
I20260926 23:34:22.756278 16172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:22.733392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.1:42597)
0926 23:34:22.734277 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:22.734331 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 23:34:22.737038 (+ 2707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:34:22.737955 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:34:22.737969 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:34:22.738907 (+ 938us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:34:22.739899 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:22.739926 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:22.741171 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:22.741178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:22.743219 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:22.743243 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:22.744136 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:22.744143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:22.744483 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:22.745349 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:34:22.745382 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:34:22.748251 (+ 2869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:34:22.751506 (+ 3255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:34:22.751518 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:34:22.751540 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:34:22.752088 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:34:22.752540 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:34:22.752545 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:34:22.752549 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:34:22.752775 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:34:22.753364 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:34:22.753376 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:34:22.753832 (+ 456us) client_negotiation.cc:769] Sending connection context
0926 23:34:22.754132 (+ 300us) client_negotiation.cc:240] Negotiation successful
0926 23:34:22.754845 (+ 713us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":368,"threads_started":1}
I20260926 23:34:22.758653 16169 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46435
I20260926 23:34:22.759140 16169 heartbeater.cc:464] Registering TS with master...
I20260926 23:34:22.760119 16169 heartbeater.cc:511] Master 127.0.97.62:46435 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 23:34:22.763082 15980 ts_manager.cc:194] Registered new tserver with Master: ae8abc5b72a04cb59f4a9dc64d532cac (127.0.97.1:36783)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 23:34:22.764991 15980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42597
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:34:22.796808 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46435
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:34:23.205787 16175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:34:23.206187 16175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:34:23.206298 16175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:34:23.214748 16175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:34:23.214938 16175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:34:23.215085 16175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:34:23.226843 16175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:34:23.228580 16175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:34:23.230221 16175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:34:23.246665 16183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:34:23.246651 16180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:34:23.246896 16181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:34:23.247675 16175 server_base.cc:1034] running on GCE node
I20260926 23:34:23.248492 16175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:34:23.249709 16175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:34:23.250946 16175 hybrid_clock.cc:648] HybridClock initialized: now 1790465663250867 us; error 68 us; skew 500 ppm
I20260926 23:34:23.251294 16175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:34:23 dist-test-slave-gwfx krb5kdc[15935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:34:23.255137 16175 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:34:23.257391 16175 webserver.cc:460] Webserver started at http://127.0.97.2:33279/ using document root <none> and password file <none>
I20260926 23:34:23.258394 16175 fs_manager.cc:359] Metadata directory not provided
I20260926 23:34:23.258538 16175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:34:23.258965 16175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:34:23.262787 16175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/data/instance:
uuid: "e9e7c7787b5040b9910b34091f82f6ce"
format_stamp: "Formatted at 2026-09-26 23:34:23 on dist-test-slave-gwfx"
server_key: "d04606eac364773339cde975cf39aa39"
server_key_iv: "19d1f082e1b5692deb59c99b58564859"
server_key_version: "encryptionkey@0"
I20260926 23:34:23.263805 16175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e9e7c7787b5040b9910b34091f82f6ce"
format_stamp: "Formatted at 2026-09-26 23:34:23 on dist-test-slave-gwfx"
server_key: "d04606eac364773339cde975cf39aa39"
server_key_iv: "19d1f082e1b5692deb59c99b58564859"
server_key_version: "encryptionkey@0"
I20260926 23:34:23.270895 16175 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260926 23:34:23.275336 16190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:23.277272 16175 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 23:34:23.277491 16175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "e9e7c7787b5040b9910b34091f82f6ce"
format_stamp: "Formatted at 2026-09-26 23:34:23 on dist-test-slave-gwfx"
server_key: "d04606eac364773339cde975cf39aa39"
server_key_iv: "19d1f082e1b5692deb59c99b58564859"
server_key_version: "encryptionkey@0"
I20260926 23:34:23.277740 16175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:34:23.293309 16175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:34:23.299461 16175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:34:23.299825 16175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:34:23.301122 16175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:34:23.302908 16175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:34:23.302997 16175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:23.303123 16175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:34:23.303174 16175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:23.340216 16175 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34735
I20260926 23:34:23.340260 16302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34735 every 8 connection(s)
I20260926 23:34:23.341967 16175 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:34:23.347976 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 16175
I20260926 23:34:23.348178 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:34:23.348668 388 external_mini_cluster.cc:1428] Setting key fa6c2cc0e94e5d1913e7c35fe5138013
Sep 26 23:34:23 dist-test-slave-gwfx krb5kdc[15935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:34:23.367203 16305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:23.345410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60205 (local address 127.0.97.62:46435)
0926 23:34:23.346033 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:23.346039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:23.347379 (+ 1340us) server_negotiation.cc:407] Connection header received
0926 23:34:23.349680 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:23.349686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:23.349791 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:23.349917 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:34:23.352136 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:23.353362 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:23.355331 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:23.355933 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:23.356066 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:23.360381 (+ 4315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:34:23.360388 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:34:23.360393 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:34:23.360445 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:34:23.363456 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:23.364464 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:23.364469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:23.364471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:23.364607 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:23.365100 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:23.365103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:23.365105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:23.365420 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:34:23.365538 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:23.366496 (+ 958us) server_negotiation.cc:299] Negotiation successful
0926 23:34:23.366767 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":359,"threads_started":1}
I20260926 23:34:23.368698 16306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:23.346154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.2:60205)
0926 23:34:23.347158 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:23.347214 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 23:34:23.349301 (+ 2087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:34:23.350234 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:34:23.350246 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:34:23.351094 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:34:23.351916 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:23.351942 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:23.353541 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:23.353546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:23.355153 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:23.355175 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:23.356201 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:23.356207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:23.356482 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:23.357313 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:34:23.357358 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:34:23.360046 (+ 2688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:34:23.363626 (+ 3580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:34:23.363638 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:34:23.363664 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:34:23.364321 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:34:23.364711 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:34:23.364716 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:34:23.364719 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:34:23.364997 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:34:23.365530 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:34:23.365539 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:34:23.366051 (+ 512us) client_negotiation.cc:769] Sending connection context
0926 23:34:23.366387 (+ 336us) client_negotiation.cc:240] Negotiation successful
0926 23:34:23.367083 (+ 696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":345,"threads_started":1}
I20260926 23:34:23.370549 16303 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46435
I20260926 23:34:23.370961 16303 heartbeater.cc:464] Registering TS with master...
I20260926 23:34:23.371935 16303 heartbeater.cc:511] Master 127.0.97.62:46435 requested a full tablet report, sending...
I20260926 23:34:23.373781 15980 ts_manager.cc:194] Registered new tserver with Master: e9e7c7787b5040b9910b34091f82f6ce (127.0.97.2:34735)
I20260926 23:34:23.374749 15980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60205
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:34:23.414772 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46435
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:34:23.769032 16169 heartbeater.cc:503] Master 127.0.97.62:46435 was elected leader, sending a full tablet report...
W20260926 23:34:23.826586 16309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:34:23.827028 16309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:34:23.827162 16309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:34:23.835911 16309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:34:23.836077 16309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:34:23.836241 16309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:34:23.847821 16309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:34:23.849610 16309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:34:23.851339 16309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:34:23.867518 16315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:34:23.867820 16317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:34:23.868021 16314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:34:23.868718 16309 server_base.cc:1034] running on GCE node
I20260926 23:34:23.869438 16309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:34:23.870590 16309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:34:23.871866 16309 hybrid_clock.cc:648] HybridClock initialized: now 1790465663871787 us; error 58 us; skew 500 ppm
I20260926 23:34:23.872267 16309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:34:23 dist-test-slave-gwfx krb5kdc[15935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:34:23.875993 16309 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:34:23.877995 16309 webserver.cc:460] Webserver started at http://127.0.97.3:43345/ using document root <none> and password file <none>
I20260926 23:34:23.878846 16309 fs_manager.cc:359] Metadata directory not provided
I20260926 23:34:23.878961 16309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:34:23.879269 16309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:34:23.882560 16309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/data/instance:
uuid: "1eb86cb9e5b64ad689580147ff05dcdb"
format_stamp: "Formatted at 2026-09-26 23:34:23 on dist-test-slave-gwfx"
server_key: "87c326a8254e6ee107818e51a93f4695"
server_key_iv: "dad5ceed5c86b397f758d128b01a1795"
server_key_version: "encryptionkey@0"
I20260926 23:34:23.883661 16309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1eb86cb9e5b64ad689580147ff05dcdb"
format_stamp: "Formatted at 2026-09-26 23:34:23 on dist-test-slave-gwfx"
server_key: "87c326a8254e6ee107818e51a93f4695"
server_key_iv: "dad5ceed5c86b397f758d128b01a1795"
server_key_version: "encryptionkey@0"
I20260926 23:34:23.890151 16309 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 23:34:23.894650 16324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:23.896622 16309 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:34:23.896806 16309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "1eb86cb9e5b64ad689580147ff05dcdb"
format_stamp: "Formatted at 2026-09-26 23:34:23 on dist-test-slave-gwfx"
server_key: "87c326a8254e6ee107818e51a93f4695"
server_key_iv: "dad5ceed5c86b397f758d128b01a1795"
server_key_version: "encryptionkey@0"
I20260926 23:34:23.897019 16309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:34:23.917603 16309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:34:23.922314 16309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:34:23.922595 16309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:34:23.923697 16309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:34:23.925309 16309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:34:23.925390 16309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:23.925551 16309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:34:23.925604 16309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:23.962306 16309 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35683
I20260926 23:34:23.962445 16436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35683 every 8 connection(s)
I20260926 23:34:23.963771 16309 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:34:23.965457 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 16309
I20260926 23:34:23.965632 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:34:23.966233 388 external_mini_cluster.cc:1428] Setting key ade90c820f6444cb2daba47b83156cbf
Sep 26 23:34:23 dist-test-slave-gwfx krb5kdc[15935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:34:23.989171 16439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:23.967324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50483 (local address 127.0.97.62:46435)
0926 23:34:23.968042 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:23.968048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:23.969086 (+ 1038us) server_negotiation.cc:407] Connection header received
0926 23:34:23.971344 (+ 2258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:23.971351 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:23.971458 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:23.971571 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:34:23.973531 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:23.974387 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:23.976584 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:23.977236 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:23.977381 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:23.982057 (+ 4676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:34:23.982064 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:34:23.982068 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:34:23.982116 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:34:23.985313 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:23.986221 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:23.986227 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:23.986229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:23.986368 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:23.986928 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:23.986932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:23.986935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:23.987277 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:34:23.987439 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:23.988348 (+ 909us) server_negotiation.cc:299] Negotiation successful
0926 23:34:23.988649 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":282,"threads_started":1}
I20260926 23:34:23.990437 16440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:23.967873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.3:50483)
0926 23:34:23.968901 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:23.968951 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 23:34:23.971009 (+ 2058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:34:23.971890 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:34:23.971901 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:34:23.972596 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:34:23.973340 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:23.973361 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:23.974557 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:23.974562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:23.976412 (+ 1850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:23.976430 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:23.977535 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:23.977544 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:23.977827 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:23.978800 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:34:23.978841 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:34:23.981743 (+ 2902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:34:23.985503 (+ 3760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:34:23.985516 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:34:23.985537 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:34:23.986072 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:34:23.986483 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:34:23.986489 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:34:23.986494 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:34:23.986793 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:34:23.987417 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:34:23.987427 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:34:23.987894 (+ 467us) client_negotiation.cc:769] Sending connection context
0926 23:34:23.988220 (+ 326us) client_negotiation.cc:240] Negotiation successful
0926 23:34:23.988897 (+ 677us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":384,"threads_started":1}
I20260926 23:34:23.992439 16437 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46435
I20260926 23:34:23.992787 16437 heartbeater.cc:464] Registering TS with master...
I20260926 23:34:23.993695 16437 heartbeater.cc:511] Master 127.0.97.62:46435 requested a full tablet report, sending...
I20260926 23:34:23.995658 15980 ts_manager.cc:194] Registered new tserver with Master: 1eb86cb9e5b64ad689580147ff05dcdb (127.0.97.3:35683)
I20260926 23:34:23.996690 15980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50483
I20260926 23:34:23.997220 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:34:24.019579 16439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:24.003839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58940 (local address 127.0.97.62:46435)
0926 23:34:24.004119 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:24.004127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:24.004773 (+ 646us) server_negotiation.cc:407] Connection header received
0926 23:34:24.004871 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:24.004876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:24.004965 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:24.005088 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:34:24.006618 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:24.007554 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:24.009133 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:24.009626 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:24.009769 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:24.012488 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:34:24.012495 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:34:24.012499 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:34:24.012548 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:34:24.016019 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:24.016962 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:24.016967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:24.016972 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:24.017152 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:24.017737 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:24.017742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:24.017746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:24.018244 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:34:24.018533 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:24.018920 (+ 387us) server_negotiation.cc:299] Negotiation successful
0926 23:34:24.019198 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 23:34:24.025080 15980 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:34:24.032963 15980 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:34:24.073725 16450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:24.048401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35683 (local address 127.0.0.1:46148)
0926 23:34:24.061856 (+ 13455us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:24.061894 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 23:34:24.062088 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:34:24.065922 (+ 3834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:34:24.065926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:34:24.065943 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:34:24.066337 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:24.066348 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:24.068212 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:24.068219 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:24.070457 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:24.070474 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:24.072517 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:24.072525 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:24.072739 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:24.072795 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 23:34:24.072989 (+ 194us) client_negotiation.cc:240] Negotiation successful
0926 23:34:24.073223 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11578,"mutex_wait_us":114,"thread_start_us":14500,"threads_started":1}
I20260926 23:34:24.073783 16452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:24.048542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46148 (local address 127.0.97.3:35683)
0926 23:34:24.065306 (+ 16764us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:24.065315 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:24.065348 (+ 33us) server_negotiation.cc:407] Connection header received
0926 23:34:24.065456 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:24.065463 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:24.065653 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:24.065836 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:34:24.066526 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:24.068036 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:24.070641 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:24.071907 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:24.072092 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:24.072365 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:24.073110 (+ 745us) server_negotiation.cc:299] Negotiation successful
0926 23:34:24.073360 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16575,"thread_start_us":16417,"threads_started":1}
I20260926 23:34:24.074915 16455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:24.050006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33556 (local address 127.0.97.2:34735)
0926 23:34:24.063825 (+ 13819us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:24.063835 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:24.065017 (+ 1182us) server_negotiation.cc:407] Connection header received
0926 23:34:24.065165 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:24.065175 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:24.065514 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:24.065778 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:34:24.067037 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:24.068635 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:24.071432 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:24.072964 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:24.073106 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:24.073295 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:24.074317 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0926 23:34:24.074523 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13530,"thread_start_us":13394,"threads_started":1}
I20260926 23:34:24.075227 16454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:24.049449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33634 (local address 127.0.97.1:36783)
0926 23:34:24.064459 (+ 15010us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:24.064468 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:24.064502 (+ 34us) server_negotiation.cc:407] Connection header received
0926 23:34:24.064622 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:24.064630 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:24.064942 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:24.065175 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:34:24.066159 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:24.067706 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:24.071445 (+ 3739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:24.073739 (+ 2294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:24.073887 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:24.074150 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:24.074663 (+ 513us) server_negotiation.cc:299] Negotiation successful
0926 23:34:24.074896 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14792,"thread_start_us":14562,"threads_started":1}
I20260926 23:34:24.075278 16451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:24.048480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36783 (local address 127.0.0.1:33634)
0926 23:34:24.063872 (+ 15392us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:24.063891 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:34:24.064016 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:34:24.065331 (+ 1315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:34:24.065340 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:34:24.065379 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:34:24.065947 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:24.065964 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:24.067909 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:24.067915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:24.071185 (+ 3270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:24.071211 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:24.074288 (+ 3077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:24.074293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:24.074451 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:24.074489 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 23:34:24.074585 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 23:34:24.074818 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15265,"mutex_wait_us":120,"thread_start_us":11340,"threads_started":1}
I20260926 23:34:24.076015 16453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:24.048722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34735 (local address 127.0.0.1:33556)
0926 23:34:24.064565 (+ 15843us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:24.064583 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:34:24.064702 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:34:24.065792 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:34:24.065797 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:34:24.065829 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:34:24.066832 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:24.066845 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:24.068856 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:24.068862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:24.071202 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:24.071211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:24.073468 (+ 2257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:24.073474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:24.073663 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:24.073702 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 23:34:24.073785 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 23:34:24.075550 (+ 1765us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15713,"mutex_wait_us":644,"thread_start_us":15555,"threads_started":1}
I20260926 23:34:24.078595 16372 tablet_service.cc:1467] Processing CreateTablet for tablet 8b8869b2d8b84583a47086df56a88303 (DEFAULT_TABLE table=test-table [id=1107e631c5bf4a2298fc628b3d13a152]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:34:24.080152 16372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b8869b2d8b84583a47086df56a88303. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:34:24.079609 16104 tablet_service.cc:1467] Processing CreateTablet for tablet 8b8869b2d8b84583a47086df56a88303 (DEFAULT_TABLE table=test-table [id=1107e631c5bf4a2298fc628b3d13a152]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:34:24.079949 16238 tablet_service.cc:1467] Processing CreateTablet for tablet 8b8869b2d8b84583a47086df56a88303 (DEFAULT_TABLE table=test-table [id=1107e631c5bf4a2298fc628b3d13a152]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:34:24.081307 16238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b8869b2d8b84583a47086df56a88303. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:34:24.081308 16104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b8869b2d8b84583a47086df56a88303. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:34:24.094053 16456 tablet_bootstrap.cc:492] T 8b8869b2d8b84583a47086df56a88303 P 1eb86cb9e5b64ad689580147ff05dcdb: Bootstrap starting.
I20260926 23:34:24.097729 16456 tablet_bootstrap.cc:654] T 8b8869b2d8b84583a47086df56a88303 P 1eb86cb9e5b64ad689580147ff05dcdb: Neither blocks nor log segments found. Creating new log.
I20260926 23:34:24.098454 16457 tablet_bootstrap.cc:492] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac: Bootstrap starting.
I20260926 23:34:24.099527 16456 log.cc:824] T 8b8869b2d8b84583a47086df56a88303 P 1eb86cb9e5b64ad689580147ff05dcdb: Log is configured to *not* fsync() on all Append() calls
I20260926 23:34:24.100239 16458 tablet_bootstrap.cc:492] T 8b8869b2d8b84583a47086df56a88303 P e9e7c7787b5040b9910b34091f82f6ce: Bootstrap starting.
I20260926 23:34:24.104822 16458 tablet_bootstrap.cc:654] T 8b8869b2d8b84583a47086df56a88303 P e9e7c7787b5040b9910b34091f82f6ce: Neither blocks nor log segments found. Creating new log.
I20260926 23:34:24.105933 16457 tablet_bootstrap.cc:654] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac: Neither blocks nor log segments found. Creating new log.
I20260926 23:34:24.106158 16458 log.cc:824] T 8b8869b2d8b84583a47086df56a88303 P e9e7c7787b5040b9910b34091f82f6ce: Log is configured to *not* fsync() on all Append() calls
I20260926 23:34:24.107577 16457 log.cc:824] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac: Log is configured to *not* fsync() on all Append() calls
I20260926 23:34:24.108848 16456 tablet_bootstrap.cc:492] T 8b8869b2d8b84583a47086df56a88303 P 1eb86cb9e5b64ad689580147ff05dcdb: No bootstrap required, opened a new log
I20260926 23:34:24.109193 16456 ts_tablet_manager.cc:1397] T 8b8869b2d8b84583a47086df56a88303 P 1eb86cb9e5b64ad689580147ff05dcdb: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260926 23:34:24.109588 16458 tablet_bootstrap.cc:492] T 8b8869b2d8b84583a47086df56a88303 P e9e7c7787b5040b9910b34091f82f6ce: No bootstrap required, opened a new log
I20260926 23:34:24.109905 16458 ts_tablet_manager.cc:1397] T 8b8869b2d8b84583a47086df56a88303 P e9e7c7787b5040b9910b34091f82f6ce: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 23:34:24.111671 16457 tablet_bootstrap.cc:492] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac: No bootstrap required, opened a new log
I20260926 23:34:24.112046 16457 ts_tablet_manager.cc:1397] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20260926 23:34:24.117991 16458 raft_consensus.cc:348] T 8b8869b2d8b84583a47086df56a88303 P e9e7c7787b5040b9910b34091f82f6ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb86cb9e5b64ad689580147ff05dcdb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35683 } } peers { permanent_uuid: "ae8abc5b72a04cb59f4a9dc64d532cac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36783 } } peers { permanent_uuid: "e9e7c7787b5040b9910b34091f82f6ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34735 } }
I20260926 23:34:24.118459 16458 raft_consensus.cc:374] T 8b8869b2d8b84583a47086df56a88303 P e9e7c7787b5040b9910b34091f82f6ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:34:24.118758 16458 raft_consensus.cc:729] T 8b8869b2d8b84583a47086df56a88303 P e9e7c7787b5040b9910b34091f82f6ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9e7c7787b5040b9910b34091f82f6ce, State: Initialized, Role: FOLLOWER
I20260926 23:34:24.119539 16458 consensus_queue.cc:260] T 8b8869b2d8b84583a47086df56a88303 P e9e7c7787b5040b9910b34091f82f6ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb86cb9e5b64ad689580147ff05dcdb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35683 } } peers { permanent_uuid: "ae8abc5b72a04cb59f4a9dc64d532cac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36783 } } peers { permanent_uuid: "e9e7c7787b5040b9910b34091f82f6ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34735 } }
I20260926 23:34:24.120893 16303 heartbeater.cc:503] Master 127.0.97.62:46435 was elected leader, sending a full tablet report...
I20260926 23:34:24.120831 16456 raft_consensus.cc:348] T 8b8869b2d8b84583a47086df56a88303 P 1eb86cb9e5b64ad689580147ff05dcdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb86cb9e5b64ad689580147ff05dcdb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35683 } } peers { permanent_uuid: "ae8abc5b72a04cb59f4a9dc64d532cac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36783 } } peers { permanent_uuid: "e9e7c7787b5040b9910b34091f82f6ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34735 } }
I20260926 23:34:24.121452 16456 raft_consensus.cc:374] T 8b8869b2d8b84583a47086df56a88303 P 1eb86cb9e5b64ad689580147ff05dcdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:34:24.121568 16456 raft_consensus.cc:729] T 8b8869b2d8b84583a47086df56a88303 P 1eb86cb9e5b64ad689580147ff05dcdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb86cb9e5b64ad689580147ff05dcdb, State: Initialized, Role: FOLLOWER
I20260926 23:34:24.122407 16456 consensus_queue.cc:260] T 8b8869b2d8b84583a47086df56a88303 P 1eb86cb9e5b64ad689580147ff05dcdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb86cb9e5b64ad689580147ff05dcdb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35683 } } peers { permanent_uuid: "ae8abc5b72a04cb59f4a9dc64d532cac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36783 } } peers { permanent_uuid: "e9e7c7787b5040b9910b34091f82f6ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34735 } }
I20260926 23:34:24.122809 16458 ts_tablet_manager.cc:1428] T 8b8869b2d8b84583a47086df56a88303 P e9e7c7787b5040b9910b34091f82f6ce: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260926 23:34:24.123967 16437 heartbeater.cc:503] Master 127.0.97.62:46435 was elected leader, sending a full tablet report...
I20260926 23:34:24.124755 16456 ts_tablet_manager.cc:1428] T 8b8869b2d8b84583a47086df56a88303 P 1eb86cb9e5b64ad689580147ff05dcdb: Time spent starting tablet: real 0.015s user 0.013s sys 0.003s
I20260926 23:34:24.123500 16457 raft_consensus.cc:348] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb86cb9e5b64ad689580147ff05dcdb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35683 } } peers { permanent_uuid: "ae8abc5b72a04cb59f4a9dc64d532cac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36783 } } peers { permanent_uuid: "e9e7c7787b5040b9910b34091f82f6ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34735 } }
I20260926 23:34:24.125924 16457 raft_consensus.cc:374] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:34:24.126075 16457 raft_consensus.cc:729] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8abc5b72a04cb59f4a9dc64d532cac, State: Initialized, Role: FOLLOWER
I20260926 23:34:24.126881 16457 consensus_queue.cc:260] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb86cb9e5b64ad689580147ff05dcdb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35683 } } peers { permanent_uuid: "ae8abc5b72a04cb59f4a9dc64d532cac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36783 } } peers { permanent_uuid: "e9e7c7787b5040b9910b34091f82f6ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34735 } }
I20260926 23:34:24.129843 16457 ts_tablet_manager.cc:1428] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
W20260926 23:34:24.215780 16438 tablet.cc:2369] T 8b8869b2d8b84583a47086df56a88303 P 1eb86cb9e5b64ad689580147ff05dcdb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:34:24.232527 16170 tablet.cc:2369] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:34:24.344934 16304 tablet.cc:2369] T 8b8869b2d8b84583a47086df56a88303 P e9e7c7787b5040b9910b34091f82f6ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:34:24.451300 16464 raft_consensus.cc:482] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:34:24.451613 16464 raft_consensus.cc:504] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb86cb9e5b64ad689580147ff05dcdb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35683 } } peers { permanent_uuid: "ae8abc5b72a04cb59f4a9dc64d532cac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36783 } } peers { permanent_uuid: "e9e7c7787b5040b9910b34091f82f6ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34735 } }
I20260926 23:34:24.453490 16464 leader_election.cc:288] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1eb86cb9e5b64ad689580147ff05dcdb (127.0.97.3:35683), e9e7c7787b5040b9910b34091f82f6ce (127.0.97.2:34735)
I20260926 23:34:24.461948 16465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:24.454003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35683 (local address 127.0.97.1:43711)
0926 23:34:24.455050 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:24.455088 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 23:34:24.455342 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:34:24.455772 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:34:24.455778 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:34:24.455813 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:34:24.456376 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:24.456394 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:24.457521 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:24.457528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:24.459684 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:24.459699 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:24.461002 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:24.461010 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:24.461193 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:24.461246 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 23:34:24.461364 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 23:34:24.461507 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"mutex_wait_us":106,"thread_start_us":519,"threads_started":1}
I20260926 23:34:24.461956 16452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:24.454003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43711 (local address 127.0.97.3:35683)
0926 23:34:24.454322 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:24.454330 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:24.455235 (+ 905us) server_negotiation.cc:407] Connection header received
0926 23:34:24.455519 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:24.455524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:24.455608 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:24.455703 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:34:24.456604 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:24.457378 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:24.459858 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:24.460752 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:24.460861 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:24.460950 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:24.461489 (+ 539us) server_negotiation.cc:299] Negotiation successful
0926 23:34:24.461649 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260926 23:34:24.463137 16466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:24.454325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34735 (local address 127.0.97.1:50363)
0926 23:34:24.455340 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:24.455359 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:34:24.455557 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:34:24.456079 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:34:24.456083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:34:24.456112 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:34:24.456563 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:24.456578 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:24.458078 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:24.458083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:24.459909 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:24.459926 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:24.461359 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:24.461372 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:24.461564 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:24.461605 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 23:34:24.462514 (+ 909us) client_negotiation.cc:240] Negotiation successful
0926 23:34:24.462713 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"mutex_wait_us":27,"thread_start_us":613,"threads_started":1}
I20260926 23:34:24.463480 16455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:24.454279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50363 (local address 127.0.97.2:34735)
0926 23:34:24.454502 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:24.454506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:24.455493 (+ 987us) server_negotiation.cc:407] Connection header received
0926 23:34:24.455779 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:24.455785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:24.455880 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:24.456011 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:34:24.456876 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:24.457896 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:24.460106 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:24.461055 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:24.461221 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:24.461327 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:24.461867 (+ 540us) server_negotiation.cc:299] Negotiation successful
0926 23:34:24.462506 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 23:34:24.463332 16392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b8869b2d8b84583a47086df56a88303" candidate_uuid: "ae8abc5b72a04cb59f4a9dc64d532cac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb86cb9e5b64ad689580147ff05dcdb" is_pre_election: true
I20260926 23:34:24.463939 16392 raft_consensus.cc:2454] T 8b8869b2d8b84583a47086df56a88303 P 1eb86cb9e5b64ad689580147ff05dcdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae8abc5b72a04cb59f4a9dc64d532cac in term 0.
I20260926 23:34:24.464771 16060 leader_election.cc:302] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb86cb9e5b64ad689580147ff05dcdb, ae8abc5b72a04cb59f4a9dc64d532cac; no voters:
I20260926 23:34:24.464887 16258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b8869b2d8b84583a47086df56a88303" candidate_uuid: "ae8abc5b72a04cb59f4a9dc64d532cac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9e7c7787b5040b9910b34091f82f6ce" is_pre_election: true
I20260926 23:34:24.465284 16464 raft_consensus.cc:2781] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:34:24.465498 16464 raft_consensus.cc:482] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:34:24.465621 16464 raft_consensus.cc:3037] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:34:24.465783 16258 raft_consensus.cc:2454] T 8b8869b2d8b84583a47086df56a88303 P e9e7c7787b5040b9910b34091f82f6ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae8abc5b72a04cb59f4a9dc64d532cac in term 0.
I20260926 23:34:24.468556 16464 raft_consensus.cc:504] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb86cb9e5b64ad689580147ff05dcdb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35683 } } peers { permanent_uuid: "ae8abc5b72a04cb59f4a9dc64d532cac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36783 } } peers { permanent_uuid: "e9e7c7787b5040b9910b34091f82f6ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34735 } }
I20260926 23:34:24.469331 16464 leader_election.cc:288] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac [CANDIDATE]: Term 1 election: Requested vote from peers 1eb86cb9e5b64ad689580147ff05dcdb (127.0.97.3:35683), e9e7c7787b5040b9910b34091f82f6ce (127.0.97.2:34735)
I20260926 23:34:24.470108 16258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b8869b2d8b84583a47086df56a88303" candidate_uuid: "ae8abc5b72a04cb59f4a9dc64d532cac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9e7c7787b5040b9910b34091f82f6ce"
I20260926 23:34:24.470117 16392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b8869b2d8b84583a47086df56a88303" candidate_uuid: "ae8abc5b72a04cb59f4a9dc64d532cac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb86cb9e5b64ad689580147ff05dcdb"
I20260926 23:34:24.470398 16392 raft_consensus.cc:3037] T 8b8869b2d8b84583a47086df56a88303 P 1eb86cb9e5b64ad689580147ff05dcdb [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:34:24.470425 16258 raft_consensus.cc:3037] T 8b8869b2d8b84583a47086df56a88303 P e9e7c7787b5040b9910b34091f82f6ce [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:34:24.474116 16258 raft_consensus.cc:2454] T 8b8869b2d8b84583a47086df56a88303 P e9e7c7787b5040b9910b34091f82f6ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae8abc5b72a04cb59f4a9dc64d532cac in term 1.
I20260926 23:34:24.474116 16392 raft_consensus.cc:2454] T 8b8869b2d8b84583a47086df56a88303 P 1eb86cb9e5b64ad689580147ff05dcdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae8abc5b72a04cb59f4a9dc64d532cac in term 1.
I20260926 23:34:24.474982 16058 leader_election.cc:302] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae8abc5b72a04cb59f4a9dc64d532cac, e9e7c7787b5040b9910b34091f82f6ce; no voters:
I20260926 23:34:24.475525 16464 raft_consensus.cc:2781] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:34:24.476130 16464 raft_consensus.cc:686] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac [term 1 LEADER]: Becoming Leader. State: Replica: ae8abc5b72a04cb59f4a9dc64d532cac, State: Running, Role: LEADER
I20260926 23:34:24.476778 16464 consensus_queue.cc:237] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb86cb9e5b64ad689580147ff05dcdb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35683 } } peers { permanent_uuid: "ae8abc5b72a04cb59f4a9dc64d532cac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36783 } } peers { permanent_uuid: "e9e7c7787b5040b9910b34091f82f6ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34735 } }
I20260926 23:34:24.483153 15980 catalog_manager.cc:5495] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac reported cstate change: term changed from 0 to 1, leader changed from <none> to ae8abc5b72a04cb59f4a9dc64d532cac (127.0.97.1). New cstate: current_term: 1 leader_uuid: "ae8abc5b72a04cb59f4a9dc64d532cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb86cb9e5b64ad689580147ff05dcdb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae8abc5b72a04cb59f4a9dc64d532cac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9e7c7787b5040b9910b34091f82f6ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34735 } health_report { overall_health: UNKNOWN } } }
I20260926 23:34:24.538789 16454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:24.531836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33650 (local address 127.0.97.1:36783)
0926 23:34:24.532042 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:24.532046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:24.532543 (+ 497us) server_negotiation.cc:407] Connection header received
0926 23:34:24.532768 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:24.532773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:24.532879 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:24.533057 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:34:24.533640 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:24.534430 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:24.536119 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:24.536562 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:24.536725 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:24.537323 (+ 598us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:34:24.537961 (+ 638us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:34:24.538097 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:24.538219 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 23:34:24.538302 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 23:34:24.549666 16392 raft_consensus.cc:1261] T 8b8869b2d8b84583a47086df56a88303 P 1eb86cb9e5b64ad689580147ff05dcdb [term 1 FOLLOWER]: Refusing update from remote peer ae8abc5b72a04cb59f4a9dc64d532cac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:34:24.549695 16258 raft_consensus.cc:1261] T 8b8869b2d8b84583a47086df56a88303 P e9e7c7787b5040b9910b34091f82f6ce [term 1 FOLLOWER]: Refusing update from remote peer ae8abc5b72a04cb59f4a9dc64d532cac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:34:24.550875 16467 consensus_queue.cc:1035] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9e7c7787b5040b9910b34091f82f6ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:34:24.551378 16464 consensus_queue.cc:1035] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb86cb9e5b64ad689580147ff05dcdb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:34:24.572012 16473 mvcc.cc:205] Tried to move back new op lower bound from 7333747361981878272 to 7333747361701212160. Current Snapshot: MvccSnapshot[applied={T|T < 7333747361981878272}]
I20260926 23:34:24.582544 15979 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:34:24.582928 15979 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:34:24.587471 15979 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e: Sending DeleteTablet for 3 replicas of tablet 8b8869b2d8b84583a47086df56a88303
I20260926 23:34:24.589192 16372 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b8869b2d8b84583a47086df56a88303 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:34:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46148
I20260926 23:34:24.589183 16238 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b8869b2d8b84583a47086df56a88303 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:34:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33556
I20260926 23:34:24.589414 16104 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b8869b2d8b84583a47086df56a88303 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:34:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33634
I20260926 23:34:24.590420 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 16041
I20260926 23:34:24.602142 16483 tablet_replica.cc:330] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac: stopping tablet replica
I20260926 23:34:24.602756 16483 raft_consensus.cc:2229] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:34:24.603525 16483 raft_consensus.cc:2258] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:34:24.606088 16483 ts_tablet_manager.cc:1905] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:34:24.614478 16483 ts_tablet_manager.cc:1918] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:34:24.614737 16483 log.cc:1196] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/wal/wals/8b8869b2d8b84583a47086df56a88303
I20260926 23:34:24.615289 16483 ts_tablet_manager.cc:1939] T 8b8869b2d8b84583a47086df56a88303 P ae8abc5b72a04cb59f4a9dc64d532cac: Deleting consensus metadata
I20260926 23:34:24.617698 15964 catalog_manager.cc:4841] TS ae8abc5b72a04cb59f4a9dc64d532cac (127.0.97.1:36783): tablet 8b8869b2d8b84583a47086df56a88303 (table test-table [id=1107e631c5bf4a2298fc628b3d13a152]) successfully deleted
I20260926 23:34:24.618618 16481 tablet_replica.cc:330] T 8b8869b2d8b84583a47086df56a88303 P e9e7c7787b5040b9910b34091f82f6ce: stopping tablet replica
I20260926 23:34:24.619134 16481 raft_consensus.cc:2229] T 8b8869b2d8b84583a47086df56a88303 P e9e7c7787b5040b9910b34091f82f6ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:34:24.619729 16481 raft_consensus.cc:2258] T 8b8869b2d8b84583a47086df56a88303 P e9e7c7787b5040b9910b34091f82f6ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:34:24.620584 16482 tablet_replica.cc:330] T 8b8869b2d8b84583a47086df56a88303 P 1eb86cb9e5b64ad689580147ff05dcdb: stopping tablet replica
I20260926 23:34:24.621016 16482 raft_consensus.cc:2229] T 8b8869b2d8b84583a47086df56a88303 P 1eb86cb9e5b64ad689580147ff05dcdb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:34:24.621842 16481 ts_tablet_manager.cc:1905] T 8b8869b2d8b84583a47086df56a88303 P e9e7c7787b5040b9910b34091f82f6ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:34:24.621610 16482 raft_consensus.cc:2258] T 8b8869b2d8b84583a47086df56a88303 P 1eb86cb9e5b64ad689580147ff05dcdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:34:24.624487 16482 ts_tablet_manager.cc:1905] T 8b8869b2d8b84583a47086df56a88303 P 1eb86cb9e5b64ad689580147ff05dcdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:34:24.628997 16481 ts_tablet_manager.cc:1918] T 8b8869b2d8b84583a47086df56a88303 P e9e7c7787b5040b9910b34091f82f6ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:34:24.629211 16481 log.cc:1196] T 8b8869b2d8b84583a47086df56a88303 P e9e7c7787b5040b9910b34091f82f6ce: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/wal/wals/8b8869b2d8b84583a47086df56a88303
I20260926 23:34:24.629742 16481 ts_tablet_manager.cc:1939] T 8b8869b2d8b84583a47086df56a88303 P e9e7c7787b5040b9910b34091f82f6ce: Deleting consensus metadata
Sep 26 23:34:24 dist-test-slave-gwfx krb5kdc[15935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:34:24.631946 15965 catalog_manager.cc:4841] TS e9e7c7787b5040b9910b34091f82f6ce (127.0.97.2:34735): tablet 8b8869b2d8b84583a47086df56a88303 (table test-table [id=1107e631c5bf4a2298fc628b3d13a152]) successfully deleted
I20260926 23:34:24.632838 16482 ts_tablet_manager.cc:1918] T 8b8869b2d8b84583a47086df56a88303 P 1eb86cb9e5b64ad689580147ff05dcdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:34:24.633041 16482 log.cc:1196] T 8b8869b2d8b84583a47086df56a88303 P 1eb86cb9e5b64ad689580147ff05dcdb: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/wal/wals/8b8869b2d8b84583a47086df56a88303
I20260926 23:34:24.633510 16482 ts_tablet_manager.cc:1939] T 8b8869b2d8b84583a47086df56a88303 P 1eb86cb9e5b64ad689580147ff05dcdb: Deleting consensus metadata
I20260926 23:34:24.635439 15967 catalog_manager.cc:4841] TS 1eb86cb9e5b64ad689580147ff05dcdb (127.0.97.3:35683): tablet 8b8869b2d8b84583a47086df56a88303 (table test-table [id=1107e631c5bf4a2298fc628b3d13a152]) successfully deleted
I20260926 23:34:24.640975 16454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:24.592131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33658 (local address 127.0.97.1:36783)
0926 23:34:24.592389 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:24.592398 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:24.616426 (+ 24028us) server_negotiation.cc:407] Connection header received
0926 23:34:24.617140 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:24.617150 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:24.617370 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:24.617467 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:34:24.620604 (+ 3137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:24.621815 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:24.624122 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:24.624866 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:24.624985 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:24.631949 (+ 6964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:34:24.631967 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:34:24.631994 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:34:24.632066 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:34:24.636834 (+ 4768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:24.637933 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:24.637945 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:24.637949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:24.638203 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:24.638860 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:24.638866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:24.638869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:24.639442 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:34:24.639845 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:24.640169 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 23:34:24.640493 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 23:34:24.641983 16164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:34:24.748461 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 16041
I20260926 23:34:24.768500 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 16175
Sep 26 23:34:24 dist-test-slave-gwfx krb5kdc[15935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:34:24.791772 16455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:24.769460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33570 (local address 127.0.97.2:34735)
0926 23:34:24.769829 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:24.769838 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:24.769866 (+ 28us) server_negotiation.cc:407] Connection header received
0926 23:34:24.769944 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:24.769948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:24.770033 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:24.770132 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:34:24.771470 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:24.772339 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:24.773754 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:24.774172 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:24.774275 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:24.785073 (+ 10798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:34:24.785095 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:34:24.785114 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:34:24.785167 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:34:24.788370 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:24.789518 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:24.789526 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:24.789528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:24.789682 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:24.790163 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:24.790168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:24.790170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:24.790630 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:34:24.790952 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:24.791286 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 23:34:24.791489 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260926 23:34:24.792711 16298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:34:24.887567 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 16175
I20260926 23:34:24.903785 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 16309
Sep 26 23:34:24 dist-test-slave-gwfx krb5kdc[15935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:34:24.921671 16452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:24.904671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46154 (local address 127.0.97.3:35683)
0926 23:34:24.904863 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:24.904867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:24.904890 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:34:24.905021 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:24.905025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:24.905130 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:24.905236 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:34:24.906840 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:24.907703 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:24.909221 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:24.909683 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:24.909820 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:24.915363 (+ 5543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:34:24.915383 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:34:24.915402 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:34:24.915454 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:34:24.918432 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:24.919415 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:24.919424 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:24.919427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:24.919582 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:24.919976 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:24.919980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:24.919982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:24.920393 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:34:24.920680 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:24.921064 (+ 384us) server_negotiation.cc:299] Negotiation successful
0926 23:34:24.921312 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 23:34:24.922351 16432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:34:25.022773 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 16309
I20260926 23:34:25.041541 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 15948
I20260926 23:34:25.043141 16010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:34:25.171280 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 15948
I20260926 23:34:45.193660 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46435
--webserver_interface=127.0.97.62
--webserver_port=43993
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:34:45.577524 16489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:34:45.577967 16489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:34:45.578200 16489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:34:45.588111 16489 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 23:34:45.588256 16489 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 23:34:45.588323 16489 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 23:34:45.588393 16489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:34:45.588467 16489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:34:45.588563 16489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:34:45.588625 16489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:34:45.588699 16489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:34:45.599421 16489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=43993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:34:45.600883 16489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:34:45.602352 16489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:34:45.627427 16498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:34:45.627439 16496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:34:45.627439 16495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:34:45.628412 16489 server_base.cc:1034] running on GCE node
I20260926 23:34:45.629412 16489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:34:45.631229 16489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:34:45.632536 16489 hybrid_clock.cc:648] HybridClock initialized: now 1790465685632434 us; error 71 us; skew 500 ppm
I20260926 23:34:45.633014 16489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:34:45 dist-test-slave-gwfx krb5kdc[15935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:34:45.637351 16489 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:34:45.639462 16489 webserver.cc:460] Webserver started at http://127.0.97.62:43993/ using document root <none> and password file <none>
I20260926 23:34:45.640341 16489 fs_manager.cc:359] Metadata directory not provided
I20260926 23:34:45.640482 16489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:34:45.647524 16489 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260926 23:34:45.651332 16505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:45.653275 16489 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260926 23:34:45.653527 16489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "ccac865f8548457ca50960ec81cd9c6e"
format_stamp: "Formatted at 2026-09-26 23:34:21 on dist-test-slave-gwfx"
server_key: "c3c8b301e151609dcf6eeba52363bb73"
server_key_iv: "108fcb6ebdd58e00bf72bbf50bb64563"
server_key_version: "encryptionkey@0"
I20260926 23:34:45.654784 16489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:34:45.679284 16489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:34:45.684516 16489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:34:45.684835 16489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:34:45.706317 16489 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46435
I20260926 23:34:45.706375 16556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46435 every 8 connection(s)
I20260926 23:34:45.708015 16489 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:34:45.715214 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 16489
I20260926 23:34:45.716369 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:36783
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46435
--webserver_port=40445
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:34:45.717926 16557 sys_catalog.cc:263] Verifying existing consensus state
I20260926 23:34:45.723637 16557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e: Bootstrap starting.
I20260926 23:34:45.750018 16557 log.cc:824] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e: Log is configured to *not* fsync() on all Append() calls
I20260926 23:34:45.765909 16557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 23:34:45.766712 16557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e: Bootstrap complete.
I20260926 23:34:45.777525 16557 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccac865f8548457ca50960ec81cd9c6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46435 } }
I20260926 23:34:45.778587 16557 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccac865f8548457ca50960ec81cd9c6e, State: Initialized, Role: FOLLOWER
I20260926 23:34:45.779495 16557 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccac865f8548457ca50960ec81cd9c6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46435 } }
I20260926 23:34:45.779776 16557 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:34:45.779933 16557 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:34:45.780128 16557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [term 1 FOLLOWER]: Advancing to term 2
I20260926 23:34:45.785033 16557 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccac865f8548457ca50960ec81cd9c6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46435 } }
I20260926 23:34:45.785742 16557 leader_election.cc:302] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccac865f8548457ca50960ec81cd9c6e; no voters:
I20260926 23:34:45.786567 16557 leader_election.cc:288] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 23:34:45.786752 16561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [term 2 FOLLOWER]: Leader election won for term 2
I20260926 23:34:45.788152 16561 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [term 2 LEADER]: Becoming Leader. State: Replica: ccac865f8548457ca50960ec81cd9c6e, State: Running, Role: LEADER
I20260926 23:34:45.788921 16561 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccac865f8548457ca50960ec81cd9c6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46435 } }
I20260926 23:34:45.789423 16557 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:34:45.792408 16562 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ccac865f8548457ca50960ec81cd9c6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccac865f8548457ca50960ec81cd9c6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46435 } } }
I20260926 23:34:45.792341 16563 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccac865f8548457ca50960ec81cd9c6e. Latest consensus state: current_term: 2 leader_uuid: "ccac865f8548457ca50960ec81cd9c6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccac865f8548457ca50960ec81cd9c6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46435 } } }
I20260926 23:34:45.792933 16562 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [sys.catalog]: This master's current role is: LEADER
I20260926 23:34:45.792989 16563 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [sys.catalog]: This master's current role is: LEADER
I20260926 23:34:45.794775 16567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:34:45.806257 16567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:34:45.808065 16567 catalog_manager.cc:1259] Loaded cluster ID: 15657a014ee248b7a2b50e0902dcf253
I20260926 23:34:45.808290 16567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:34:45.813092 16567 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:34:45.815610 16567 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e: Loaded TSK: 0
I20260926 23:34:45.860467 16567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e: Generated new TSK 1
I20260926 23:34:45.861258 16567 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 23:34:46.124305 16559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:34:46.124737 16559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:34:46.124889 16559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:34:46.133533 16559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:34:46.133719 16559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:34:46.133896 16559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:34:46.145694 16559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:36783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=40445
--tserver_master_addrs=127.0.97.62:46435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:34:46.147620 16559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:34:46.149358 16559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:34:46.170154 16584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:34:46.170145 16585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:34:46.170835 16587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:34:46.173957 16559 server_base.cc:1034] running on GCE node
I20260926 23:34:46.174942 16559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:34:46.176347 16559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:34:46.177624 16559 hybrid_clock.cc:648] HybridClock initialized: now 1790465686177581 us; error 44 us; skew 500 ppm
I20260926 23:34:46.178077 16559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:34:46 dist-test-slave-gwfx krb5kdc[15935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:34:46.182040 16559 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:34:46.184073 16559 webserver.cc:460] Webserver started at http://127.0.97.1:40445/ using document root <none> and password file <none>
I20260926 23:34:46.185000 16559 fs_manager.cc:359] Metadata directory not provided
I20260926 23:34:46.185153 16559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:34:46.193202 16559 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:34:46.197160 16594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:46.199368 16559 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 23:34:46.199688 16559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "ae8abc5b72a04cb59f4a9dc64d532cac"
format_stamp: "Formatted at 2026-09-26 23:34:22 on dist-test-slave-gwfx"
server_key: "00c43a93fd78032679f5511a10a0a81c"
server_key_iv: "1ad40631d1d9c466d7940c65184ea2d6"
server_key_version: "encryptionkey@0"
I20260926 23:34:46.201077 16559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:34:46.233150 16559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:34:46.238250 16559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:34:46.238580 16559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:34:46.239918 16559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:34:46.241621 16559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:34:46.241740 16559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:46.241858 16559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:34:46.241930 16559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:46.276034 16559 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36783
I20260926 23:34:46.276120 16706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36783 every 8 connection(s)
I20260926 23:34:46.277758 16559 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:34:46.281207 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 16559
I20260926 23:34:46.282097 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:34735
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46435
--webserver_port=33279
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:34:46 dist-test-slave-gwfx krb5kdc[15935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:34:46.306618 16709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:46.281074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46993 (local address 127.0.97.62:46435)
0926 23:34:46.281707 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:46.281721 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:46.282769 (+ 1048us) server_negotiation.cc:407] Connection header received
0926 23:34:46.285183 (+ 2414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:46.285196 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:46.285593 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:46.285943 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:34:46.287830 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:46.289224 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:46.291455 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:46.292501 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:46.292727 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:46.297760 (+ 5033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:34:46.297771 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:34:46.297792 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:34:46.297844 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:34:46.301215 (+ 3371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:46.302123 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:46.302132 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:46.302137 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:46.302332 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:46.302838 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:46.302843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:46.302845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:46.303448 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:34:46.303657 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:46.304573 (+ 916us) server_negotiation.cc:299] Negotiation successful
0926 23:34:46.304997 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":262,"threads_started":1}
I20260926 23:34:46.306612 16710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:46.281626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.1:46993)
0926 23:34:46.282554 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:46.282605 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 23:34:46.284579 (+ 1974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:34:46.286125 (+ 1546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:34:46.286136 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:34:46.286920 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:34:46.287591 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:46.287612 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:46.289414 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:46.289419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:46.291195 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:46.291215 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:46.292809 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:46.292818 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:46.293161 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:46.294096 (+ 935us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:34:46.294137 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:34:46.297370 (+ 3233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:34:46.301418 (+ 4048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:34:46.301428 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:34:46.301445 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:34:46.301956 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:34:46.302458 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:34:46.302464 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:34:46.302468 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:34:46.302708 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:34:46.303602 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:34:46.303613 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:34:46.303995 (+ 382us) client_negotiation.cc:769] Sending connection context
0926 23:34:46.304288 (+ 293us) client_negotiation.cc:240] Negotiation successful
0926 23:34:46.304997 (+ 709us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":349,"threads_started":1}
I20260926 23:34:46.310212 16707 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46435
I20260926 23:34:46.310711 16707 heartbeater.cc:464] Registering TS with master...
I20260926 23:34:46.311832 16707 heartbeater.cc:511] Master 127.0.97.62:46435 requested a full tablet report, sending...
I20260926 23:34:46.314872 16522 ts_manager.cc:194] Registered new tserver with Master: ae8abc5b72a04cb59f4a9dc64d532cac (127.0.97.1:36783)
I20260926 23:34:46.316622 16522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46993
W20260926 23:34:46.687206 16711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:34:46.687613 16711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:34:46.687738 16711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:34:46.696126 16711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:34:46.696274 16711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:34:46.696444 16711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:34:46.709161 16711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:34735
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33279
--tserver_master_addrs=127.0.97.62:46435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:34:46.711225 16711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:34:46.713220 16711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:34:46.731994 16716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:34:46.732097 16717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:34:46.732017 16719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:34:46.733038 16711 server_base.cc:1034] running on GCE node
I20260926 23:34:46.733886 16711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:34:46.735214 16711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:34:46.736454 16711 hybrid_clock.cc:648] HybridClock initialized: now 1790465686736386 us; error 62 us; skew 500 ppm
I20260926 23:34:46.736858 16711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:34:46 dist-test-slave-gwfx krb5kdc[15935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:34:46.740605 16711 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:34:46.742517 16711 webserver.cc:460] Webserver started at http://127.0.97.2:33279/ using document root <none> and password file <none>
I20260926 23:34:46.743495 16711 fs_manager.cc:359] Metadata directory not provided
I20260926 23:34:46.743637 16711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:34:46.750443 16711 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 23:34:46.754136 16726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:46.756068 16711 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:34:46.756270 16711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "e9e7c7787b5040b9910b34091f82f6ce"
format_stamp: "Formatted at 2026-09-26 23:34:23 on dist-test-slave-gwfx"
server_key: "d04606eac364773339cde975cf39aa39"
server_key_iv: "19d1f082e1b5692deb59c99b58564859"
server_key_version: "encryptionkey@0"
I20260926 23:34:46.757454 16711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:34:46.803040 16711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:34:46.808471 16711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:34:46.808755 16711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:34:46.809978 16711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:34:46.811866 16711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:34:46.811997 16711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:46.812106 16711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:34:46.812166 16711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:46.849334 16711 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34735
I20260926 23:34:46.849395 16838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34735 every 8 connection(s)
I20260926 23:34:46.851244 16711 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:34:46.855768 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 16711
I20260926 23:34:46.856472 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35683
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46435
--webserver_port=43345
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 23:34:46 dist-test-slave-gwfx krb5kdc[15935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:34:46.877556 16841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:46.854544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48493 (local address 127.0.97.62:46435)
0926 23:34:46.855139 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:46.855144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:46.857050 (+ 1906us) server_negotiation.cc:407] Connection header received
0926 23:34:46.858929 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:46.858936 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:46.859058 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:46.859213 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:34:46.861389 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:46.862374 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:46.864288 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:46.864811 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:46.864963 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:46.870078 (+ 5115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:34:46.870084 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:34:46.870088 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:34:46.870134 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:34:46.873134 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:46.874288 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:46.874294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:46.874296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:46.874457 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:46.875118 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:46.875125 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:46.875128 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:46.875606 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:34:46.875749 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:46.876873 (+ 1124us) server_negotiation.cc:299] Negotiation successful
0926 23:34:46.877151 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":282,"threads_started":1}
I20260926 23:34:46.879283 16842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:46.855358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.2:48493)
0926 23:34:46.856393 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:46.856451 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 23:34:46.858529 (+ 2078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:34:46.859546 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:34:46.859557 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:34:46.860322 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:34:46.861171 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:46.861199 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:46.862578 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:46.862582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:46.864116 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:46.864134 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:46.865105 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:46.865111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:46.865374 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:46.866350 (+ 976us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:34:46.866386 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:34:46.869675 (+ 3289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:34:46.873345 (+ 3670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:34:46.873357 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:34:46.873384 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:34:46.874121 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:34:46.874564 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:34:46.874568 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:34:46.874572 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:34:46.874924 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:34:46.875758 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:34:46.875768 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:34:46.876335 (+ 567us) client_negotiation.cc:769] Sending connection context
0926 23:34:46.876708 (+ 373us) client_negotiation.cc:240] Negotiation successful
0926 23:34:46.877414 (+ 706us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":400,"threads_started":1}
I20260926 23:34:46.881307 16839 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46435
I20260926 23:34:46.881688 16839 heartbeater.cc:464] Registering TS with master...
I20260926 23:34:46.882717 16839 heartbeater.cc:511] Master 127.0.97.62:46435 requested a full tablet report, sending...
I20260926 23:34:46.884688 16522 ts_manager.cc:194] Registered new tserver with Master: e9e7c7787b5040b9910b34091f82f6ce (127.0.97.2:34735)
I20260926 23:34:46.886005 16522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48493
W20260926 23:34:47.269102 16843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:34:47.269479 16843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:34:47.269588 16843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:34:47.279220 16843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:34:47.279409 16843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:34:47.279608 16843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:34:47.292405 16843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43345
--tserver_master_addrs=127.0.97.62:46435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:34:47.294116 16843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:34:47.295789 16843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:34:47.312011 16848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:34:47.313011 16849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:34:47.314579 16851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:34:47.315109 16843 server_base.cc:1034] running on GCE node
I20260926 23:34:47.315903 16843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:34:47.317178 16843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:34:47.318456 16843 hybrid_clock.cc:648] HybridClock initialized: now 1790465687318391 us; error 54 us; skew 500 ppm
I20260926 23:34:47.318989 16843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:34:47.321805 16707 heartbeater.cc:503] Master 127.0.97.62:46435 was elected leader, sending a full tablet report...
Sep 26 23:34:47 dist-test-slave-gwfx krb5kdc[15935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:34:47.323251 16843 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:34:47.325405 16843 webserver.cc:460] Webserver started at http://127.0.97.3:43345/ using document root <none> and password file <none>
I20260926 23:34:47.326334 16843 fs_manager.cc:359] Metadata directory not provided
I20260926 23:34:47.326478 16843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:34:47.333560 16843 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:34:47.337414 16858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:47.339457 16843 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:34:47.339669 16843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "1eb86cb9e5b64ad689580147ff05dcdb"
format_stamp: "Formatted at 2026-09-26 23:34:23 on dist-test-slave-gwfx"
server_key: "87c326a8254e6ee107818e51a93f4695"
server_key_iv: "dad5ceed5c86b397f758d128b01a1795"
server_key_version: "encryptionkey@0"
I20260926 23:34:47.340888 16843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:34:47.373633 16843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:34:47.379262 16843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:34:47.379578 16843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:34:47.380971 16843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:34:47.382881 16843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:34:47.382999 16843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:47.383098 16843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:34:47.383160 16843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:34:47.420305 16843 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35683
I20260926 23:34:47.420370 16970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35683 every 8 connection(s)
I20260926 23:34:47.422227 16843 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:34:47.431648 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 16843
Sep 26 23:34:47 dist-test-slave-gwfx krb5kdc[15935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:34:47.450189 16976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:47.432833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43530 (local address 127.0.97.62:46435)
0926 23:34:47.433766 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:47.433773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:47.434259 (+ 486us) server_negotiation.cc:407] Connection header received
0926 23:34:47.434627 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:47.434634 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:47.434774 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:47.434885 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:34:47.436847 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.437858 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.439661 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.440156 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.440284 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:47.443558 (+ 3274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:34:47.443563 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:34:47.443566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:34:47.443618 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:34:47.447175 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:47.447982 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:47.447988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:47.447991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:47.448146 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:47.448533 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:47.448537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:47.448539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:47.448866 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:34:47.449162 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:47.449462 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 23:34:47.449734 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":437,"threads_started":1}
I20260926 23:34:47.450677 16973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:47.425982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48373 (local address 127.0.97.62:46435)
0926 23:34:47.426517 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:47.426525 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:47.428147 (+ 1622us) server_negotiation.cc:407] Connection header received
0926 23:34:47.430557 (+ 2410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:47.430563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:47.430707 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:47.430837 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:34:47.433592 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.434593 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.436906 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.437486 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.437631 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:47.442907 (+ 5276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:34:47.442913 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:34:47.442916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:34:47.442961 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:34:47.446757 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:47.447956 (+ 1199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:47.447961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:47.447964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:47.448122 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:47.448658 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:47.448661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:47.448664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:47.448962 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:34:47.449095 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:47.449986 (+ 891us) server_negotiation.cc:299] Negotiation successful
0926 23:34:47.450235 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":239,"threads_started":1}
I20260926 23:34:47.452145 16974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:47.426913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.3:48373)
0926 23:34:47.427884 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:47.427936 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 23:34:47.430112 (+ 2176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:34:47.431204 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:34:47.431216 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:34:47.432075 (+ 859us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:34:47.433252 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:47.433298 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.434847 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.434852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:47.436705 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:47.436725 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.437834 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.437844 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:47.438157 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:47.439185 (+ 1028us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:34:47.439217 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:34:47.442579 (+ 3362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:34:47.446982 (+ 4403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:34:47.446998 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:34:47.447021 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:34:47.447749 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:34:47.448250 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:34:47.448254 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:34:47.448257 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:34:47.448529 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:34:47.449100 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:34:47.449110 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:34:47.449573 (+ 463us) client_negotiation.cc:769] Sending connection context
0926 23:34:47.449863 (+ 290us) client_negotiation.cc:240] Negotiation successful
0926 23:34:47.450580 (+ 717us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":365,"threads_started":1}
I20260926 23:34:47.453853 16971 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46435
I20260926 23:34:47.454223 16971 heartbeater.cc:464] Registering TS with master...
I20260926 23:34:47.455163 16971 heartbeater.cc:511] Master 127.0.97.62:46435 requested a full tablet report, sending...
I20260926 23:34:47.457026 16522 ts_manager.cc:194] Registered new tserver with Master: 1eb86cb9e5b64ad689580147ff05dcdb (127.0.97.3:35683)
I20260926 23:34:47.458036 16522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48373
I20260926 23:34:47.467015 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:34:47.475628 16973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:47.468277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43534 (local address 127.0.97.62:46435)
0926 23:34:47.468496 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:47.468501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:47.469055 (+ 554us) server_negotiation.cc:407] Connection header received
0926 23:34:47.469424 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:47.469430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:47.469671 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:47.469858 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:34:47.470437 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.471433 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.473255 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.473773 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.473889 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:47.474516 (+ 627us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:34:47.474933 (+ 417us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 23:34:47.475326 (+ 393us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43534: token expired
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 23:34:47.475776 16973 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43534: token expired
W20260926 23:34:47.476543 16443 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:46435)
I20260926 23:34:47.492813 16973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:47.477177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43544 (local address 127.0.97.62:46435)
0926 23:34:47.477378 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:47.477384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:47.477404 (+ 20us) server_negotiation.cc:407] Connection header received
0926 23:34:47.477465 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:47.477469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:47.477538 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:47.477625 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:34:47.479301 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.480147 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.481954 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.482427 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.482548 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:47.485814 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:34:47.485820 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:34:47.485824 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:34:47.485876 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:34:47.489137 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:47.490315 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:47.490321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:47.490324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:47.490546 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:47.491017 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:47.491021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:47.491024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:47.491473 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:34:47.491809 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:47.492145 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 23:34:47.492437 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 23:34:47.497609 16522 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:34:47.506536 16522 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:34:47.545585 16987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:47.522895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40018 (local address 127.0.97.2:34735)
0926 23:34:47.535229 (+ 12334us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:47.535241 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:47.536471 (+ 1230us) server_negotiation.cc:407] Connection header received
0926 23:34:47.536671 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:47.536679 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:47.536907 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:47.537128 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:34:47.538109 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.539725 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.542083 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.543704 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.543852 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:47.544121 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:47.544947 (+ 826us) server_negotiation.cc:299] Negotiation successful
0926 23:34:47.545206 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12127,"thread_start_us":11988,"threads_started":1}
I20260926 23:34:47.545661 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:47.522825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34735 (local address 127.0.0.1:40018)
0926 23:34:47.536334 (+ 13509us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:47.536356 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:34:47.536486 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:34:47.537130 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:34:47.537135 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:34:47.537165 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:34:47.537894 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:47.537906 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.539855 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.539859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:47.541887 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:47.541905 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.544295 (+ 2390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.544302 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:47.544568 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:47.544702 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 23:34:47.544875 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 23:34:47.545138 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13390,"mutex_wait_us":158,"thread_start_us":8901,"threads_started":1}
I20260926 23:34:47.546131 16984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:47.521441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59582 (local address 127.0.97.1:36783)
0926 23:34:47.535791 (+ 14350us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:47.535799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:47.535833 (+ 34us) server_negotiation.cc:407] Connection header received
0926 23:34:47.535951 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:47.535961 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:47.536308 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:47.536537 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:34:47.537607 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.539433 (+ 1826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.542742 (+ 3309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.544260 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.544405 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:47.544639 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:47.545057 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 23:34:47.545252 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14144,"thread_start_us":14192,"threads_started":1}
I20260926 23:34:47.546723 16986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:47.521086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36783 (local address 127.0.0.1:59582)
0926 23:34:47.532294 (+ 11208us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:47.532367 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0926 23:34:47.532639 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:34:47.536836 (+ 4197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:34:47.536844 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:34:47.536870 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:34:47.537314 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:47.537332 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.539646 (+ 2314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.539651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:47.542495 (+ 2844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:47.542514 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.544574 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.544581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:47.544783 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:47.544819 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 23:34:47.544916 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 23:34:47.546327 (+ 1411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10823,"mutex_wait_us":1375,"thread_start_us":14975,"threads_started":1}
I20260926 23:34:47.547524 16983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:47.521319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35683 (local address 127.0.0.1:49776)
0926 23:34:47.534935 (+ 13616us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:47.534958 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:34:47.535112 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:34:47.536615 (+ 1503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:34:47.536623 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:34:47.536655 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:34:47.537312 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:47.537332 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.539403 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.539409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:47.542878 (+ 3469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:47.542892 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.545622 (+ 2730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.545628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:47.545946 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:47.545977 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 23:34:47.546064 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 23:34:47.547159 (+ 1095us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13458,"mutex_wait_us":279,"thread_start_us":12872,"threads_started":1}
I20260926 23:34:47.547667 16985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:47.521930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49776 (local address 127.0.97.3:35683)
0926 23:34:47.535783 (+ 13853us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:47.535791 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:47.535833 (+ 42us) server_negotiation.cc:407] Connection header received
0926 23:34:47.535951 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:47.535961 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:47.536308 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:47.536537 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:34:47.537633 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.539186 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.543445 (+ 4259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.544987 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.545164 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:47.545393 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:47.546298 (+ 905us) server_negotiation.cc:299] Negotiation successful
0926 23:34:47.546587 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13652,"thread_start_us":13469,"threads_started":1}
I20260926 23:34:47.550648 16642 tablet_service.cc:1467] Processing CreateTablet for tablet 01ef7c022b104325b140596a660a00fb (DEFAULT_TABLE table=test-table [id=e32103397b624d50bd1028fa3e226e84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:34:47.550665 16774 tablet_service.cc:1467] Processing CreateTablet for tablet 01ef7c022b104325b140596a660a00fb (DEFAULT_TABLE table=test-table [id=e32103397b624d50bd1028fa3e226e84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:34:47.551853 16906 tablet_service.cc:1467] Processing CreateTablet for tablet 01ef7c022b104325b140596a660a00fb (DEFAULT_TABLE table=test-table [id=e32103397b624d50bd1028fa3e226e84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:34:47.552537 16642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ef7c022b104325b140596a660a00fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:34:47.552537 16774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ef7c022b104325b140596a660a00fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:34:47.553570 16906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ef7c022b104325b140596a660a00fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:34:47.569270 16988 tablet_bootstrap.cc:492] T 01ef7c022b104325b140596a660a00fb P ae8abc5b72a04cb59f4a9dc64d532cac: Bootstrap starting.
I20260926 23:34:47.573549 16988 tablet_bootstrap.cc:654] T 01ef7c022b104325b140596a660a00fb P ae8abc5b72a04cb59f4a9dc64d532cac: Neither blocks nor log segments found. Creating new log.
I20260926 23:34:47.574779 16989 tablet_bootstrap.cc:492] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce: Bootstrap starting.
I20260926 23:34:47.575212 16988 log.cc:824] T 01ef7c022b104325b140596a660a00fb P ae8abc5b72a04cb59f4a9dc64d532cac: Log is configured to *not* fsync() on all Append() calls
I20260926 23:34:47.576364 16990 tablet_bootstrap.cc:492] T 01ef7c022b104325b140596a660a00fb P 1eb86cb9e5b64ad689580147ff05dcdb: Bootstrap starting.
I20260926 23:34:47.581192 16990 tablet_bootstrap.cc:654] T 01ef7c022b104325b140596a660a00fb P 1eb86cb9e5b64ad689580147ff05dcdb: Neither blocks nor log segments found. Creating new log.
I20260926 23:34:47.581192 16989 tablet_bootstrap.cc:654] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce: Neither blocks nor log segments found. Creating new log.
I20260926 23:34:47.582022 16988 tablet_bootstrap.cc:492] T 01ef7c022b104325b140596a660a00fb P ae8abc5b72a04cb59f4a9dc64d532cac: No bootstrap required, opened a new log
I20260926 23:34:47.582378 16988 ts_tablet_manager.cc:1397] T 01ef7c022b104325b140596a660a00fb P ae8abc5b72a04cb59f4a9dc64d532cac: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260926 23:34:47.582885 16989 log.cc:824] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce: Log is configured to *not* fsync() on all Append() calls
I20260926 23:34:47.582885 16990 log.cc:824] T 01ef7c022b104325b140596a660a00fb P 1eb86cb9e5b64ad689580147ff05dcdb: Log is configured to *not* fsync() on all Append() calls
I20260926 23:34:47.589951 16989 tablet_bootstrap.cc:492] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce: No bootstrap required, opened a new log
I20260926 23:34:47.589951 16990 tablet_bootstrap.cc:492] T 01ef7c022b104325b140596a660a00fb P 1eb86cb9e5b64ad689580147ff05dcdb: No bootstrap required, opened a new log
I20260926 23:34:47.590402 16989 ts_tablet_manager.cc:1397] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.000s
I20260926 23:34:47.590440 16990 ts_tablet_manager.cc:1397] T 01ef7c022b104325b140596a660a00fb P 1eb86cb9e5b64ad689580147ff05dcdb: Time spent bootstrapping tablet: real 0.014s user 0.013s sys 0.000s
I20260926 23:34:47.592087 16988 raft_consensus.cc:348] T 01ef7c022b104325b140596a660a00fb P ae8abc5b72a04cb59f4a9dc64d532cac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8abc5b72a04cb59f4a9dc64d532cac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36783 } } peers { permanent_uuid: "1eb86cb9e5b64ad689580147ff05dcdb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35683 } } peers { permanent_uuid: "e9e7c7787b5040b9910b34091f82f6ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34735 } }
I20260926 23:34:47.592576 16988 raft_consensus.cc:374] T 01ef7c022b104325b140596a660a00fb P ae8abc5b72a04cb59f4a9dc64d532cac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:34:47.592689 16988 raft_consensus.cc:729] T 01ef7c022b104325b140596a660a00fb P ae8abc5b72a04cb59f4a9dc64d532cac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8abc5b72a04cb59f4a9dc64d532cac, State: Initialized, Role: FOLLOWER
I20260926 23:34:47.593425 16988 consensus_queue.cc:260] T 01ef7c022b104325b140596a660a00fb P ae8abc5b72a04cb59f4a9dc64d532cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8abc5b72a04cb59f4a9dc64d532cac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36783 } } peers { permanent_uuid: "1eb86cb9e5b64ad689580147ff05dcdb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35683 } } peers { permanent_uuid: "e9e7c7787b5040b9910b34091f82f6ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34735 } }
I20260926 23:34:47.595672 16988 ts_tablet_manager.cc:1428] T 01ef7c022b104325b140596a660a00fb P ae8abc5b72a04cb59f4a9dc64d532cac: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260926 23:34:47.602805 16990 raft_consensus.cc:348] T 01ef7c022b104325b140596a660a00fb P 1eb86cb9e5b64ad689580147ff05dcdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8abc5b72a04cb59f4a9dc64d532cac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36783 } } peers { permanent_uuid: "1eb86cb9e5b64ad689580147ff05dcdb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35683 } } peers { permanent_uuid: "e9e7c7787b5040b9910b34091f82f6ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34735 } }
I20260926 23:34:47.602803 16989 raft_consensus.cc:348] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8abc5b72a04cb59f4a9dc64d532cac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36783 } } peers { permanent_uuid: "1eb86cb9e5b64ad689580147ff05dcdb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35683 } } peers { permanent_uuid: "e9e7c7787b5040b9910b34091f82f6ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34735 } }
I20260926 23:34:47.603452 16990 raft_consensus.cc:374] T 01ef7c022b104325b140596a660a00fb P 1eb86cb9e5b64ad689580147ff05dcdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:34:47.603454 16989 raft_consensus.cc:374] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:34:47.603572 16990 raft_consensus.cc:729] T 01ef7c022b104325b140596a660a00fb P 1eb86cb9e5b64ad689580147ff05dcdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb86cb9e5b64ad689580147ff05dcdb, State: Initialized, Role: FOLLOWER
I20260926 23:34:47.603581 16989 raft_consensus.cc:729] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9e7c7787b5040b9910b34091f82f6ce, State: Initialized, Role: FOLLOWER
I20260926 23:34:47.604401 16990 consensus_queue.cc:260] T 01ef7c022b104325b140596a660a00fb P 1eb86cb9e5b64ad689580147ff05dcdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8abc5b72a04cb59f4a9dc64d532cac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36783 } } peers { permanent_uuid: "1eb86cb9e5b64ad689580147ff05dcdb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35683 } } peers { permanent_uuid: "e9e7c7787b5040b9910b34091f82f6ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34735 } }
I20260926 23:34:47.604405 16989 consensus_queue.cc:260] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8abc5b72a04cb59f4a9dc64d532cac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36783 } } peers { permanent_uuid: "1eb86cb9e5b64ad689580147ff05dcdb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35683 } } peers { permanent_uuid: "e9e7c7787b5040b9910b34091f82f6ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34735 } }
I20260926 23:34:47.605839 16971 heartbeater.cc:503] Master 127.0.97.62:46435 was elected leader, sending a full tablet report...
I20260926 23:34:47.606359 16839 heartbeater.cc:503] Master 127.0.97.62:46435 was elected leader, sending a full tablet report...
I20260926 23:34:47.606933 16990 ts_tablet_manager.cc:1428] T 01ef7c022b104325b140596a660a00fb P 1eb86cb9e5b64ad689580147ff05dcdb: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20260926 23:34:47.608237 16989 ts_tablet_manager.cc:1428] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
W20260926 23:34:47.674472 16972 tablet.cc:2369] T 01ef7c022b104325b140596a660a00fb P 1eb86cb9e5b64ad689580147ff05dcdb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:34:47.780732 16708 tablet.cc:2369] T 01ef7c022b104325b140596a660a00fb P ae8abc5b72a04cb59f4a9dc64d532cac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:34:47.814257 16996 raft_consensus.cc:482] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:34:47.814656 16996 raft_consensus.cc:504] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8abc5b72a04cb59f4a9dc64d532cac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36783 } } peers { permanent_uuid: "1eb86cb9e5b64ad689580147ff05dcdb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35683 } } peers { permanent_uuid: "e9e7c7787b5040b9910b34091f82f6ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34735 } }
I20260926 23:34:47.817009 16996 leader_election.cc:288] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae8abc5b72a04cb59f4a9dc64d532cac (127.0.97.1:36783), 1eb86cb9e5b64ad689580147ff05dcdb (127.0.97.3:35683)
I20260926 23:34:47.824585 16984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:47.817870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35691 (local address 127.0.97.1:36783)
0926 23:34:47.818118 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:47.818124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:47.818635 (+ 511us) server_negotiation.cc:407] Connection header received
0926 23:34:47.818949 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:47.818956 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:47.819030 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:47.819112 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:34:47.819722 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.820549 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.822588 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.823553 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.823697 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:47.823826 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:47.824139 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 23:34:47.824290 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 23:34:47.824694 16997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:47.817662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36783 (local address 127.0.97.2:35691)
0926 23:34:47.818502 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:47.818520 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 23:34:47.818760 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:34:47.819163 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:34:47.819168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:34:47.819198 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:34:47.819577 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:47.819588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.820719 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.820724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:47.822393 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:47.822412 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.823809 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.823816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:47.823968 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:47.824000 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 23:34:47.824093 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 23:34:47.824264 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":481,"threads_started":1}
I20260926 23:34:47.824826 16985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:47.817643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55105 (local address 127.0.97.3:35683)
0926 23:34:47.817872 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:47.817877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:47.819217 (+ 1340us) server_negotiation.cc:407] Connection header received
0926 23:34:47.819400 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:47.819404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:47.819480 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:47.819572 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:34:47.820168 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.821037 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.822898 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.823828 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.823966 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:47.824087 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:47.824395 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 23:34:47.824548 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 23:34:47.824976 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:47.817662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35683 (local address 127.0.97.2:55105)
0926 23:34:47.819058 (+ 1396us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:47.819074 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:34:47.819242 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:34:47.819604 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:34:47.819608 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:34:47.819636 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:34:47.819996 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:47.820006 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.821194 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.821198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:47.822723 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:34:47.822739 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.824066 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.824073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:34:47.824234 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:47.824258 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 23:34:47.824337 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 23:34:47.824489 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1265,"mutex_wait_us":349,"thread_start_us":619,"threads_started":1}
I20260926 23:34:47.826239 16926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ef7c022b104325b140596a660a00fb" candidate_uuid: "e9e7c7787b5040b9910b34091f82f6ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb86cb9e5b64ad689580147ff05dcdb" is_pre_election: true
I20260926 23:34:47.826236 16662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ef7c022b104325b140596a660a00fb" candidate_uuid: "e9e7c7787b5040b9910b34091f82f6ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8abc5b72a04cb59f4a9dc64d532cac" is_pre_election: true
I20260926 23:34:47.826887 16926 raft_consensus.cc:2454] T 01ef7c022b104325b140596a660a00fb P 1eb86cb9e5b64ad689580147ff05dcdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9e7c7787b5040b9910b34091f82f6ce in term 0.
I20260926 23:34:47.826891 16662 raft_consensus.cc:2454] T 01ef7c022b104325b140596a660a00fb P ae8abc5b72a04cb59f4a9dc64d532cac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9e7c7787b5040b9910b34091f82f6ce in term 0.
I20260926 23:34:47.827683 16727 leader_election.cc:302] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1eb86cb9e5b64ad689580147ff05dcdb, ae8abc5b72a04cb59f4a9dc64d532cac, e9e7c7787b5040b9910b34091f82f6ce; no voters:
I20260926 23:34:47.828176 16996 raft_consensus.cc:2781] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:34:47.828327 16996 raft_consensus.cc:482] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:34:47.828523 16996 raft_consensus.cc:3037] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:34:47.831280 16996 raft_consensus.cc:504] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8abc5b72a04cb59f4a9dc64d532cac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36783 } } peers { permanent_uuid: "1eb86cb9e5b64ad689580147ff05dcdb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35683 } } peers { permanent_uuid: "e9e7c7787b5040b9910b34091f82f6ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34735 } }
I20260926 23:34:47.832082 16996 leader_election.cc:288] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce [CANDIDATE]: Term 1 election: Requested vote from peers ae8abc5b72a04cb59f4a9dc64d532cac (127.0.97.1:36783), 1eb86cb9e5b64ad689580147ff05dcdb (127.0.97.3:35683)
I20260926 23:34:47.832789 16662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ef7c022b104325b140596a660a00fb" candidate_uuid: "e9e7c7787b5040b9910b34091f82f6ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8abc5b72a04cb59f4a9dc64d532cac"
I20260926 23:34:47.832789 16926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ef7c022b104325b140596a660a00fb" candidate_uuid: "e9e7c7787b5040b9910b34091f82f6ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb86cb9e5b64ad689580147ff05dcdb"
I20260926 23:34:47.833074 16926 raft_consensus.cc:3037] T 01ef7c022b104325b140596a660a00fb P 1eb86cb9e5b64ad689580147ff05dcdb [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:34:47.833074 16662 raft_consensus.cc:3037] T 01ef7c022b104325b140596a660a00fb P ae8abc5b72a04cb59f4a9dc64d532cac [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:34:47.836853 16662 raft_consensus.cc:2454] T 01ef7c022b104325b140596a660a00fb P ae8abc5b72a04cb59f4a9dc64d532cac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9e7c7787b5040b9910b34091f82f6ce in term 1.
I20260926 23:34:47.836896 16926 raft_consensus.cc:2454] T 01ef7c022b104325b140596a660a00fb P 1eb86cb9e5b64ad689580147ff05dcdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9e7c7787b5040b9910b34091f82f6ce in term 1.
I20260926 23:34:47.837420 16727 leader_election.cc:302] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae8abc5b72a04cb59f4a9dc64d532cac, e9e7c7787b5040b9910b34091f82f6ce; no voters:
I20260926 23:34:47.837852 16996 raft_consensus.cc:2781] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:34:47.838397 16996 raft_consensus.cc:686] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce [term 1 LEADER]: Becoming Leader. State: Replica: e9e7c7787b5040b9910b34091f82f6ce, State: Running, Role: LEADER
I20260926 23:34:47.839145 16996 consensus_queue.cc:237] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8abc5b72a04cb59f4a9dc64d532cac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36783 } } peers { permanent_uuid: "1eb86cb9e5b64ad689580147ff05dcdb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35683 } } peers { permanent_uuid: "e9e7c7787b5040b9910b34091f82f6ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34735 } }
I20260926 23:34:47.844736 16522 catalog_manager.cc:5495] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce reported cstate change: term changed from 0 to 1, leader changed from <none> to e9e7c7787b5040b9910b34091f82f6ce (127.0.97.2). New cstate: current_term: 1 leader_uuid: "e9e7c7787b5040b9910b34091f82f6ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8abc5b72a04cb59f4a9dc64d532cac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eb86cb9e5b64ad689580147ff05dcdb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9e7c7787b5040b9910b34091f82f6ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34735 } health_report { overall_health: HEALTHY } } }
W20260926 23:34:47.853761 16840 tablet.cc:2369] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:34:47.921999 16987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:47.916046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40032 (local address 127.0.97.2:34735)
0926 23:34:47.916239 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:47.916244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:47.916269 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:34:47.916503 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:47.916508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:47.916605 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:47.916754 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:34:47.917164 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.917951 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.919521 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.919988 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.920102 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:47.920674 (+ 572us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:34:47.921378 (+ 704us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:34:47.921505 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:47.921619 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 23:34:47.921711 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 23:34:47.931892 16662 raft_consensus.cc:1261] T 01ef7c022b104325b140596a660a00fb P ae8abc5b72a04cb59f4a9dc64d532cac [term 1 FOLLOWER]: Refusing update from remote peer e9e7c7787b5040b9910b34091f82f6ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:34:47.931892 16926 raft_consensus.cc:1261] T 01ef7c022b104325b140596a660a00fb P 1eb86cb9e5b64ad689580147ff05dcdb [term 1 FOLLOWER]: Refusing update from remote peer e9e7c7787b5040b9910b34091f82f6ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:34:47.933032 16999 consensus_queue.cc:1035] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb86cb9e5b64ad689580147ff05dcdb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:34:47.933521 16996 consensus_queue.cc:1035] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8abc5b72a04cb59f4a9dc64d532cac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:34:47.971112 16521 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:34:47.971562 16521 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:34:47.976332 16521 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e: Sending DeleteTablet for 3 replicas of tablet 01ef7c022b104325b140596a660a00fb
I20260926 23:34:47.977895 16906 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ef7c022b104325b140596a660a00fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:34:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49776
I20260926 23:34:47.977887 16642 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ef7c022b104325b140596a660a00fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:34:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59582
I20260926 23:34:47.978247 16774 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ef7c022b104325b140596a660a00fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:34:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40018
I20260926 23:34:47.979786 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 16559
I20260926 23:34:47.991328 17014 tablet_replica.cc:330] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce: stopping tablet replica
I20260926 23:34:47.992062 17014 raft_consensus.cc:2229] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:34:47.994647 17013 tablet_replica.cc:330] T 01ef7c022b104325b140596a660a00fb P ae8abc5b72a04cb59f4a9dc64d532cac: stopping tablet replica
I20260926 23:34:47.994951 17012 tablet_replica.cc:330] T 01ef7c022b104325b140596a660a00fb P 1eb86cb9e5b64ad689580147ff05dcdb: stopping tablet replica
I20260926 23:34:47.995276 17013 raft_consensus.cc:2229] T 01ef7c022b104325b140596a660a00fb P ae8abc5b72a04cb59f4a9dc64d532cac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:34:47.995368 17012 raft_consensus.cc:2229] T 01ef7c022b104325b140596a660a00fb P 1eb86cb9e5b64ad689580147ff05dcdb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:34:47.995935 17012 raft_consensus.cc:2258] T 01ef7c022b104325b140596a660a00fb P 1eb86cb9e5b64ad689580147ff05dcdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:34:47.995934 17013 raft_consensus.cc:2258] T 01ef7c022b104325b140596a660a00fb P ae8abc5b72a04cb59f4a9dc64d532cac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:34:47.992965 17014 raft_consensus.cc:2258] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:34:47.999250 17014 ts_tablet_manager.cc:1905] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:34:47.999287 17012 ts_tablet_manager.cc:1905] T 01ef7c022b104325b140596a660a00fb P 1eb86cb9e5b64ad689580147ff05dcdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:34:48.006676 17013 ts_tablet_manager.cc:1905] T 01ef7c022b104325b140596a660a00fb P ae8abc5b72a04cb59f4a9dc64d532cac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:34:48.007808 17012 ts_tablet_manager.cc:1918] T 01ef7c022b104325b140596a660a00fb P 1eb86cb9e5b64ad689580147ff05dcdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:34:48.008035 17012 log.cc:1196] T 01ef7c022b104325b140596a660a00fb P 1eb86cb9e5b64ad689580147ff05dcdb: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/wal/wals/01ef7c022b104325b140596a660a00fb
I20260926 23:34:48.008631 17012 ts_tablet_manager.cc:1939] T 01ef7c022b104325b140596a660a00fb P 1eb86cb9e5b64ad689580147ff05dcdb: Deleting consensus metadata
I20260926 23:34:48.011180 16509 catalog_manager.cc:4841] TS 1eb86cb9e5b64ad689580147ff05dcdb (127.0.97.3:35683): tablet 01ef7c022b104325b140596a660a00fb (table test-table [id=e32103397b624d50bd1028fa3e226e84]) successfully deleted
I20260926 23:34:48.014828 16984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:47.980924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59584 (local address 127.0.97.1:36783)
0926 23:34:47.981189 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:47.981196 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:47.990958 (+ 9762us) server_negotiation.cc:407] Connection header received
0926 23:34:47.991389 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:47.991398 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:47.991531 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:47.991635 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:34:47.996426 (+ 4791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:47.997602 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:47.999929 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:48.000655 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:48.000768 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:48.005148 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:34:48.005165 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:34:48.005188 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:34:48.005248 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:34:48.009858 (+ 4610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:48.011082 (+ 1224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:48.011095 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:48.011099 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:48.011317 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:48.011891 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:48.011898 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:48.011901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:48.012507 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:34:48.013026 (+ 519us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:48.013975 (+ 949us) server_negotiation.cc:299] Negotiation successful
0926 23:34:48.014305 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 23:34:48.016104 17013 ts_tablet_manager.cc:1918] T 01ef7c022b104325b140596a660a00fb P ae8abc5b72a04cb59f4a9dc64d532cac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:34:48.016335 17013 log.cc:1196] T 01ef7c022b104325b140596a660a00fb P ae8abc5b72a04cb59f4a9dc64d532cac: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/wal/wals/01ef7c022b104325b140596a660a00fb
I20260926 23:34:48.016803 16702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:34:48.016997 17014 ts_tablet_manager.cc:1918] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:34:48.017243 17014 log.cc:1196] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/wal/wals/01ef7c022b104325b140596a660a00fb
I20260926 23:34:48.018025 17014 ts_tablet_manager.cc:1939] T 01ef7c022b104325b140596a660a00fb P e9e7c7787b5040b9910b34091f82f6ce: Deleting consensus metadata
I20260926 23:34:48.020625 16507 catalog_manager.cc:4841] TS e9e7c7787b5040b9910b34091f82f6ce (127.0.97.2:34735): tablet 01ef7c022b104325b140596a660a00fb (table test-table [id=e32103397b624d50bd1028fa3e226e84]) successfully deleted
I20260926 23:34:48.147163 17013 ts_tablet_manager.cc:1939] T 01ef7c022b104325b140596a660a00fb P ae8abc5b72a04cb59f4a9dc64d532cac: Deleting consensus metadata
I20260926 23:34:48.147466 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 16559
I20260926 23:34:48.165774 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 16711
W20260926 23:34:48.165752 16506 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:36783 (error 108)
I20260926 23:34:48.168555 17018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:48.167527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36783 (local address 127.0.0.1:59600)
0926 23:34:48.168121 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:48.168272 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":342,"threads_started":1}
W20260926 23:34:48.168887 16506 catalog_manager.cc:4584] TS ae8abc5b72a04cb59f4a9dc64d532cac (127.0.97.1:36783): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 01ef7c022b104325b140596a660a00fb: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36783: connect: Connection refused (error 111)
I20260926 23:34:48.183729 16987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:48.167060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40046 (local address 127.0.97.2:34735)
0926 23:34:48.167244 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:48.167249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:48.167268 (+ 19us) server_negotiation.cc:407] Connection header received
0926 23:34:48.167337 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:48.167341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:48.167423 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:48.167524 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:34:48.169522 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:48.170488 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:48.172131 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:48.172624 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:48.172765 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:48.176022 (+ 3257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:34:48.176051 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:34:48.176081 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:34:48.176172 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:34:48.180167 (+ 3995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:48.181140 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:48.181151 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:48.181154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:48.181354 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:48.181810 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:48.181814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:48.181817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:48.182298 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:34:48.182727 (+ 429us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:48.183060 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 23:34:48.183337 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 23:34:48.184569 16834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:34:48.222995 17018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:48.222267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36783 (local address 127.0.0.1:59606)
0926 23:34:48.222578 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:48.222755 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260926 23:34:48.224363 17018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:48.223895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36783 (local address 127.0.0.1:59622)
0926 23:34:48.224060 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:48.224174 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 23:34:48.285597 17018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:48.284824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36783 (local address 127.0.0.1:59634)
0926 23:34:48.285158 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:48.285290 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260926 23:34:48.287051 17018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:48.286518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36783 (local address 127.0.0.1:59636)
0926 23:34:48.286733 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:48.286846 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 23:34:48.317063 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 16711
I20260926 23:34:48.337061 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 16843
I20260926 23:34:48.353340 17020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:48.337994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49784 (local address 127.0.97.3:35683)
0926 23:34:48.338573 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0926 23:34:48.338579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:34:48.338597 (+ 18us) server_negotiation.cc:407] Connection header received
0926 23:34:48.338664 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:34:48.338668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:34:48.338802 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:34:48.338913 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:34:48.340520 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:48.341324 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:48.342648 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:34:48.343208 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:34:48.343335 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:34:48.346470 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:34:48.346491 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:34:48.346514 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:34:48.346570 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:34:48.349898 (+ 3328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:48.350869 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:48.350878 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:48.350881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:48.351058 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:34:48.351596 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:34:48.351600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:34:48.351602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:34:48.352076 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:34:48.352362 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0926 23:34:48.352708 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 23:34:48.352922 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":271,"threads_started":1}
I20260926 23:34:48.354131 16966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:34:48.368139 17018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:48.367424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36783 (local address 127.0.0.1:59652)
0926 23:34:48.367715 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:48.367860 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260926 23:34:48.369441 17018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:48.368941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36783 (local address 127.0.0.1:59656)
0926 23:34:48.369161 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:48.369257 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260926 23:34:48.452009 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 16843
I20260926 23:34:48.468981 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 16489
I20260926 23:34:48.470394 16552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:34:48.603076 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 16489
I20260926 23:34:48.603127 17018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:34:48.602355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36783 (local address 127.0.0.1:59662)
0926 23:34:48.602781 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0926 23:34:48.602911 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218}
I20260926 23:35:08.625973 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46435
--webserver_interface=127.0.97.62
--webserver_port=43993
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:35:09.041994 17023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:35:09.042454 17023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:35:09.042588 17023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:35:09.051101 17023 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 23:35:09.051262 17023 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 23:35:09.051321 17023 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 23:35:09.051400 17023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:35:09.051479 17023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:35:09.051582 17023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:35:09.051646 17023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:35:09.051725 17023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:35:09.063561 17023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=43993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:35:09.065382 17023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:35:09.067224 17023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:35:09.083563 17031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:09.083590 17028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:09.083686 17029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:35:09.086786 17023 server_base.cc:1034] running on GCE node
I20260926 23:35:09.088008 17023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:35:09.090175 17023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:35:09.091516 17023 hybrid_clock.cc:648] HybridClock initialized: now 1790465709091398 us; error 74 us; skew 500 ppm
I20260926 23:35:09.092017 17023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:35:09 dist-test-slave-gwfx krb5kdc[15935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:35:09.100908 17023 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:35:09.103310 17023 webserver.cc:460] Webserver started at http://127.0.97.62:43993/ using document root <none> and password file <none>
I20260926 23:35:09.104323 17023 fs_manager.cc:359] Metadata directory not provided
I20260926 23:35:09.104446 17023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:35:09.112284 17023 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 23:35:09.116669 17038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:09.119189 17023 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20260926 23:35:09.119508 17023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "ccac865f8548457ca50960ec81cd9c6e"
format_stamp: "Formatted at 2026-09-26 23:34:21 on dist-test-slave-gwfx"
server_key: "c3c8b301e151609dcf6eeba52363bb73"
server_key_iv: "108fcb6ebdd58e00bf72bbf50bb64563"
server_key_version: "encryptionkey@0"
I20260926 23:35:09.120954 17023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:35:09.136708 17023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:35:09.143615 17023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:35:09.143961 17023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:35:09.167781 17023 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46435
I20260926 23:35:09.167836 17089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46435 every 8 connection(s)
I20260926 23:35:09.170053 17023 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:35:09.178408 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 17023
I20260926 23:35:09.178973 17090 sys_catalog.cc:263] Verifying existing consensus state
I20260926 23:35:09.179517 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:36783
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46435
--webserver_port=40445
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:35:09.184158 17090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e: Bootstrap starting.
I20260926 23:35:09.208952 17090 log.cc:824] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e: Log is configured to *not* fsync() on all Append() calls
I20260926 23:35:09.228783 17090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 23:35:09.229406 17090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e: Bootstrap complete.
I20260926 23:35:09.239305 17090 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccac865f8548457ca50960ec81cd9c6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46435 } }
I20260926 23:35:09.240330 17090 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccac865f8548457ca50960ec81cd9c6e, State: Initialized, Role: FOLLOWER
I20260926 23:35:09.241118 17090 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccac865f8548457ca50960ec81cd9c6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46435 } }
I20260926 23:35:09.241374 17090 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:35:09.241506 17090 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:35:09.241677 17090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [term 2 FOLLOWER]: Advancing to term 3
I20260926 23:35:09.246129 17090 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccac865f8548457ca50960ec81cd9c6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46435 } }
I20260926 23:35:09.246789 17090 leader_election.cc:302] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccac865f8548457ca50960ec81cd9c6e; no voters:
I20260926 23:35:09.247488 17090 leader_election.cc:288] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 23:35:09.247686 17094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [term 3 FOLLOWER]: Leader election won for term 3
I20260926 23:35:09.249056 17094 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [term 3 LEADER]: Becoming Leader. State: Replica: ccac865f8548457ca50960ec81cd9c6e, State: Running, Role: LEADER
I20260926 23:35:09.249821 17094 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccac865f8548457ca50960ec81cd9c6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46435 } }
I20260926 23:35:09.250183 17090 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:35:09.254930 17095 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ccac865f8548457ca50960ec81cd9c6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccac865f8548457ca50960ec81cd9c6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46435 } } }
I20260926 23:35:09.255511 17095 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [sys.catalog]: This master's current role is: LEADER
I20260926 23:35:09.253883 17096 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccac865f8548457ca50960ec81cd9c6e. Latest consensus state: current_term: 3 leader_uuid: "ccac865f8548457ca50960ec81cd9c6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccac865f8548457ca50960ec81cd9c6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46435 } } }
I20260926 23:35:09.256595 17096 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e [sys.catalog]: This master's current role is: LEADER
I20260926 23:35:09.267123 17103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:35:09.279271 17103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:35:09.281421 17103 catalog_manager.cc:1259] Loaded cluster ID: 15657a014ee248b7a2b50e0902dcf253
I20260926 23:35:09.281642 17103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:35:09.286962 17103 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:35:09.289305 17103 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e: Loaded TSK: 0
I20260926 23:35:09.289634 17103 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ccac865f8548457ca50960ec81cd9c6e: Loaded TSK: 1
F20260926 23:35:09.291226 17103 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f98c66459ad google::LogMessage::Fail() at ??:0
@ 0x7f98c664888a google::LogMessage::SendToLog() at ??:0
@ 0x7f98c66454de google::LogMessage::Flush() at ??:0
@ 0x7f98c6646a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f98ca3a97d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f98ca39873e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f98dc0741c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f98dc073783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f98dc0ff2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f98dc0ff0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f98dc1495e9 std::function<>::operator()() at ??:0
@ 0x7f98dc077cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f98dc075c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f98dc0fd610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f98dc0fd439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f98c9291345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f98c929491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f98c9294799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f98c926feaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f98cc5496db start_thread at ??:0
@ 0x7f98c54df61f clone at ??:0
W20260926 23:35:09.642573 17092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:35:09.643074 17092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:35:09.643255 17092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:35:09.651952 17092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:35:09.652141 17092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:35:09.652349 17092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:35:09.664297 17092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:36783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=40445
--tserver_master_addrs=127.0.97.62:46435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:35:09.666267 17092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:35:09.668192 17092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:35:09.684417 17140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:09.684546 17137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:09.684582 17138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:35:09.685971 17092 server_base.cc:1034] running on GCE node
I20260926 23:35:09.686841 17092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:35:09.688057 17092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:35:09.689297 17092 hybrid_clock.cc:648] HybridClock initialized: now 1790465709689216 us; error 71 us; skew 500 ppm
I20260926 23:35:09.689685 17092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:35:09 dist-test-slave-gwfx krb5kdc[15935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:35:09.693508 17092 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:35:09.695493 17092 webserver.cc:460] Webserver started at http://127.0.97.1:40445/ using document root <none> and password file <none>
I20260926 23:35:09.696353 17092 fs_manager.cc:359] Metadata directory not provided
I20260926 23:35:09.696483 17092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:35:09.703171 17092 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 23:35:09.706451 17147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:09.708230 17092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 23:35:09.708423 17092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "ae8abc5b72a04cb59f4a9dc64d532cac"
format_stamp: "Formatted at 2026-09-26 23:34:22 on dist-test-slave-gwfx"
server_key: "00c43a93fd78032679f5511a10a0a81c"
server_key_iv: "1ad40631d1d9c466d7940c65184ea2d6"
server_key_version: "encryptionkey@0"
I20260926 23:35:09.709583 17092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:35:09.726246 17092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:35:09.731752 17092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:35:09.732113 17092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:35:09.733422 17092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:35:09.735232 17092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:35:09.735342 17092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:09.735472 17092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:35:09.735548 17092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:09.772892 17092 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36783
I20260926 23:35:09.772956 17259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36783 every 8 connection(s)
I20260926 23:35:09.774750 17092 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:35:09.775745 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 17092
I20260926 23:35:09.776582 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:34735
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46435
--webserver_port=33279
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:35:09.781558 17262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:09.779083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.1:46149)
0926 23:35:09.780118 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:09.780380 (+ 262us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":705,"thread_start_us":432,"threads_started":1}
W20260926 23:35:09.782536 17260 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46435 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46435: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
I20260926 23:35:09.783905 17262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:09.783434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.1:44401)
0926 23:35:09.783654 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:09.783748 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 23:35:09.785060 17262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:09.784605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.1:45767)
0926 23:35:09.784817 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:09.784914 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260926 23:35:09.785326 17260 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 23:35:10.193428 17263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:35:10.193881 17263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:35:10.194025 17263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:35:10.202533 17263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:35:10.202744 17263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:35:10.202960 17263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:35:10.214929 17263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:34735
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33279
--tserver_master_addrs=127.0.97.62:46435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:35:10.216761 17263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:35:10.218806 17263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:35:10.235085 17271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:10.235180 17268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:10.235143 17269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:35:10.236420 17263 server_base.cc:1034] running on GCE node
I20260926 23:35:10.237231 17263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:35:10.238584 17263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:35:10.239833 17263 hybrid_clock.cc:648] HybridClock initialized: now 1790465710239778 us; error 61 us; skew 500 ppm
I20260926 23:35:10.240316 17263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:35:10 dist-test-slave-gwfx krb5kdc[15935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:35:10.244304 17263 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:35:10.246294 17263 webserver.cc:460] Webserver started at http://127.0.97.2:33279/ using document root <none> and password file <none>
I20260926 23:35:10.247275 17263 fs_manager.cc:359] Metadata directory not provided
I20260926 23:35:10.247408 17263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:35:10.254673 17263 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 23:35:10.258359 17278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:10.260392 17263 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:35:10.260623 17263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "e9e7c7787b5040b9910b34091f82f6ce"
format_stamp: "Formatted at 2026-09-26 23:34:23 on dist-test-slave-gwfx"
server_key: "d04606eac364773339cde975cf39aa39"
server_key_iv: "19d1f082e1b5692deb59c99b58564859"
server_key_version: "encryptionkey@0"
I20260926 23:35:10.261883 17263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:35:10.287811 17263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:35:10.293571 17263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:35:10.293912 17263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:35:10.295252 17263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:35:10.297394 17263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:35:10.297530 17263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:10.297698 17263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:35:10.297789 17263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:10.338107 17263 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34735
I20260926 23:35:10.338240 17390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34735 every 8 connection(s)
I20260926 23:35:10.340277 17263 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:35:10.340988 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 17263
I20260926 23:35:10.341910 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35683
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46435
--webserver_port=43345
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:35:10.347939 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:10.344661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.2:51615)
0926 23:35:10.345862 (+ 1201us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:10.346215 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":793,"thread_start_us":468,"threads_started":1}
W20260926 23:35:10.348974 17391 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46435 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46435: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
I20260926 23:35:10.350250 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:10.349726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.2:48203)
0926 23:35:10.349948 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:10.350068 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 23:35:10.351632 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:10.351118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.2:52499)
0926 23:35:10.351348 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:10.351457 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 23:35:10.351977 17391 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 23:35:10.765102 17394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:35:10.765504 17394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:35:10.765614 17394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:35:10.774014 17394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:35:10.774160 17394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:35:10.774339 17394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:35:10.787007 17394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43345
--tserver_master_addrs=127.0.97.62:46435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:35:10.787218 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:10.786211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.1:35593)
0926 23:35:10.786836 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:10.786949 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":282,"threads_started":1}
I20260926 23:35:10.788834 17394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:35:10.790558 17394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:35:10.807576 17401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:10.807521 17403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:10.807478 17400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:35:10.808929 17394 server_base.cc:1034] running on GCE node
I20260926 23:35:10.809820 17394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:35:10.811295 17394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:35:10.812539 17394 hybrid_clock.cc:648] HybridClock initialized: now 1790465710812478 us; error 57 us; skew 500 ppm
I20260926 23:35:10.813017 17394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:35:10 dist-test-slave-gwfx krb5kdc[15935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:35:10.817346 17394 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:35:10.819820 17394 webserver.cc:460] Webserver started at http://127.0.97.3:43345/ using document root <none> and password file <none>
I20260926 23:35:10.820834 17394 fs_manager.cc:359] Metadata directory not provided
I20260926 23:35:10.820986 17394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:35:10.828958 17394 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 23:35:10.833284 17410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:10.835577 17394 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 23:35:10.835845 17394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "1eb86cb9e5b64ad689580147ff05dcdb"
format_stamp: "Formatted at 2026-09-26 23:34:23 on dist-test-slave-gwfx"
server_key: "87c326a8254e6ee107818e51a93f4695"
server_key_iv: "dad5ceed5c86b397f758d128b01a1795"
server_key_version: "encryptionkey@0"
I20260926 23:35:10.837414 17394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:35:10.855659 17394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:35:10.861961 17394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:35:10.862295 17394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:35:10.863747 17394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:35:10.865655 17394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:35:10.865763 17394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:10.865877 17394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:35:10.865954 17394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:10.904072 17394 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35683
I20260926 23:35:10.904158 17522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35683 every 8 connection(s)
I20260926 23:35:10.905843 17394 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:35:10.906432 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 17394
I20260926 23:35:10.911984 17526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:10.909513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.3:46417)
0926 23:35:10.910491 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:10.910829 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":352,"threads_started":1}
W20260926 23:35:10.912864 17523 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46435 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46435: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
I20260926 23:35:10.914096 17526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:10.913579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.3:36731)
0926 23:35:10.913870 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:10.913962 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 23:35:10.915242 17526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:10.914851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.3:44787)
0926 23:35:10.915022 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:10.915113 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 23:35:10.915566 17523 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 23:35:11.354032 17527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:11.352812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.2:33105)
0926 23:35:11.353577 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:11.353723 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":368,"threads_started":1}
I20260926 23:35:11.789625 17528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:11.788666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.1:42239)
0926 23:35:11.789285 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:11.789394 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":347,"threads_started":1}
I20260926 23:35:11.917420 17529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:11.916359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.3:51321)
0926 23:35:11.917047 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:11.917152 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":340,"threads_started":1}
I20260926 23:35:12.356457 17530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:12.355437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.2:46363)
0926 23:35:12.356087 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:12.356222 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":304,"threads_started":1}
I20260926 23:35:12.791999 17531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:12.791012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.1:47681)
0926 23:35:12.791634 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:12.791752 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":313,"threads_started":1}
I20260926 23:35:12.919706 17532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:12.918740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.3:38303)
0926 23:35:12.919391 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:12.919515 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":315,"threads_started":1}
I20260926 23:35:13.358606 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:13.357646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.2:60527)
0926 23:35:13.358270 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:13.358388 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":324,"threads_started":1}
I20260926 23:35:13.794348 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:13.793311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.1:59097)
0926 23:35:13.793981 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:13.794116 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":321,"threads_started":1}
I20260926 23:35:13.922086 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:13.920963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.3:46055)
0926 23:35:13.921688 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:13.921838 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":377,"threads_started":1}
I20260926 23:35:14.361033 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:14.359972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.2:42845)
0926 23:35:14.360642 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:14.360779 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":311,"threads_started":1}
I20260926 23:35:14.796816 17537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:14.795699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.1:37457)
0926 23:35:14.796476 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:14.796597 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":407,"threads_started":1}
I20260926 23:35:14.924532 17538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:14.923460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.3:55233)
0926 23:35:14.924095 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:14.924289 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":309,"threads_started":1}
I20260926 23:35:15.363461 17539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:15.362414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.2:59759)
0926 23:35:15.363123 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:15.363238 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":391,"threads_started":1}
I20260926 23:35:15.799335 17540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:15.798195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.1:51269)
0926 23:35:15.798876 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:15.799015 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":331,"threads_started":1}
I20260926 23:35:15.926893 17541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:15.925903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.3:51367)
0926 23:35:15.926527 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:15.926634 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":332,"threads_started":1}
I20260926 23:35:16.365700 17542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:16.364712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.2:35143)
0926 23:35:16.365349 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:16.365492 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":392,"threads_started":1}
I20260926 23:35:16.801852 17543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:16.800739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.1:50185)
0926 23:35:16.801469 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:16.801593 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":374,"threads_started":1}
I20260926 23:35:16.929311 17544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:16.928140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.3:48543)
0926 23:35:16.928951 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:16.929106 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":363,"threads_started":1}
I20260926 23:35:17.368325 17545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:17.367127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.2:32811)
0926 23:35:17.367866 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:17.367990 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":345,"threads_started":1}
I20260926 23:35:17.804127 17546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:17.803226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.1:44623)
0926 23:35:17.803814 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:17.803933 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":283,"threads_started":1}
I20260926 23:35:17.931888 17547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:17.930659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.3:51069)
0926 23:35:17.931477 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:17.931602 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":401,"threads_started":1}
I20260926 23:35:18.370517 17548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:18.369524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.2:50533)
0926 23:35:18.370219 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:18.370318 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":323,"threads_started":1}
I20260926 23:35:18.806469 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:18.805418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.1:55913)
0926 23:35:18.806146 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:18.806265 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":382,"threads_started":1}
I20260926 23:35:18.934408 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:18.933278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.3:34319)
0926 23:35:18.934048 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:18.934182 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":359,"threads_started":1}
I20260926 23:35:19.372790 17551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:19.371782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.2:46887)
0926 23:35:19.372445 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:19.372566 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":307,"threads_started":1}
I20260926 23:35:19.808995 17552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:19.808004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.1:40149)
0926 23:35:19.808670 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:19.808787 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":346,"threads_started":1}
I20260926 23:35:19.936611 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:19.935596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.3:46273)
0926 23:35:19.936285 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:19.936413 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":303,"threads_started":1}
I20260926 23:35:20.375077 17554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:20.374088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.2:59533)
0926 23:35:20.374773 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:20.374873 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":353,"threads_started":1}
I20260926 23:35:20.811467 17555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:20.810423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.1:36377)
0926 23:35:20.811105 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:20.811226 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":338,"threads_started":1}
I20260926 23:35:20.938917 17556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:20.937912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.3:56209)
0926 23:35:20.938559 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:20.938666 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":372,"threads_started":1}
I20260926 23:35:21.377543 17557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:21.376523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.2:37701)
0926 23:35:21.377175 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:21.377301 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":323,"threads_started":1}
I20260926 23:35:21.814023 17558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:21.812937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.1:42853)
0926 23:35:21.813561 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:21.813689 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":311,"threads_started":1}
I20260926 23:35:21.941260 17559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:21.940161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.3:44881)
0926 23:35:21.940818 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:21.940961 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":309,"threads_started":1}
I20260926 23:35:22.379736 17560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:22.378907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.2:60925)
0926 23:35:22.379436 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:22.379537 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":263,"threads_started":1}
I20260926 23:35:22.816447 17561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:22.815402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.1:42235)
0926 23:35:22.816077 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:22.816195 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":349,"threads_started":1}
I20260926 23:35:22.944023 17562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:22.942978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.3:39993)
0926 23:35:22.943634 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:22.943835 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":334,"threads_started":1}
I20260926 23:35:23.382018 17563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:23.380980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.2:44565)
0926 23:35:23.381667 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:23.381779 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":328,"threads_started":1}
I20260926 23:35:23.818619 17564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:23.817666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.1:57607)
0926 23:35:23.818284 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:23.818403 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":320,"threads_started":1}
I20260926 23:35:23.946151 17565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:23.945155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.3:42731)
0926 23:35:23.945760 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:23.945919 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":299,"threads_started":1}
I20260926 23:35:24.384298 17566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:24.383379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.2:33249)
0926 23:35:24.383955 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:24.384057 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":319,"threads_started":1}
I20260926 23:35:24.821813 17567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:24.820281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.1:47209)
0926 23:35:24.821303 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:24.821440 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":852,"thread_start_us":689,"threads_started":1}
I20260926 23:35:24.948357 17568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:24.947417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.3:46069)
0926 23:35:24.948007 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:24.948111 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":312,"threads_started":1}
I20260926 23:35:25.386896 17569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:25.385721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.2:36895)
0926 23:35:25.386480 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:25.386598 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":409,"threads_started":1}
I20260926 23:35:25.824470 17570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:25.823352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.1:35615)
0926 23:35:25.824032 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:25.824252 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":321,"threads_started":1}
I20260926 23:35:25.917965 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 17092
I20260926 23:35:25.939213 17571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:25.919656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49540 (local address 127.0.97.1:36783)
0926 23:35:25.920642 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:25.920664 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:25.920722 (+ 58us) server_negotiation.cc:407] Connection header received
0926 23:35:25.922499 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:25.922510 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:25.923025 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:25.923380 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:25.925010 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:25.926306 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:25.927721 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:25.928453 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:25.928637 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:25.931717 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:25.931727 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:25.931747 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:35:25.931796 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:25.935353 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:25.936175 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:25.936183 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:25.936192 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:25.936416 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:25.936828 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:25.936832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:25.936834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:25.937476 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:25.937807 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:25.938312 (+ 505us) server_negotiation.cc:299] Negotiation successful
0926 23:35:25.938874 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":749,"thread_start_us":508,"threads_started":1}
I20260926 23:35:25.941226 17255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:35:25.951071 17573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:25.949942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46435 (local address 127.0.97.3:41071)
0926 23:35:25.950633 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:25.950844 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":339,"threads_started":1}
I20260926 23:35:26.037356 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 17092
I20260926 23:35:26.055650 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 17263
I20260926 23:35:26.081239 17574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:26.056975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33926 (local address 127.0.97.2:34735)
0926 23:35:26.057859 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:26.057875 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:26.057922 (+ 47us) server_negotiation.cc:407] Connection header received
0926 23:35:26.060030 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:26.060045 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:26.060537 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:26.060883 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:26.062815 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:26.064992 (+ 2177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:26.066818 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:26.067840 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:26.068051 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:26.071568 (+ 3517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:26.071579 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:26.071602 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:35:26.071661 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:26.075981 (+ 4320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:26.077220 (+ 1239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:26.077236 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:26.077245 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:26.077657 (+ 412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:26.078254 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:26.078259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:26.078262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:26.079361 (+ 1099us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:26.079716 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:26.080326 (+ 610us) server_negotiation.cc:299] Negotiation successful
0926 23:35:26.080900 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":303,"threads_started":1}
I20260926 23:35:26.083312 17386 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:35:26.176055 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 17263
I20260926 23:35:26.193192 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 17394
I20260926 23:35:26.215286 17576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:26.194515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42052 (local address 127.0.97.3:35683)
0926 23:35:26.195238 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:26.195257 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:26.195308 (+ 51us) server_negotiation.cc:407] Connection header received
0926 23:35:26.197143 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:26.197154 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:26.197713 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:26.198139 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:26.199952 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:26.201416 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:26.202923 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:26.203771 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:26.203999 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:26.207130 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:26.207141 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:26.207166 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:35:26.207238 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:26.211035 (+ 3797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:26.211977 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:26.211986 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:26.211996 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:26.212216 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:26.212658 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:26.212663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:26.212666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:26.213402 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:26.213760 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:26.214286 (+ 526us) server_negotiation.cc:299] Negotiation successful
0926 23:35:26.214906 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":302,"threads_started":1}
I20260926 23:35:26.217240 17518 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:35:26.307886 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 17394
2026-09-26T23:35:26Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (66150 ms)
[----------] 30 tests from SecurityITest (325561 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:35:26 dist-test-slave-gwfx krb5kdc[17580](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:35:26 dist-test-slave-gwfx krb5kdc[17580](info): set up 2 sockets
Sep 26 23:35:26 dist-test-slave-gwfx krb5kdc[17580](info): commencing operation
krb5kdc: starting...
W20260926 23:35:27.367102 388 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:35:27 dist-test-slave-gwfx krb5kdc[17580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:35:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:35:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:35:27.663864 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39383
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:35:28.044493 17593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:35:28.044834 17593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:35:28.044957 17593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:35:28.054553 17593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:35:28.054757 17593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:35:28.054818 17593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:35:28.054916 17593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:35:28.054991 17593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:35:28.066932 17593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:35:28.068830 17593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:35:28.070595 17593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:35:28.088167 17598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:28.088199 17599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:28.088224 17601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:35:28.088338 17593 server_base.cc:1034] running on GCE node
I20260926 23:35:28.089818 17593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:35:28.091965 17593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:35:28.093292 17593 hybrid_clock.cc:648] HybridClock initialized: now 1790465728093221 us; error 60 us; skew 500 ppm
I20260926 23:35:28.093815 17593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:35:28 dist-test-slave-gwfx krb5kdc[17580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:35:28.097981 17593 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:35:28.100837 17593 webserver.cc:460] Webserver started at http://127.0.97.62:34709/ using document root <none> and password file <none>
I20260926 23:35:28.101922 17593 fs_manager.cc:359] Metadata directory not provided
I20260926 23:35:28.102119 17593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:35:28.102535 17593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:35:28.106825 17593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "9d041948b9ef429982071cadaf94f23e"
format_stamp: "Formatted at 2026-09-26 23:35:28 on dist-test-slave-gwfx"
server_key: "4bbfc6b1ef03b2d4d9c831f0afb6dab4"
server_key_iv: "721648332a35fc12040528af774fdcc4"
server_key_version: "encryptionkey@0"
I20260926 23:35:28.108078 17593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "9d041948b9ef429982071cadaf94f23e"
format_stamp: "Formatted at 2026-09-26 23:35:28 on dist-test-slave-gwfx"
server_key: "4bbfc6b1ef03b2d4d9c831f0afb6dab4"
server_key_iv: "721648332a35fc12040528af774fdcc4"
server_key_version: "encryptionkey@0"
I20260926 23:35:28.116950 17593 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20260926 23:35:28.122217 17608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:28.124430 17593 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 23:35:28.124730 17593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "9d041948b9ef429982071cadaf94f23e"
format_stamp: "Formatted at 2026-09-26 23:35:28 on dist-test-slave-gwfx"
server_key: "4bbfc6b1ef03b2d4d9c831f0afb6dab4"
server_key_iv: "721648332a35fc12040528af774fdcc4"
server_key_version: "encryptionkey@0"
I20260926 23:35:28.125041 17593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:35:28.143021 17593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:35:28.149081 17593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:35:28.149477 17593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:35:28.174044 17593 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39383
I20260926 23:35:28.174081 17659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39383 every 8 connection(s)
I20260926 23:35:28.175925 17593 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:35:28.180473 17660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:35:28.185545 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 17593
I20260926 23:35:28.185797 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:35:28.186510 388 external_mini_cluster.cc:1428] Setting key 6195ec9bc52998fef3e21bda859cf09e
I20260926 23:35:28.193041 17660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d041948b9ef429982071cadaf94f23e: Bootstrap starting.
I20260926 23:35:28.197913 17660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d041948b9ef429982071cadaf94f23e: Neither blocks nor log segments found. Creating new log.
I20260926 23:35:28.199749 17660 log.cc:824] T 00000000000000000000000000000000 P 9d041948b9ef429982071cadaf94f23e: Log is configured to *not* fsync() on all Append() calls
I20260926 23:35:28.204294 17660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d041948b9ef429982071cadaf94f23e: No bootstrap required, opened a new log
Sep 26 23:35:28 dist-test-slave-gwfx krb5kdc[17580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:35:28.214439 17660 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d041948b9ef429982071cadaf94f23e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d041948b9ef429982071cadaf94f23e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39383 } }
I20260926 23:35:28.215020 17660 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d041948b9ef429982071cadaf94f23e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:35:28.215143 17660 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d041948b9ef429982071cadaf94f23e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d041948b9ef429982071cadaf94f23e, State: Initialized, Role: FOLLOWER
I20260926 23:35:28.215876 17660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d041948b9ef429982071cadaf94f23e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d041948b9ef429982071cadaf94f23e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39383 } }
I20260926 23:35:28.216157 17660 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d041948b9ef429982071cadaf94f23e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:35:28.216296 17660 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d041948b9ef429982071cadaf94f23e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:35:28.216521 17660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d041948b9ef429982071cadaf94f23e [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:35:28.219298 17663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:28.189067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53442 (local address 127.0.97.62:39383)
0926 23:35:28.190196 (+ 1129us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:28.190217 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:28.190286 (+ 69us) server_negotiation.cc:407] Connection header received
0926 23:35:28.192620 (+ 2334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:28.192659 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:28.193266 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:28.193777 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:28.195884 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:28.198049 (+ 2165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:28.200418 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:28.201939 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:28.202176 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:28.207949 (+ 5773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:28.207962 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:28.207989 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:35:28.208067 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:28.212904 (+ 4837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:28.214226 (+ 1322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:28.214239 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:28.214249 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:28.214574 (+ 325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:28.215231 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:28.215237 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:28.215239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:28.216182 (+ 943us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:28.216622 (+ 440us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:28.217107 (+ 485us) server_negotiation.cc:299] Negotiation successful
0926 23:35:28.217775 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":529,"threads_started":1}
I20260926 23:35:28.219532 17660 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d041948b9ef429982071cadaf94f23e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d041948b9ef429982071cadaf94f23e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39383 } }
I20260926 23:35:28.220345 17660 leader_election.cc:302] T 00000000000000000000000000000000 P 9d041948b9ef429982071cadaf94f23e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d041948b9ef429982071cadaf94f23e; no voters:
I20260926 23:35:28.221271 17660 leader_election.cc:288] T 00000000000000000000000000000000 P 9d041948b9ef429982071cadaf94f23e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:35:28.221474 17665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d041948b9ef429982071cadaf94f23e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:35:28.223486 17665 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d041948b9ef429982071cadaf94f23e [term 1 LEADER]: Becoming Leader. State: Replica: 9d041948b9ef429982071cadaf94f23e, State: Running, Role: LEADER
I20260926 23:35:28.224435 17665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d041948b9ef429982071cadaf94f23e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d041948b9ef429982071cadaf94f23e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39383 } }
I20260926 23:35:28.224975 17660 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d041948b9ef429982071cadaf94f23e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:35:28.227979 17666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d041948b9ef429982071cadaf94f23e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d041948b9ef429982071cadaf94f23e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d041948b9ef429982071cadaf94f23e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39383 } } }
I20260926 23:35:28.228539 17666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d041948b9ef429982071cadaf94f23e [sys.catalog]: This master's current role is: LEADER
I20260926 23:35:28.229243 17667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d041948b9ef429982071cadaf94f23e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d041948b9ef429982071cadaf94f23e. Latest consensus state: current_term: 1 leader_uuid: "9d041948b9ef429982071cadaf94f23e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d041948b9ef429982071cadaf94f23e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39383 } } }
I20260926 23:35:28.229873 17667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d041948b9ef429982071cadaf94f23e [sys.catalog]: This master's current role is: LEADER
I20260926 23:35:28.232785 17670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:35:28.240613 17670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:35:28.252496 17670 catalog_manager.cc:1347] Generated new cluster ID: 80b01b3d76324479b5af061969a13a2e
I20260926 23:35:28.252703 17670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:35:28.266664 17670 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:35:28.269127 17670 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:35:28.281955 17670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d041948b9ef429982071cadaf94f23e: Generated new TSK 0
I20260926 23:35:28.283069 17670 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:35:28.363306 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39383
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:35:28.728956 17686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:35:28.729274 17686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:35:28.729408 17686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:35:28.737573 17686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:35:28.737716 17686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:35:28.737867 17686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:35:28.748526 17686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:35:28.750027 17686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:35:28.751535 17686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:35:28.768203 17691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:28.768226 17694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:28.768309 17692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:35:28.772898 17686 server_base.cc:1034] running on GCE node
I20260926 23:35:28.773751 17686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:35:28.774991 17686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:35:28.776281 17686 hybrid_clock.cc:648] HybridClock initialized: now 1790465728776224 us; error 48 us; skew 500 ppm
I20260926 23:35:28.776746 17686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:35:28 dist-test-slave-gwfx krb5kdc[17580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:35:28.780587 17686 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:35:28.782634 17686 webserver.cc:460] Webserver started at http://127.0.97.1:45993/ using document root <none> and password file <none>
I20260926 23:35:28.783483 17686 fs_manager.cc:359] Metadata directory not provided
I20260926 23:35:28.783612 17686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:35:28.783932 17686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:35:28.787150 17686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-0/data/instance:
uuid: "cd87529b018346e08173ffbb9b279490"
format_stamp: "Formatted at 2026-09-26 23:35:28 on dist-test-slave-gwfx"
server_key: "10613752315141c5924973447a4b9925"
server_key_iv: "322982f826dfb3fbaed5e30e525ad111"
server_key_version: "encryptionkey@0"
I20260926 23:35:28.788015 17686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-0/wal/instance:
uuid: "cd87529b018346e08173ffbb9b279490"
format_stamp: "Formatted at 2026-09-26 23:35:28 on dist-test-slave-gwfx"
server_key: "10613752315141c5924973447a4b9925"
server_key_iv: "322982f826dfb3fbaed5e30e525ad111"
server_key_version: "encryptionkey@0"
I20260926 23:35:28.794123 17686 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 23:35:28.798761 17701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:28.800638 17686 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:35:28.800863 17686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "cd87529b018346e08173ffbb9b279490"
format_stamp: "Formatted at 2026-09-26 23:35:28 on dist-test-slave-gwfx"
server_key: "10613752315141c5924973447a4b9925"
server_key_iv: "322982f826dfb3fbaed5e30e525ad111"
server_key_version: "encryptionkey@0"
I20260926 23:35:28.801095 17686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:35:28.823774 17686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:35:28.828733 17686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:35:28.829080 17686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:35:28.830406 17686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:35:28.832144 17686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:35:28.832232 17686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:28.832365 17686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:35:28.832417 17686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:28.870361 17686 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44757
I20260926 23:35:28.870477 17813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44757 every 8 connection(s)
I20260926 23:35:28.871811 17686 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:35:28.877979 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 17686
I20260926 23:35:28.878252 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:35:28.879223 388 external_mini_cluster.cc:1428] Setting key 3a4b1d781b7b6befb863596e5061b30f
Sep 26 23:35:28 dist-test-slave-gwfx krb5kdc[17580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:35:28.899077 17816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:28.875164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52961 (local address 127.0.97.62:39383)
0926 23:35:28.875720 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:28.875725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:28.876845 (+ 1120us) server_negotiation.cc:407] Connection header received
0926 23:35:28.880288 (+ 3443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:28.880294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:28.880396 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:28.880504 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:28.883034 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:28.884197 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:28.886075 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:28.886742 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:28.886901 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:28.891317 (+ 4416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:28.891327 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:28.891333 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:35:28.891398 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:28.894671 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:28.895934 (+ 1263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:28.895941 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:28.895944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:28.896132 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:28.896728 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:28.896734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:28.896736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:28.897167 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:28.897316 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:28.898347 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0926 23:35:28.898620 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":291,"threads_started":1}
I20260926 23:35:28.900254 17817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:28.875589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39383 (local address 127.0.97.1:52961)
0926 23:35:28.876608 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:28.876675 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:28.879566 (+ 2891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:28.880968 (+ 1402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:28.880980 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:28.881997 (+ 1017us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:35:28.882781 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:28.882809 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:28.884444 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:28.884448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:28.885876 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:28.885893 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:28.887035 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:28.887041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:28.887280 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:28.887986 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:35:28.888017 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:35:28.890932 (+ 2915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:35:28.894952 (+ 4020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:35:28.894963 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:35:28.894982 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:35:28.895688 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:35:28.896264 (+ 576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:35:28.896269 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:35:28.896274 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:35:28.896550 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:35:28.897315 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:35:28.897324 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:35:28.897819 (+ 495us) client_negotiation.cc:769] Sending connection context
0926 23:35:28.898170 (+ 351us) client_negotiation.cc:240] Negotiation successful
0926 23:35:28.898839 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":398,"threads_started":1}
I20260926 23:35:28.902294 17814 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39383
I20260926 23:35:28.902801 17814 heartbeater.cc:464] Registering TS with master...
I20260926 23:35:28.903885 17814 heartbeater.cc:511] Master 127.0.97.62:39383 requested a full tablet report, sending...
I20260926 23:35:28.907186 17625 ts_manager.cc:194] Registered new tserver with Master: cd87529b018346e08173ffbb9b279490 (127.0.97.1:44757)
I20260926 23:35:28.909036 17625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52961
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:35:28.948484 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39383
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:35:29.380571 17820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:35:29.380997 17820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:35:29.381115 17820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:35:29.389806 17820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:35:29.390046 17820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:35:29.390201 17820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:35:29.401875 17820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:35:29.403654 17820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:35:29.405356 17820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:35:29.421743 17826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:29.421701 17825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:29.421785 17828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:35:29.422547 17820 server_base.cc:1034] running on GCE node
I20260926 23:35:29.423346 17820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:35:29.424468 17820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:35:29.425872 17820 hybrid_clock.cc:648] HybridClock initialized: now 1790465729425784 us; error 82 us; skew 500 ppm
I20260926 23:35:29.426355 17820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:35:29 dist-test-slave-gwfx krb5kdc[17580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465729, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:35:29.436273 17820 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:35:29.438201 17820 webserver.cc:460] Webserver started at http://127.0.97.2:32881/ using document root <none> and password file <none>
I20260926 23:35:29.439108 17820 fs_manager.cc:359] Metadata directory not provided
I20260926 23:35:29.439234 17820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:35:29.439539 17820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:35:29.443177 17820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-1/data/instance:
uuid: "dc898e10dc5f4a21ba7b17c78859c956"
format_stamp: "Formatted at 2026-09-26 23:35:29 on dist-test-slave-gwfx"
server_key: "3982b12630aad62474ef33fc1f29041b"
server_key_iv: "73705d24974f6a1280b36126489dd3b2"
server_key_version: "encryptionkey@0"
I20260926 23:35:29.444070 17820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-1/wal/instance:
uuid: "dc898e10dc5f4a21ba7b17c78859c956"
format_stamp: "Formatted at 2026-09-26 23:35:29 on dist-test-slave-gwfx"
server_key: "3982b12630aad62474ef33fc1f29041b"
server_key_iv: "73705d24974f6a1280b36126489dd3b2"
server_key_version: "encryptionkey@0"
I20260926 23:35:29.451725 17820 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260926 23:35:29.456667 17835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:29.458814 17820 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260926 23:35:29.459059 17820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "dc898e10dc5f4a21ba7b17c78859c956"
format_stamp: "Formatted at 2026-09-26 23:35:29 on dist-test-slave-gwfx"
server_key: "3982b12630aad62474ef33fc1f29041b"
server_key_iv: "73705d24974f6a1280b36126489dd3b2"
server_key_version: "encryptionkey@0"
I20260926 23:35:29.459281 17820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:35:29.484301 17820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:35:29.489845 17820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:35:29.490144 17820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:35:29.491331 17820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:35:29.492971 17820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:35:29.493059 17820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:29.493150 17820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:35:29.493203 17820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:29.532450 17820 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46327
I20260926 23:35:29.532581 17947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46327 every 8 connection(s)
I20260926 23:35:29.533934 17820 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:35:29.539593 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 17820
I20260926 23:35:29.539770 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:35:29.540231 388 external_mini_cluster.cc:1428] Setting key 13a89b0c1a80fc0e5ec519d635032e31
Sep 26 23:35:29 dist-test-slave-gwfx krb5kdc[17580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465729, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:35:29.560998 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:29.537390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50323 (local address 127.0.97.62:39383)
0926 23:35:29.538016 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:29.538023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:29.539553 (+ 1530us) server_negotiation.cc:407] Connection header received
0926 23:35:29.542312 (+ 2759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:29.542319 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:29.542438 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:29.542579 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:29.545071 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:29.546136 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:29.548552 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:29.549154 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:29.549333 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:29.554643 (+ 5310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:29.554649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:29.554652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:35:29.554727 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:29.557408 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:29.558333 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:29.558338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:29.558340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:29.558483 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:29.559038 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:29.559043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:29.559046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:29.559385 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:29.559537 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:29.560338 (+ 801us) server_negotiation.cc:299] Negotiation successful
0926 23:35:29.560607 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":281,"threads_started":1}
I20260926 23:35:29.562081 17951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:29.538011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39383 (local address 127.0.97.2:50323)
0926 23:35:29.539294 (+ 1283us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:29.539356 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:29.541803 (+ 2447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:29.543116 (+ 1313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:29.543128 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:29.543883 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:35:29.544699 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:29.544728 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:29.546391 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:29.546397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:29.548311 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:29.548335 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:29.549563 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:29.549571 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:29.549899 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:29.551113 (+ 1214us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:35:29.551157 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:35:29.554315 (+ 3158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:35:29.557620 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:35:29.557630 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:35:29.557647 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:35:29.558183 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:35:29.558606 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:35:29.558611 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:35:29.558627 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:35:29.558914 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:35:29.559529 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:35:29.559539 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:35:29.559938 (+ 399us) client_negotiation.cc:769] Sending connection context
0926 23:35:29.560203 (+ 265us) client_negotiation.cc:240] Negotiation successful
0926 23:35:29.560816 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"thread_start_us":457,"threads_started":1}
I20260926 23:35:29.564559 17948 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39383
I20260926 23:35:29.564975 17948 heartbeater.cc:464] Registering TS with master...
I20260926 23:35:29.565840 17948 heartbeater.cc:511] Master 127.0.97.62:39383 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 23:35:29.568023 17625 ts_manager.cc:194] Registered new tserver with Master: dc898e10dc5f4a21ba7b17c78859c956 (127.0.97.2:46327)
I20260926 23:35:29.569144 17625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50323
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:35:29.608212 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39383
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:35:29.913893 17814 heartbeater.cc:503] Master 127.0.97.62:39383 was elected leader, sending a full tablet report...
W20260926 23:35:30.015803 17954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:35:30.016215 17954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:35:30.016389 17954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:35:30.024878 17954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:35:30.025069 17954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:35:30.025240 17954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:35:30.036728 17954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:35:30.038476 17954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:35:30.040107 17954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:35:30.056216 17962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:30.056309 17960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:30.056192 17959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:35:30.062568 17954 server_base.cc:1034] running on GCE node
I20260926 23:35:30.063640 17954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:35:30.065092 17954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:35:30.066393 17954 hybrid_clock.cc:648] HybridClock initialized: now 1790465730066320 us; error 63 us; skew 500 ppm
I20260926 23:35:30.066890 17954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:35:30 dist-test-slave-gwfx krb5kdc[17580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465730, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:35:30.071224 17954 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:35:30.073254 17954 webserver.cc:460] Webserver started at http://127.0.97.3:44547/ using document root <none> and password file <none>
I20260926 23:35:30.074224 17954 fs_manager.cc:359] Metadata directory not provided
I20260926 23:35:30.074374 17954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:35:30.074800 17954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:35:30.078581 17954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-2/data/instance:
uuid: "e765f735346a4b82a83307f7fcdd4a7a"
format_stamp: "Formatted at 2026-09-26 23:35:30 on dist-test-slave-gwfx"
server_key: "93c4d459c0d100430a17592c32953b7a"
server_key_iv: "a1a0578ca4e0ba8217d9a391949035c7"
server_key_version: "encryptionkey@0"
I20260926 23:35:30.079602 17954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e765f735346a4b82a83307f7fcdd4a7a"
format_stamp: "Formatted at 2026-09-26 23:35:30 on dist-test-slave-gwfx"
server_key: "93c4d459c0d100430a17592c32953b7a"
server_key_iv: "a1a0578ca4e0ba8217d9a391949035c7"
server_key_version: "encryptionkey@0"
I20260926 23:35:30.086521 17954 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260926 23:35:30.091586 17969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:30.093694 17954 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260926 23:35:30.093986 17954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "e765f735346a4b82a83307f7fcdd4a7a"
format_stamp: "Formatted at 2026-09-26 23:35:30 on dist-test-slave-gwfx"
server_key: "93c4d459c0d100430a17592c32953b7a"
server_key_iv: "a1a0578ca4e0ba8217d9a391949035c7"
server_key_version: "encryptionkey@0"
I20260926 23:35:30.094290 17954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:35:30.115724 17954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:35:30.121500 17954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:35:30.122028 17954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:35:30.123567 17954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:35:30.125490 17954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:35:30.125586 17954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:30.125721 17954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:35:30.125777 17954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:30.164719 17954 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34081
I20260926 23:35:30.164788 18081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34081 every 8 connection(s)
I20260926 23:35:30.166352 17954 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:35:30.169400 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 17954
I20260926 23:35:30.169608 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:35:30.170192 388 external_mini_cluster.cc:1428] Setting key b9eefe73eafb2a69203d730618bf1150
Sep 26 23:35:30 dist-test-slave-gwfx krb5kdc[17580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465730, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:35:30.192988 18084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:30.169559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34975 (local address 127.0.97.62:39383)
0926 23:35:30.170238 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:30.170246 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:30.171536 (+ 1290us) server_negotiation.cc:407] Connection header received
0926 23:35:30.173982 (+ 2446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:30.173987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:30.174077 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:30.174173 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:30.175935 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.176786 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.178638 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.179177 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.179295 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:30.183779 (+ 4484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:30.183787 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:30.183863 (+ 76us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:35:30.183961 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:30.188091 (+ 4130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:30.189253 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:30.189259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:30.189262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:30.189463 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:30.190182 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:30.190188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:30.190191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:30.190603 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:30.190844 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:30.192315 (+ 1471us) server_negotiation.cc:299] Negotiation successful
0926 23:35:30.192628 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":337,"threads_started":1}
I20260926 23:35:30.194021 18085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:30.170298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39383 (local address 127.0.97.3:34975)
0926 23:35:30.171336 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:30.171401 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:30.173663 (+ 2262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:30.174479 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:30.174488 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:30.175138 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:35:30.175756 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:30.175776 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.176940 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.176944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:30.178454 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:30.178469 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.179465 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.179473 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:30.179688 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:30.180372 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:35:30.180402 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:35:30.183382 (+ 2980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:35:30.188376 (+ 4994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:35:30.188386 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:35:30.188404 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:35:30.189032 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:35:30.189666 (+ 634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:35:30.189672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:35:30.189676 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:35:30.189982 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:35:30.190949 (+ 967us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:35:30.190959 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:35:30.191701 (+ 742us) client_negotiation.cc:769] Sending connection context
0926 23:35:30.192051 (+ 350us) client_negotiation.cc:240] Negotiation successful
0926 23:35:30.192544 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":414,"threads_started":1}
I20260926 23:35:30.196557 18082 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39383
I20260926 23:35:30.197154 18082 heartbeater.cc:464] Registering TS with master...
I20260926 23:35:30.198906 18082 heartbeater.cc:511] Master 127.0.97.62:39383 requested a full tablet report, sending...
I20260926 23:35:30.201164 17625 ts_manager.cc:194] Registered new tserver with Master: e765f735346a4b82a83307f7fcdd4a7a (127.0.97.3:34081)
I20260926 23:35:30.202193 17625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34975
I20260926 23:35:30.215898 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:35:30.241004 18084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:30.221373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53444 (local address 127.0.97.62:39383)
0926 23:35:30.221635 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:30.221641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:30.222165 (+ 524us) server_negotiation.cc:407] Connection header received
0926 23:35:30.222424 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:30.222430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:30.222528 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:30.222670 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:30.224665 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.225677 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.227506 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.228038 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.228203 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:30.231715 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:30.231723 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:30.231726 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:35:30.231796 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:30.235978 (+ 4182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:30.237489 (+ 1511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:30.237497 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:30.237500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:30.237837 (+ 337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:30.238855 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:30.238861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:30.238863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:30.239374 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:30.239602 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:30.240363 (+ 761us) server_negotiation.cc:299] Negotiation successful
0926 23:35:30.240638 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260926 23:35:30.246567 17625 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:35:30.256845 17625 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:35:30.290208 18096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:30.274825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45286 (local address 127.0.97.3:34081)
0926 23:35:30.279237 (+ 4412us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:30.279250 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:30.279303 (+ 53us) server_negotiation.cc:407] Connection header received
0926 23:35:30.279462 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:30.279473 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:30.279876 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:30.280626 (+ 750us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:35:30.281256 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.282930 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.286831 (+ 3901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.288742 (+ 1911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.288885 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:30.289208 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:30.289553 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 23:35:30.289774 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4138,"thread_start_us":3713,"threads_started":1}
I20260926 23:35:30.290247 18095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:30.274671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34081 (local address 127.0.0.1:45286)
0926 23:35:30.276760 (+ 2089us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:30.276833 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:30.277211 (+ 378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:30.280320 (+ 3109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:30.280332 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:30.280384 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:35:30.281000 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:30.281016 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.283114 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.283120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:30.286468 (+ 3348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:30.286492 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.288982 (+ 2490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.288989 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:30.289256 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:30.289306 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 23:35:30.289455 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 23:35:30.289752 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1581,"mutex_wait_us":48,"thread_start_us":1352,"threads_started":1}
I20260926 23:35:30.291517 18100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:30.276912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44757 (local address 127.0.0.1:34022)
0926 23:35:30.279511 (+ 2599us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:30.279539 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:30.279747 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:30.281759 (+ 2012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:30.281764 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:30.281790 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:35:30.282268 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:30.282282 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.283942 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.283949 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:30.287879 (+ 3930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:30.287902 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.290415 (+ 2513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.290423 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:30.290628 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:30.290669 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 23:35:30.290821 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 23:35:30.291028 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2425,"mutex_wait_us":325,"thread_start_us":4290,"threads_started":1}
I20260926 23:35:30.292155 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:30.277804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36376 (local address 127.0.97.2:46327)
0926 23:35:30.280289 (+ 2485us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:30.280299 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:30.281743 (+ 1444us) server_negotiation.cc:407] Connection header received
0926 23:35:30.281850 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:30.281857 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:30.282094 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:30.282291 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:35:30.283224 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.284881 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.288621 (+ 3740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.290068 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.290205 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:30.290810 (+ 605us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:30.291642 (+ 832us) server_negotiation.cc:299] Negotiation successful
0926 23:35:30.291839 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2321,"thread_start_us":2171,"threads_started":1}
I20260926 23:35:30.292472 18097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:30.276202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34022 (local address 127.0.97.1:44757)
0926 23:35:30.280739 (+ 4537us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:30.280747 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:30.280781 (+ 34us) server_negotiation.cc:407] Connection header received
0926 23:35:30.280904 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:30.280912 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:30.281242 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:30.281464 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:35:30.282464 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.283799 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.288095 (+ 4296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.289751 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.289927 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:30.290213 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:30.291958 (+ 1745us) server_negotiation.cc:299] Negotiation successful
0926 23:35:30.292167 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4348,"thread_start_us":4172,"threads_started":1}
I20260926 23:35:30.292716 18098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:30.278358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46327 (local address 127.0.0.1:36376)
0926 23:35:30.281480 (+ 3122us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:30.281499 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:30.281613 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:30.282549 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:30.282554 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:30.282581 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:35:30.282979 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:30.282992 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.285110 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.285117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:30.288330 (+ 3213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:30.288354 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.290964 (+ 2610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.290970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:30.291130 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:30.291162 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 23:35:30.291258 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 23:35:30.291397 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2992,"mutex_wait_us":89,"thread_start_us":786,"threads_started":1}
I20260926 23:35:30.296291 18017 tablet_service.cc:1467] Processing CreateTablet for tablet 689810d6162242f4bd08089edb1c816a (DEFAULT_TABLE table=test-table [id=185e925618fd4cc08fd87adb1d14312c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:35:30.296770 17883 tablet_service.cc:1467] Processing CreateTablet for tablet 689810d6162242f4bd08089edb1c816a (DEFAULT_TABLE table=test-table [id=185e925618fd4cc08fd87adb1d14312c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:35:30.297372 17749 tablet_service.cc:1467] Processing CreateTablet for tablet 689810d6162242f4bd08089edb1c816a (DEFAULT_TABLE table=test-table [id=185e925618fd4cc08fd87adb1d14312c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:35:30.298465 17883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 689810d6162242f4bd08089edb1c816a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:35:30.298811 18017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 689810d6162242f4bd08089edb1c816a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:35:30.299229 17749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 689810d6162242f4bd08089edb1c816a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:35:30.313612 18101 tablet_bootstrap.cc:492] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956: Bootstrap starting.
I20260926 23:35:30.317029 18102 tablet_bootstrap.cc:492] T 689810d6162242f4bd08089edb1c816a P e765f735346a4b82a83307f7fcdd4a7a: Bootstrap starting.
I20260926 23:35:30.317495 18101 tablet_bootstrap.cc:654] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956: Neither blocks nor log segments found. Creating new log.
I20260926 23:35:30.318244 18103 tablet_bootstrap.cc:492] T 689810d6162242f4bd08089edb1c816a P cd87529b018346e08173ffbb9b279490: Bootstrap starting.
I20260926 23:35:30.319450 18101 log.cc:824] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956: Log is configured to *not* fsync() on all Append() calls
I20260926 23:35:30.322836 18103 tablet_bootstrap.cc:654] T 689810d6162242f4bd08089edb1c816a P cd87529b018346e08173ffbb9b279490: Neither blocks nor log segments found. Creating new log.
I20260926 23:35:30.323658 18102 tablet_bootstrap.cc:654] T 689810d6162242f4bd08089edb1c816a P e765f735346a4b82a83307f7fcdd4a7a: Neither blocks nor log segments found. Creating new log.
I20260926 23:35:30.324545 18103 log.cc:824] T 689810d6162242f4bd08089edb1c816a P cd87529b018346e08173ffbb9b279490: Log is configured to *not* fsync() on all Append() calls
I20260926 23:35:30.325136 18102 log.cc:824] T 689810d6162242f4bd08089edb1c816a P e765f735346a4b82a83307f7fcdd4a7a: Log is configured to *not* fsync() on all Append() calls
I20260926 23:35:30.325618 18101 tablet_bootstrap.cc:492] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956: No bootstrap required, opened a new log
I20260926 23:35:30.326164 18101 ts_tablet_manager.cc:1397] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.004s
I20260926 23:35:30.330082 18102 tablet_bootstrap.cc:492] T 689810d6162242f4bd08089edb1c816a P e765f735346a4b82a83307f7fcdd4a7a: No bootstrap required, opened a new log
I20260926 23:35:30.330478 18102 ts_tablet_manager.cc:1397] T 689810d6162242f4bd08089edb1c816a P e765f735346a4b82a83307f7fcdd4a7a: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260926 23:35:30.332679 18103 tablet_bootstrap.cc:492] T 689810d6162242f4bd08089edb1c816a P cd87529b018346e08173ffbb9b279490: No bootstrap required, opened a new log
I20260926 23:35:30.333140 18103 ts_tablet_manager.cc:1397] T 689810d6162242f4bd08089edb1c816a P cd87529b018346e08173ffbb9b279490: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.004s
I20260926 23:35:30.339359 18101 raft_consensus.cc:348] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e765f735346a4b82a83307f7fcdd4a7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 } } peers { permanent_uuid: "dc898e10dc5f4a21ba7b17c78859c956" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46327 } } peers { permanent_uuid: "cd87529b018346e08173ffbb9b279490" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44757 } }
I20260926 23:35:30.339968 18101 raft_consensus.cc:374] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:35:30.340173 18101 raft_consensus.cc:729] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc898e10dc5f4a21ba7b17c78859c956, State: Initialized, Role: FOLLOWER
I20260926 23:35:30.341800 18101 consensus_queue.cc:260] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e765f735346a4b82a83307f7fcdd4a7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 } } peers { permanent_uuid: "dc898e10dc5f4a21ba7b17c78859c956" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46327 } } peers { permanent_uuid: "cd87529b018346e08173ffbb9b279490" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44757 } }
I20260926 23:35:30.343921 17948 heartbeater.cc:503] Master 127.0.97.62:39383 was elected leader, sending a full tablet report...
I20260926 23:35:30.344300 18103 raft_consensus.cc:348] T 689810d6162242f4bd08089edb1c816a P cd87529b018346e08173ffbb9b279490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e765f735346a4b82a83307f7fcdd4a7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 } } peers { permanent_uuid: "dc898e10dc5f4a21ba7b17c78859c956" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46327 } } peers { permanent_uuid: "cd87529b018346e08173ffbb9b279490" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44757 } }
I20260926 23:35:30.344810 18101 ts_tablet_manager.cc:1428] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956: Time spent starting tablet: real 0.018s user 0.013s sys 0.007s
I20260926 23:35:30.344988 18103 raft_consensus.cc:374] T 689810d6162242f4bd08089edb1c816a P cd87529b018346e08173ffbb9b279490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:35:30.345114 18103 raft_consensus.cc:729] T 689810d6162242f4bd08089edb1c816a P cd87529b018346e08173ffbb9b279490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd87529b018346e08173ffbb9b279490, State: Initialized, Role: FOLLOWER
I20260926 23:35:30.345558 18102 raft_consensus.cc:348] T 689810d6162242f4bd08089edb1c816a P e765f735346a4b82a83307f7fcdd4a7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e765f735346a4b82a83307f7fcdd4a7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 } } peers { permanent_uuid: "dc898e10dc5f4a21ba7b17c78859c956" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46327 } } peers { permanent_uuid: "cd87529b018346e08173ffbb9b279490" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44757 } }
I20260926 23:35:30.346037 18103 consensus_queue.cc:260] T 689810d6162242f4bd08089edb1c816a P cd87529b018346e08173ffbb9b279490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e765f735346a4b82a83307f7fcdd4a7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 } } peers { permanent_uuid: "dc898e10dc5f4a21ba7b17c78859c956" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46327 } } peers { permanent_uuid: "cd87529b018346e08173ffbb9b279490" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44757 } }
I20260926 23:35:30.346292 18102 raft_consensus.cc:374] T 689810d6162242f4bd08089edb1c816a P e765f735346a4b82a83307f7fcdd4a7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:35:30.346400 18102 raft_consensus.cc:729] T 689810d6162242f4bd08089edb1c816a P e765f735346a4b82a83307f7fcdd4a7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e765f735346a4b82a83307f7fcdd4a7a, State: Initialized, Role: FOLLOWER
I20260926 23:35:30.347254 18102 consensus_queue.cc:260] T 689810d6162242f4bd08089edb1c816a P e765f735346a4b82a83307f7fcdd4a7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e765f735346a4b82a83307f7fcdd4a7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 } } peers { permanent_uuid: "dc898e10dc5f4a21ba7b17c78859c956" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46327 } } peers { permanent_uuid: "cd87529b018346e08173ffbb9b279490" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44757 } }
I20260926 23:35:30.350425 18102 ts_tablet_manager.cc:1428] T 689810d6162242f4bd08089edb1c816a P e765f735346a4b82a83307f7fcdd4a7a: Time spent starting tablet: real 0.020s user 0.018s sys 0.000s
I20260926 23:35:30.350549 18103 ts_tablet_manager.cc:1428] T 689810d6162242f4bd08089edb1c816a P cd87529b018346e08173ffbb9b279490: Time spent starting tablet: real 0.017s user 0.015s sys 0.004s
I20260926 23:35:30.350772 18082 heartbeater.cc:503] Master 127.0.97.62:39383 was elected leader, sending a full tablet report...
W20260926 23:35:30.375077 17815 tablet.cc:2369] T 689810d6162242f4bd08089edb1c816a P cd87529b018346e08173ffbb9b279490: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:35:30.393263 18107 raft_consensus.cc:482] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:35:30.393700 18107 raft_consensus.cc:504] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e765f735346a4b82a83307f7fcdd4a7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 } } peers { permanent_uuid: "dc898e10dc5f4a21ba7b17c78859c956" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46327 } } peers { permanent_uuid: "cd87529b018346e08173ffbb9b279490" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44757 } }
I20260926 23:35:30.396245 18107 leader_election.cc:288] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e765f735346a4b82a83307f7fcdd4a7a (127.0.97.3:34081), cd87529b018346e08173ffbb9b279490 (127.0.97.1:44757)
I20260926 23:35:30.404597 18097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:30.396884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46261 (local address 127.0.97.1:44757)
0926 23:35:30.397137 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:30.397145 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:30.398580 (+ 1435us) server_negotiation.cc:407] Connection header received
0926 23:35:30.398812 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:30.398818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:30.398894 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:30.398971 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:35:30.399846 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.400678 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.402725 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.403551 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.403685 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:30.403820 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:30.404175 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 23:35:30.404318 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 23:35:30.404711 18096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:30.396913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39953 (local address 127.0.97.3:34081)
0926 23:35:30.397183 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:30.397190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:30.398281 (+ 1091us) server_negotiation.cc:407] Connection header received
0926 23:35:30.398536 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:30.398542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:30.398655 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:30.398787 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:35:30.399566 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.400458 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.402621 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.403576 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.403715 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:30.403858 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:30.404276 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 23:35:30.404434 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 23:35:30.404867 18111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:30.396661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34081 (local address 127.0.97.2:39953)
0926 23:35:30.398123 (+ 1462us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:30.398147 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:30.398370 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:30.398811 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:30.398816 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:30.398843 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:35:30.399402 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:30.399418 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.400611 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.400617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:30.402415 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:30.402433 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.403867 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.403875 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:30.404080 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:30.404117 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 23:35:30.404210 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 23:35:30.404387 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1300,"thread_start_us":994,"threads_started":1}
I20260926 23:35:30.405217 18110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:30.396809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44757 (local address 127.0.97.2:46261)
0926 23:35:30.398362 (+ 1553us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:30.398381 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:30.398621 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:30.399096 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:30.399101 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:30.399130 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:35:30.399614 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:30.399624 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.400843 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.400848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:30.402504 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:30.402521 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.403822 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.403829 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:30.403998 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:30.404026 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 23:35:30.404124 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 23:35:30.404296 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1407,"thread_start_us":678,"threads_started":1}
I20260926 23:35:30.406441 17769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "689810d6162242f4bd08089edb1c816a" candidate_uuid: "dc898e10dc5f4a21ba7b17c78859c956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd87529b018346e08173ffbb9b279490" is_pre_election: true
I20260926 23:35:30.406441 18037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "689810d6162242f4bd08089edb1c816a" candidate_uuid: "dc898e10dc5f4a21ba7b17c78859c956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e765f735346a4b82a83307f7fcdd4a7a" is_pre_election: true
I20260926 23:35:30.407352 17769 raft_consensus.cc:2454] T 689810d6162242f4bd08089edb1c816a P cd87529b018346e08173ffbb9b279490 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc898e10dc5f4a21ba7b17c78859c956 in term 0.
I20260926 23:35:30.407341 18037 raft_consensus.cc:2454] T 689810d6162242f4bd08089edb1c816a P e765f735346a4b82a83307f7fcdd4a7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc898e10dc5f4a21ba7b17c78859c956 in term 0.
I20260926 23:35:30.408465 17837 leader_election.cc:302] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc898e10dc5f4a21ba7b17c78859c956, e765f735346a4b82a83307f7fcdd4a7a; no voters:
I20260926 23:35:30.409111 18107 raft_consensus.cc:2781] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:35:30.409319 18107 raft_consensus.cc:482] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:35:30.409514 18107 raft_consensus.cc:3037] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:35:30.412881 18107 raft_consensus.cc:504] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e765f735346a4b82a83307f7fcdd4a7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 } } peers { permanent_uuid: "dc898e10dc5f4a21ba7b17c78859c956" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46327 } } peers { permanent_uuid: "cd87529b018346e08173ffbb9b279490" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44757 } }
I20260926 23:35:30.413695 18107 leader_election.cc:288] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956 [CANDIDATE]: Term 1 election: Requested vote from peers e765f735346a4b82a83307f7fcdd4a7a (127.0.97.3:34081), cd87529b018346e08173ffbb9b279490 (127.0.97.1:44757)
I20260926 23:35:30.414526 17769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "689810d6162242f4bd08089edb1c816a" candidate_uuid: "dc898e10dc5f4a21ba7b17c78859c956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd87529b018346e08173ffbb9b279490"
I20260926 23:35:30.414835 17769 raft_consensus.cc:3037] T 689810d6162242f4bd08089edb1c816a P cd87529b018346e08173ffbb9b279490 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:35:30.415067 18037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "689810d6162242f4bd08089edb1c816a" candidate_uuid: "dc898e10dc5f4a21ba7b17c78859c956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e765f735346a4b82a83307f7fcdd4a7a"
I20260926 23:35:30.415340 18037 raft_consensus.cc:3037] T 689810d6162242f4bd08089edb1c816a P e765f735346a4b82a83307f7fcdd4a7a [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:35:30.417776 17769 raft_consensus.cc:2454] T 689810d6162242f4bd08089edb1c816a P cd87529b018346e08173ffbb9b279490 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc898e10dc5f4a21ba7b17c78859c956 in term 1.
I20260926 23:35:30.418049 18037 raft_consensus.cc:2454] T 689810d6162242f4bd08089edb1c816a P e765f735346a4b82a83307f7fcdd4a7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc898e10dc5f4a21ba7b17c78859c956 in term 1.
W20260926 23:35:30.418376 18083 tablet.cc:2369] T 689810d6162242f4bd08089edb1c816a P e765f735346a4b82a83307f7fcdd4a7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:35:30.418630 17837 leader_election.cc:302] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: cd87529b018346e08173ffbb9b279490, dc898e10dc5f4a21ba7b17c78859c956, e765f735346a4b82a83307f7fcdd4a7a; no voters:
I20260926 23:35:30.419121 18107 raft_consensus.cc:2781] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:35:30.419785 18107 raft_consensus.cc:686] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956 [term 1 LEADER]: Becoming Leader. State: Replica: dc898e10dc5f4a21ba7b17c78859c956, State: Running, Role: LEADER
I20260926 23:35:30.420365 18107 consensus_queue.cc:237] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e765f735346a4b82a83307f7fcdd4a7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 } } peers { permanent_uuid: "dc898e10dc5f4a21ba7b17c78859c956" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46327 } } peers { permanent_uuid: "cd87529b018346e08173ffbb9b279490" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44757 } }
I20260926 23:35:30.426671 17623 catalog_manager.cc:5495] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc898e10dc5f4a21ba7b17c78859c956 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "dc898e10dc5f4a21ba7b17c78859c956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e765f735346a4b82a83307f7fcdd4a7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc898e10dc5f4a21ba7b17c78859c956" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd87529b018346e08173ffbb9b279490" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44757 } health_report { overall_health: UNKNOWN } } }
I20260926 23:35:30.459137 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:30.452560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36386 (local address 127.0.97.2:46327)
0926 23:35:30.452803 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:30.452809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:30.452869 (+ 60us) server_negotiation.cc:407] Connection header received
0926 23:35:30.453007 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:30.453012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:30.453127 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:30.453295 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:35:30.453811 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.454719 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.456412 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.456876 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.456999 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:30.457673 (+ 674us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:35:30.458397 (+ 724us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:35:30.458531 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:30.458763 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 23:35:30.458859 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 23:35:30.469600 18037 raft_consensus.cc:1261] T 689810d6162242f4bd08089edb1c816a P e765f735346a4b82a83307f7fcdd4a7a [term 1 FOLLOWER]: Refusing update from remote peer dc898e10dc5f4a21ba7b17c78859c956: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:35:30.469687 17769 raft_consensus.cc:1261] T 689810d6162242f4bd08089edb1c816a P cd87529b018346e08173ffbb9b279490 [term 1 FOLLOWER]: Refusing update from remote peer dc898e10dc5f4a21ba7b17c78859c956: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:35:30.470914 18112 consensus_queue.cc:1035] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e765f735346a4b82a83307f7fcdd4a7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:35:30.471366 18107 consensus_queue.cc:1035] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd87529b018346e08173ffbb9b279490" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:35:30.488636 18116 mvcc.cc:205] Tried to move back new op lower bound from 7333747631989448704 to 7333747631806943232. Current Snapshot: MvccSnapshot[applied={T|T < 7333747631989448704}]
I20260926 23:35:30.515976 17623 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:35:30.516283 17623 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:35:30.521698 17623 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d041948b9ef429982071cadaf94f23e: Sending DeleteTablet for 3 replicas of tablet 689810d6162242f4bd08089edb1c816a
I20260926 23:35:30.523437 18017 tablet_service.cc:1514] Processing DeleteTablet for tablet 689810d6162242f4bd08089edb1c816a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:35:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45286
I20260926 23:35:30.523566 17749 tablet_service.cc:1514] Processing DeleteTablet for tablet 689810d6162242f4bd08089edb1c816a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:35:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34022
I20260926 23:35:30.525146 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 17686
I20260926 23:35:30.525266 18126 tablet_replica.cc:330] T 689810d6162242f4bd08089edb1c816a P cd87529b018346e08173ffbb9b279490: stopping tablet replica
I20260926 23:35:30.525566 17883 tablet_service.cc:1514] Processing DeleteTablet for tablet 689810d6162242f4bd08089edb1c816a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:35:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36376
I20260926 23:35:30.525861 18126 raft_consensus.cc:2229] T 689810d6162242f4bd08089edb1c816a P cd87529b018346e08173ffbb9b279490 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:35:30.526503 18126 raft_consensus.cc:2258] T 689810d6162242f4bd08089edb1c816a P cd87529b018346e08173ffbb9b279490 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:35:30.527395 18125 tablet_replica.cc:330] T 689810d6162242f4bd08089edb1c816a P e765f735346a4b82a83307f7fcdd4a7a: stopping tablet replica
I20260926 23:35:30.527916 18125 raft_consensus.cc:2229] T 689810d6162242f4bd08089edb1c816a P e765f735346a4b82a83307f7fcdd4a7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:35:30.528498 18125 raft_consensus.cc:2258] T 689810d6162242f4bd08089edb1c816a P e765f735346a4b82a83307f7fcdd4a7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:35:30.531580 18125 ts_tablet_manager.cc:1905] T 689810d6162242f4bd08089edb1c816a P e765f735346a4b82a83307f7fcdd4a7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:35:30.534355 18128 tablet_replica.cc:330] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956: stopping tablet replica
I20260926 23:35:30.535081 18128 raft_consensus.cc:2229] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:35:30.535930 18128 raft_consensus.cc:2258] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 26 23:35:30 dist-test-slave-gwfx krb5kdc[17580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:35:30.539848 18125 ts_tablet_manager.cc:1918] T 689810d6162242f4bd08089edb1c816a P e765f735346a4b82a83307f7fcdd4a7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:35:30.540100 18125 log.cc:1196] T 689810d6162242f4bd08089edb1c816a P e765f735346a4b82a83307f7fcdd4a7a: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-2/wal/wals/689810d6162242f4bd08089edb1c816a
I20260926 23:35:30.540688 18125 ts_tablet_manager.cc:1939] T 689810d6162242f4bd08089edb1c816a P e765f735346a4b82a83307f7fcdd4a7a: Deleting consensus metadata
I20260926 23:35:30.542984 17610 catalog_manager.cc:4841] TS e765f735346a4b82a83307f7fcdd4a7a (127.0.97.3:34081): tablet 689810d6162242f4bd08089edb1c816a (table test-table [id=185e925618fd4cc08fd87adb1d14312c]) successfully deleted
I20260926 23:35:30.545526 18128 ts_tablet_manager.cc:1905] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:35:30.546154 18126 ts_tablet_manager.cc:1905] T 689810d6162242f4bd08089edb1c816a P cd87529b018346e08173ffbb9b279490: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:35:30.549896 18097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:30.526345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34026 (local address 127.0.97.1:44757)
0926 23:35:30.526536 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:30.526542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:30.529403 (+ 2861us) server_negotiation.cc:407] Connection header received
0926 23:35:30.529750 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:30.529757 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:30.529859 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:30.529940 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:30.531891 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.532912 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.534837 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.535487 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.535590 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:30.541486 (+ 5896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:30.541509 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:30.541531 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:35:30.541581 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:30.545841 (+ 4260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:30.546912 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:30.546922 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:30.546926 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:30.547149 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:30.547632 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:30.547637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:30.547640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:30.548190 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:30.548536 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:30.549178 (+ 642us) server_negotiation.cc:299] Negotiation successful
0926 23:35:30.549471 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 23:35:30.551635 17809 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:35:30.554100 18128 ts_tablet_manager.cc:1918] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:35:30.554323 18128 log.cc:1196] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790465400694755-388-0/minicluster-data/ts-1/wal/wals/689810d6162242f4bd08089edb1c816a
I20260926 23:35:30.554970 18128 ts_tablet_manager.cc:1939] T 689810d6162242f4bd08089edb1c816a P dc898e10dc5f4a21ba7b17c78859c956: Deleting consensus metadata
I20260926 23:35:30.556921 17610 catalog_manager.cc:4841] TS dc898e10dc5f4a21ba7b17c78859c956 (127.0.97.2:46327): tablet 689810d6162242f4bd08089edb1c816a (table test-table [id=185e925618fd4cc08fd87adb1d14312c]) successfully deleted
I20260926 23:35:30.649415 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 17686
I20260926 23:35:30.669947 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 17820
W20260926 23:35:30.669835 17612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:44757 (error 108)
I20260926 23:35:30.673447 18098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:30.672860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44757 (local address 127.0.0.1:34040)
0926 23:35:30.673087 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:30.673247 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260926 23:35:30.674528 17612 catalog_manager.cc:4584] TS cd87529b018346e08173ffbb9b279490 (127.0.97.1:44757): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 689810d6162242f4bd08089edb1c816a: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44757: connect: Connection refused (error 111)
Sep 26 23:35:30 dist-test-slave-gwfx krb5kdc[17580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:35:30.688506 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:30.671925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36396 (local address 127.0.97.2:46327)
0926 23:35:30.672124 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:30.672130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:30.672157 (+ 27us) server_negotiation.cc:407] Connection header received
0926 23:35:30.672268 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:30.672272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:30.672349 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:30.672469 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:30.674117 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.675283 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.676639 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.677097 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.677214 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:30.681553 (+ 4339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:30.681571 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:30.681590 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:35:30.681643 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:30.685141 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:30.685932 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:30.685947 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:30.685949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:30.686139 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:30.686615 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:30.686620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:30.686622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:30.687222 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:30.687583 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:30.687884 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 23:35:30.688123 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 23:35:30.689275 17943 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:35:30.728900 18098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:30.728174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44757 (local address 127.0.0.1:34056)
0926 23:35:30.728499 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:30.728669 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260926 23:35:30.730387 18098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:30.729904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44757 (local address 127.0.0.1:34072)
0926 23:35:30.730082 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:30.730219 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 23:35:30.791297 18098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:30.790679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44757 (local address 127.0.0.1:34080)
0926 23:35:30.790994 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:30.791098 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 23:35:30.792726 18098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:30.792322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44757 (local address 127.0.0.1:34090)
0926 23:35:30.792494 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:30.792583 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 23:35:30.813695 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 17820
I20260926 23:35:30.833901 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 17954
Sep 26 23:35:30 dist-test-slave-gwfx krb5kdc[17580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:35:30.851439 18096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:30.834898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45302 (local address 127.0.97.3:34081)
0926 23:35:30.835160 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:30.835166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:30.835190 (+ 24us) server_negotiation.cc:407] Connection header received
0926 23:35:30.835334 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:30.835339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:30.835439 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:30.835568 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:30.837139 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.837999 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.839667 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:30.840144 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:30.840264 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:30.844739 (+ 4475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:30.844758 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:30.844779 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:35:30.844822 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:30.847993 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:30.848956 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:30.848964 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:30.848966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:30.849138 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:30.849522 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:30.849525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:30.849528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:30.849961 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:30.850228 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:30.850580 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 23:35:30.850980 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 23:35:30.852298 18077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:35:30.873932 18098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:30.873372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44757 (local address 127.0.0.1:34102)
0926 23:35:30.873612 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:30.873710 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260926 23:35:30.875026 18098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:30.874592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44757 (local address 127.0.0.1:34116)
0926 23:35:30.874826 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:30.874901 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260926 23:35:30.945291 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 17954
I20260926 23:35:30.960904 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 17593
I20260926 23:35:30.962307 17655 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:35:31.092588 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 17593
2026-09-26T23:35:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4791 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T23:35:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:35:31Z Disabled control of system clock
I20260926 23:35:31.187601 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41167
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 23:35:31.613431 18137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:35:31.613881 18137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:35:31.613999 18137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:35:31.622768 18137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:35:31.622946 18137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:35:31.623013 18137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:35:31.623102 18137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:35:31.623155 18137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:35:31.633854 18137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41167
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:35:31.635654 18137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:35:31.637964 18137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:35:31.651779 18142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:31.651842 18145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:31.651829 18143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:35:31.653029 18137 server_base.cc:1034] running on GCE node
I20260926 23:35:31.654098 18137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:35:31.656121 18137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:35:31.657383 18137 hybrid_clock.cc:648] HybridClock initialized: now 1790465731657316 us; error 64 us; skew 500 ppm
I20260926 23:35:31.657828 18137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:35:31.661068 18137 webserver.cc:460] Webserver started at http://127.0.97.62:35151/ using document root <none> and password file <none>
I20260926 23:35:31.662009 18137 fs_manager.cc:359] Metadata directory not provided
I20260926 23:35:31.662139 18137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:35:31.662508 18137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:35:31.666738 18137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "c103125deedd4843b32505809ef7bea7"
format_stamp: "Formatted at 2026-09-26 23:35:31 on dist-test-slave-gwfx"
server_key: "25afa01d35cb1f33b3ce15d639bfad27"
server_key_iv: "684bfef170e1f92bd2e8c5484c06b1ab"
server_key_version: "encryptionkey@0"
I20260926 23:35:31.667848 18137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "c103125deedd4843b32505809ef7bea7"
format_stamp: "Formatted at 2026-09-26 23:35:31 on dist-test-slave-gwfx"
server_key: "25afa01d35cb1f33b3ce15d639bfad27"
server_key_iv: "684bfef170e1f92bd2e8c5484c06b1ab"
server_key_version: "encryptionkey@0"
I20260926 23:35:31.676132 18137 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.005s
I20260926 23:35:31.681083 18151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:31.683076 18137 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260926 23:35:31.683306 18137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "c103125deedd4843b32505809ef7bea7"
format_stamp: "Formatted at 2026-09-26 23:35:31 on dist-test-slave-gwfx"
server_key: "25afa01d35cb1f33b3ce15d639bfad27"
server_key_iv: "684bfef170e1f92bd2e8c5484c06b1ab"
server_key_version: "encryptionkey@0"
I20260926 23:35:31.683573 18137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:35:31.701186 18137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:35:31.702409 18137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:35:31.702792 18137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:35:31.723372 18137 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41167
I20260926 23:35:31.723404 18202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41167 every 8 connection(s)
I20260926 23:35:31.725132 18137 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:35:31.730090 18203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:35:31.730960 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 18137
I20260926 23:35:31.731150 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:35:31.731786 388 external_mini_cluster.cc:1428] Setting key 0f858a371fe1351999e43ffc1395870d
I20260926 23:35:31.745245 18203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c103125deedd4843b32505809ef7bea7: Bootstrap starting.
I20260926 23:35:31.747659 18206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:31.734231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42956 (local address 127.0.97.62:41167)
0926 23:35:31.735507 (+ 1276us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:31.735529 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:31.735587 (+ 58us) server_negotiation.cc:407] Connection header received
0926 23:35:31.737617 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:31.737651 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:31.738357 (+ 706us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:31.738937 (+ 580us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:31.739618 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:31.741018 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:31.742538 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:31.743412 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:31.743610 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:31.744927 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:31.744936 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:31.744956 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:35:31.745660 (+ 704us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:31.745757 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:35:31.745782 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:31.745904 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:31.746127 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 23:35:31.746509 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":446,"threads_started":1}
I20260926 23:35:31.749186 18203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c103125deedd4843b32505809ef7bea7: Neither blocks nor log segments found. Creating new log.
I20260926 23:35:31.750468 18203 log.cc:824] T 00000000000000000000000000000000 P c103125deedd4843b32505809ef7bea7: Log is configured to *not* fsync() on all Append() calls
I20260926 23:35:31.754137 18203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c103125deedd4843b32505809ef7bea7: No bootstrap required, opened a new log
I20260926 23:35:31.762929 18203 raft_consensus.cc:348] T 00000000000000000000000000000000 P c103125deedd4843b32505809ef7bea7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c103125deedd4843b32505809ef7bea7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41167 } }
I20260926 23:35:31.763480 18203 raft_consensus.cc:374] T 00000000000000000000000000000000 P c103125deedd4843b32505809ef7bea7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:35:31.763571 18203 raft_consensus.cc:729] T 00000000000000000000000000000000 P c103125deedd4843b32505809ef7bea7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c103125deedd4843b32505809ef7bea7, State: Initialized, Role: FOLLOWER
I20260926 23:35:31.764259 18203 consensus_queue.cc:260] T 00000000000000000000000000000000 P c103125deedd4843b32505809ef7bea7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c103125deedd4843b32505809ef7bea7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41167 } }
I20260926 23:35:31.764559 18203 raft_consensus.cc:388] T 00000000000000000000000000000000 P c103125deedd4843b32505809ef7bea7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:35:31.764735 18203 raft_consensus.cc:482] T 00000000000000000000000000000000 P c103125deedd4843b32505809ef7bea7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:35:31.764930 18203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c103125deedd4843b32505809ef7bea7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:35:31.767740 18203 raft_consensus.cc:504] T 00000000000000000000000000000000 P c103125deedd4843b32505809ef7bea7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c103125deedd4843b32505809ef7bea7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41167 } }
I20260926 23:35:31.768458 18203 leader_election.cc:302] T 00000000000000000000000000000000 P c103125deedd4843b32505809ef7bea7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c103125deedd4843b32505809ef7bea7; no voters:
I20260926 23:35:31.769133 18203 leader_election.cc:288] T 00000000000000000000000000000000 P c103125deedd4843b32505809ef7bea7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:35:31.769301 18208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c103125deedd4843b32505809ef7bea7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:35:31.770047 18208 raft_consensus.cc:686] T 00000000000000000000000000000000 P c103125deedd4843b32505809ef7bea7 [term 1 LEADER]: Becoming Leader. State: Replica: c103125deedd4843b32505809ef7bea7, State: Running, Role: LEADER
I20260926 23:35:31.770645 18208 consensus_queue.cc:237] T 00000000000000000000000000000000 P c103125deedd4843b32505809ef7bea7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c103125deedd4843b32505809ef7bea7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41167 } }
I20260926 23:35:31.771566 18203 sys_catalog.cc:564] T 00000000000000000000000000000000 P c103125deedd4843b32505809ef7bea7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:35:31.774286 18210 sys_catalog.cc:455] T 00000000000000000000000000000000 P c103125deedd4843b32505809ef7bea7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c103125deedd4843b32505809ef7bea7. Latest consensus state: current_term: 1 leader_uuid: "c103125deedd4843b32505809ef7bea7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c103125deedd4843b32505809ef7bea7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41167 } } }
I20260926 23:35:31.774710 18210 sys_catalog.cc:458] T 00000000000000000000000000000000 P c103125deedd4843b32505809ef7bea7 [sys.catalog]: This master's current role is: LEADER
I20260926 23:35:31.775115 18209 sys_catalog.cc:455] T 00000000000000000000000000000000 P c103125deedd4843b32505809ef7bea7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c103125deedd4843b32505809ef7bea7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c103125deedd4843b32505809ef7bea7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41167 } } }
I20260926 23:35:31.775408 18209 sys_catalog.cc:458] T 00000000000000000000000000000000 P c103125deedd4843b32505809ef7bea7 [sys.catalog]: This master's current role is: LEADER
I20260926 23:35:31.776918 18214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:35:31.783553 18214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:35:31.795452 18214 catalog_manager.cc:1347] Generated new cluster ID: eda22c5ff36f40cf8494b3a72e23c60b
I20260926 23:35:31.795606 18214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:35:31.818418 18214 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:35:31.820307 18214 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:35:31.832262 18214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c103125deedd4843b32505809ef7bea7: Generated new TSK 0
I20260926 23:35:31.833515 18214 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:35:31.851177 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41167
--builtin_ntp_servers=127.0.97.20:34305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 23:35:32.281064 18227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:35:32.281492 18227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:35:32.281620 18227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:35:32.291321 18227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:35:32.291508 18227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:35:32.291669 18227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:35:32.303673 18227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:35:32.305619 18227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:35:32.308763 18227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:35:32.329213 18232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:32.329154 18233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:32.329166 18235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:35:32.329875 18227 server_base.cc:1034] running on GCE node
I20260926 23:35:32.331113 18227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:35:32.332525 18227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:35:32.333874 18227 hybrid_clock.cc:648] HybridClock initialized: now 1790465732333812 us; error 66 us; skew 500 ppm
I20260926 23:35:32.334414 18227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:35:32.337543 18227 webserver.cc:460] Webserver started at http://127.0.97.1:35081/ using document root <none> and password file <none>
I20260926 23:35:32.338469 18227 fs_manager.cc:359] Metadata directory not provided
I20260926 23:35:32.338615 18227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:35:32.339046 18227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:35:32.343178 18227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-0/data/instance:
uuid: "35446c828eae4da29f41dbf9d80cb9a3"
format_stamp: "Formatted at 2026-09-26 23:35:32 on dist-test-slave-gwfx"
server_key: "68b2347da06b7ac5a07b70e86a267f3f"
server_key_iv: "1f0bc89304921dec7c95a1c4aa9fe50e"
server_key_version: "encryptionkey@0"
I20260926 23:35:32.344414 18227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-0/wal/instance:
uuid: "35446c828eae4da29f41dbf9d80cb9a3"
format_stamp: "Formatted at 2026-09-26 23:35:32 on dist-test-slave-gwfx"
server_key: "68b2347da06b7ac5a07b70e86a267f3f"
server_key_iv: "1f0bc89304921dec7c95a1c4aa9fe50e"
server_key_version: "encryptionkey@0"
I20260926 23:35:32.351922 18227 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260926 23:35:32.357116 18241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:32.359138 18227 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 23:35:32.359370 18227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "35446c828eae4da29f41dbf9d80cb9a3"
format_stamp: "Formatted at 2026-09-26 23:35:32 on dist-test-slave-gwfx"
server_key: "68b2347da06b7ac5a07b70e86a267f3f"
server_key_iv: "1f0bc89304921dec7c95a1c4aa9fe50e"
server_key_version: "encryptionkey@0"
I20260926 23:35:32.359656 18227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:35:32.373307 18227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:35:32.374450 18227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:35:32.374843 18227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:35:32.376142 18227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:35:32.378093 18227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:35:32.378211 18227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:32.378329 18227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:35:32.378400 18227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:32.414672 18227 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46349
I20260926 23:35:32.414744 18353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46349 every 8 connection(s)
I20260926 23:35:32.416428 18227 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:35:32.419164 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 18227
I20260926 23:35:32.419370 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:35:32.419986 388 external_mini_cluster.cc:1428] Setting key 42981e578a4150ef8a515ac2400c5515
I20260926 23:35:32.424170 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41167
--builtin_ntp_servers=127.0.97.20:34305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 23:35:32.433465 18356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:32.419652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58013 (local address 127.0.97.62:41167)
0926 23:35:32.420311 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:32.420316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:32.422048 (+ 1732us) server_negotiation.cc:407] Connection header received
0926 23:35:32.424752 (+ 2704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:32.424761 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:32.424886 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:32.425034 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:32.426763 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:32.427830 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:32.429878 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:32.430512 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:32.430654 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:32.432405 (+ 1751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:32.432411 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:32.432417 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:35:32.432448 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:32.432508 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:35:32.432528 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:32.432643 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:32.432930 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 23:35:32.433063 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":354,"threads_started":1}
I20260926 23:35:32.435292 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:32.420396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41167 (local address 127.0.97.1:58013)
0926 23:35:32.421808 (+ 1412us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:32.421880 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:32.424241 (+ 2361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:32.425445 (+ 1204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:32.425458 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:32.425633 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:35:32.426475 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:32.426499 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:32.428058 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:32.428064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:32.429691 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:32.429709 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:32.430809 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:32.430816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:32.431079 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:32.431918 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:35:32.431921 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:35:32.431983 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:35:32.432145 (+ 162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:35:32.432672 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:35:32.432684 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:35:32.432692 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 23:35:32.432910 (+ 218us) client_negotiation.cc:240] Negotiation successful
0926 23:35:32.433369 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":425,"threads_started":1}
I20260926 23:35:32.437505 18354 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41167
I20260926 23:35:32.437920 18354 heartbeater.cc:464] Registering TS with master...
I20260926 23:35:32.439110 18354 heartbeater.cc:511] Master 127.0.97.62:41167 requested a full tablet report, sending...
I20260926 23:35:32.442034 18168 ts_manager.cc:194] Registered new tserver with Master: 35446c828eae4da29f41dbf9d80cb9a3 (127.0.97.1:46349)
I20260926 23:35:32.443919 18168 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:58013
W20260926 23:35:32.821106 18358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:35:32.821481 18358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:35:32.821602 18358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:35:32.829677 18358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:35:32.829810 18358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:35:32.829972 18358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:35:32.840883 18358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:35:32.842665 18358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:35:32.845007 18358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:35:32.860982 18364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:32.860872 18366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:32.860838 18363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:35:32.862020 18358 server_base.cc:1034] running on GCE node
I20260926 23:35:32.862924 18358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:35:32.864349 18358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:35:32.865597 18358 hybrid_clock.cc:648] HybridClock initialized: now 1790465732865538 us; error 60 us; skew 500 ppm
I20260926 23:35:32.866019 18358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:35:32.869117 18358 webserver.cc:460] Webserver started at http://127.0.97.2:45779/ using document root <none> and password file <none>
I20260926 23:35:32.870069 18358 fs_manager.cc:359] Metadata directory not provided
I20260926 23:35:32.870180 18358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:35:32.870575 18358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:35:32.874434 18358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-1/data/instance:
uuid: "413bca71aa954f548861a1ead8b76ec4"
format_stamp: "Formatted at 2026-09-26 23:35:32 on dist-test-slave-gwfx"
server_key: "7ba80deb67243e80d6f6994f467bdd9c"
server_key_iv: "14af7046890959a4ecc24c2a6fb1c54b"
server_key_version: "encryptionkey@0"
I20260926 23:35:32.875448 18358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-1/wal/instance:
uuid: "413bca71aa954f548861a1ead8b76ec4"
format_stamp: "Formatted at 2026-09-26 23:35:32 on dist-test-slave-gwfx"
server_key: "7ba80deb67243e80d6f6994f467bdd9c"
server_key_iv: "14af7046890959a4ecc24c2a6fb1c54b"
server_key_version: "encryptionkey@0"
I20260926 23:35:32.882005 18358 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260926 23:35:32.886605 18372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:32.888656 18358 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:35:32.888866 18358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "413bca71aa954f548861a1ead8b76ec4"
format_stamp: "Formatted at 2026-09-26 23:35:32 on dist-test-slave-gwfx"
server_key: "7ba80deb67243e80d6f6994f467bdd9c"
server_key_iv: "14af7046890959a4ecc24c2a6fb1c54b"
server_key_version: "encryptionkey@0"
I20260926 23:35:32.889119 18358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:35:32.921097 18358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:35:32.922207 18358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:35:32.922611 18358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:35:32.924070 18358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:35:32.926051 18358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:35:32.926159 18358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:32.926259 18358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:35:32.926316 18358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:32.962385 18358 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39779
I20260926 23:35:32.962461 18484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39779 every 8 connection(s)
I20260926 23:35:32.964047 18358 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:35:32.970432 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 18358
I20260926 23:35:32.970646 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:35:32.971334 388 external_mini_cluster.cc:1428] Setting key 518227c14d0e14aafcdcb3656c51f7b6
I20260926 23:35:32.975551 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41167
--builtin_ntp_servers=127.0.97.20:34305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 23:35:32.981868 18487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:32.967255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37555 (local address 127.0.97.62:41167)
0926 23:35:32.967791 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:32.967799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:32.969425 (+ 1626us) server_negotiation.cc:407] Connection header received
0926 23:35:32.972461 (+ 3036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:32.972468 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:32.972585 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:32.972733 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:32.974590 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:32.975809 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:32.978019 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:32.978759 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:32.978908 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:32.980805 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:32.980811 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:32.980816 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:35:32.980846 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:32.980906 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:35:32.980923 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:32.981021 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:32.981309 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 23:35:32.981456 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":252,"threads_started":1}
I20260926 23:35:32.983306 18488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:32.968071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41167 (local address 127.0.97.2:37555)
0926 23:35:32.969215 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:32.969274 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:32.971936 (+ 2662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:32.973148 (+ 1212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:32.973166 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:32.973337 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:35:32.974361 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:32.974389 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:32.976073 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:32.976079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:32.977773 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:32.977797 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:32.979063 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:32.979072 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:32.979353 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:32.980325 (+ 972us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:35:32.980328 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:35:32.980389 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:35:32.980527 (+ 138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:35:32.981057 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:35:32.981069 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:35:32.981077 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 23:35:32.981285 (+ 208us) client_negotiation.cc:240] Negotiation successful
0926 23:35:32.981794 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":337,"threads_started":1}
I20260926 23:35:32.985158 18485 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41167
I20260926 23:35:32.985616 18485 heartbeater.cc:464] Registering TS with master...
I20260926 23:35:32.986884 18485 heartbeater.cc:511] Master 127.0.97.62:41167 requested a full tablet report, sending...
I20260926 23:35:32.988867 18168 ts_manager.cc:194] Registered new tserver with Master: 413bca71aa954f548861a1ead8b76ec4 (127.0.97.2:39779)
I20260926 23:35:32.990090 18168 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:37555
W20260926 23:35:33.391723 18489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:35:33.392140 18489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:35:33.392253 18489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:35:33.401005 18489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:35:33.401156 18489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:35:33.401289 18489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:35:33.411760 18489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:35:33.413435 18489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:35:33.416273 18489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:35:33.433100 18494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:33.434345 18495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:33.434511 18497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:35:33.435851 18489 server_base.cc:1034] running on GCE node
I20260926 23:35:33.436794 18489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:35:33.438268 18489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:35:33.439554 18489 hybrid_clock.cc:648] HybridClock initialized: now 1790465733439450 us; error 91 us; skew 500 ppm
I20260926 23:35:33.440006 18489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:35:33.443205 18489 webserver.cc:460] Webserver started at http://127.0.97.3:42215/ using document root <none> and password file <none>
I20260926 23:35:33.444242 18489 fs_manager.cc:359] Metadata directory not provided
I20260926 23:35:33.444463 18489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:35:33.444938 18489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:35:33.448529 18354 heartbeater.cc:503] Master 127.0.97.62:41167 was elected leader, sending a full tablet report...
I20260926 23:35:33.449162 18489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-2/data/instance:
uuid: "2ecd07944d5e4272aca25d935174886d"
format_stamp: "Formatted at 2026-09-26 23:35:33 on dist-test-slave-gwfx"
server_key: "21352875751790aea225161cfce8c588"
server_key_iv: "7b36b15ff40b68c838736cdf5fe11d25"
server_key_version: "encryptionkey@0"
I20260926 23:35:33.450361 18489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2ecd07944d5e4272aca25d935174886d"
format_stamp: "Formatted at 2026-09-26 23:35:33 on dist-test-slave-gwfx"
server_key: "21352875751790aea225161cfce8c588"
server_key_iv: "7b36b15ff40b68c838736cdf5fe11d25"
server_key_version: "encryptionkey@0"
I20260926 23:35:33.457651 18489 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260926 23:35:33.462962 18503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:33.465241 18489 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260926 23:35:33.465493 18489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "2ecd07944d5e4272aca25d935174886d"
format_stamp: "Formatted at 2026-09-26 23:35:33 on dist-test-slave-gwfx"
server_key: "21352875751790aea225161cfce8c588"
server_key_iv: "7b36b15ff40b68c838736cdf5fe11d25"
server_key_version: "encryptionkey@0"
I20260926 23:35:33.465785 18489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:35:33.490299 18489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:35:33.491511 18489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:35:33.491895 18489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:35:33.493510 18489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:35:33.496272 18489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:35:33.496424 18489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:33.496565 18489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:35:33.496650 18489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:33.535804 18489 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39285
I20260926 23:35:33.536083 18615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39285 every 8 connection(s)
I20260926 23:35:33.537415 18489 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:35:33.541863 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 18489
I20260926 23:35:33.542083 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:35:33.542579 388 external_mini_cluster.cc:1428] Setting key 0b1f025f5f3dba84880f3c36d6c2efa2
I20260926 23:35:33.554067 18618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:33.540726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58327 (local address 127.0.97.62:41167)
0926 23:35:33.541293 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:33.541299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:33.543019 (+ 1720us) server_negotiation.cc:407] Connection header received
0926 23:35:33.545316 (+ 2297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:33.545322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:33.545425 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:33.545587 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:33.547299 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.548333 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.550314 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.551194 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.551370 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:33.553079 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:33.553085 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:33.553089 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:35:33.553115 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:33.553180 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:35:33.553196 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:33.553299 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:33.553570 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 23:35:33.553694 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":280,"threads_started":1}
I20260926 23:35:33.555338 18619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:33.541353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41167 (local address 127.0.97.3:58327)
0926 23:35:33.542745 (+ 1392us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:33.542811 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:33.545005 (+ 2194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:33.545917 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:33.545928 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:33.546101 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:35:33.547053 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:33.547078 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.548531 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.548536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:33.550103 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:33.550120 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.551479 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.551485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:33.551752 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:33.552639 (+ 887us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:35:33.552641 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:35:33.552707 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:35:33.552820 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:35:33.553302 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:35:33.553308 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:35:33.553315 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 23:35:33.553494 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 23:35:33.553918 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":403,"threads_started":1}
I20260926 23:35:33.557354 18616 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41167
I20260926 23:35:33.557758 18616 heartbeater.cc:464] Registering TS with master...
I20260926 23:35:33.558866 18616 heartbeater.cc:511] Master 127.0.97.62:41167 requested a full tablet report, sending...
I20260926 23:35:33.560861 18168 ts_manager.cc:194] Registered new tserver with Master: 2ecd07944d5e4272aca25d935174886d (127.0.97.3:39285)
I20260926 23:35:33.561884 18168 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:58327
I20260926 23:35:33.573496 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:35:33.586589 18618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:33.578908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42970 (local address 127.0.97.62:41167)
0926 23:35:33.579082 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:33.579086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:33.579709 (+ 623us) server_negotiation.cc:407] Connection header received
0926 23:35:33.579946 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:33.579998 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:33.580105 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:33.580224 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:33.580851 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.581714 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.583508 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.584105 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.584260 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:33.585647 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:33.585653 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:33.585657 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:35:33.585684 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:33.585729 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:35:33.585744 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:33.585852 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:33.586046 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 23:35:33.586165 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 23:35:33.591812 18168 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:35:33.599962 18168 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:35:33.632807 18629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:33.617003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39285 (local address 127.0.0.1:41870)
0926 23:35:33.618964 (+ 1961us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:33.619031 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:33.619479 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:33.621152 (+ 1673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:33.621164 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:33.621221 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:35:33.621862 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:33.621896 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.624381 (+ 2485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.624389 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:33.628175 (+ 3786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:33.628216 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.631243 (+ 3027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.631251 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:33.631815 (+ 564us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:33.631899 (+ 84us) client_negotiation.cc:769] Sending connection context
0926 23:35:33.632091 (+ 192us) client_negotiation.cc:240] Negotiation successful
0926 23:35:33.632364 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1487,"thread_start_us":1196,"threads_started":1}
I20260926 23:35:33.634268 18630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:33.617232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41870 (local address 127.0.97.3:39285)
0926 23:35:33.618061 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:33.618070 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:33.619734 (+ 1664us) server_negotiation.cc:407] Connection header received
0926 23:35:33.619846 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:33.619854 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:33.620177 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:33.620382 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:35:33.622294 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.624107 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.628599 (+ 4492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.630374 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.630547 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:33.630954 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:33.633563 (+ 2609us) server_negotiation.cc:299] Negotiation successful
0926 23:35:33.633818 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":464,"threads_started":1}
I20260926 23:35:33.641919 18629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:33.619716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39779 (local address 127.0.0.1:40224)
0926 23:35:33.633157 (+ 13441us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:33.633180 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:33.633326 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:33.635797 (+ 2471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:33.635802 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:33.635836 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:35:33.636080 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:33.636093 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.637750 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.637756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:33.639688 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:33.639705 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.641089 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.641095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:33.641298 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:33.641332 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 23:35:33.641425 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 23:35:33.641584 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13289,"mutex_wait_us":185,"thread_start_us":15498,"threads_started":1}
I20260926 23:35:33.641225 18551 tablet_service.cc:1467] Processing CreateTablet for tablet a7c8f59b32e6492ab7b2a8bd13046f5c (DEFAULT_TABLE table=test-table [id=5351769161b244d8a5c2ea8da460dc47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:35:33.642323 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:33.619814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40224 (local address 127.0.97.2:39779)
0926 23:35:33.635089 (+ 15275us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:33.635098 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:33.635141 (+ 43us) server_negotiation.cc:407] Connection header received
0926 23:35:33.635278 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:33.635287 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:33.635565 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:33.635789 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:35:33.636240 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.637516 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.639877 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.640894 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.641030 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:33.641312 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:33.641519 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 23:35:33.641710 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15088,"thread_start_us":14942,"threads_started":1}
I20260926 23:35:33.643308 18631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:33.618510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46349 (local address 127.0.0.1:51300)
0926 23:35:33.629107 (+ 10597us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:33.629145 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:33.629379 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:33.634196 (+ 4817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:33.634204 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:33.634248 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:35:33.634828 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:33.634846 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.637429 (+ 2583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.637437 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:33.640167 (+ 2730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:33.640192 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.642460 (+ 2268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.642467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:33.642669 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:33.642732 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 23:35:33.642810 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 23:35:33.642961 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10374,"mutex_wait_us":71,"thread_start_us":9957,"threads_started":1}
I20260926 23:35:33.643559 18551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7c8f59b32e6492ab7b2a8bd13046f5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:35:33.643604 18632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:33.618837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51300 (local address 127.0.97.1:46349)
0926 23:35:33.633141 (+ 14304us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:33.633166 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:33.633212 (+ 46us) server_negotiation.cc:407] Connection header received
0926 23:35:33.633365 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:33.633376 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:33.633808 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:33.635149 (+ 1341us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:35:33.635484 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.637207 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.640483 (+ 3276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.642140 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.642400 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:33.642789 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:33.642934 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 23:35:33.643152 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14037,"thread_start_us":14238,"threads_started":1}
I20260926 23:35:33.648479 18420 tablet_service.cc:1467] Processing CreateTablet for tablet a7c8f59b32e6492ab7b2a8bd13046f5c (DEFAULT_TABLE table=test-table [id=5351769161b244d8a5c2ea8da460dc47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:35:33.648476 18289 tablet_service.cc:1467] Processing CreateTablet for tablet a7c8f59b32e6492ab7b2a8bd13046f5c (DEFAULT_TABLE table=test-table [id=5351769161b244d8a5c2ea8da460dc47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:35:33.650496 18289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7c8f59b32e6492ab7b2a8bd13046f5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:35:33.650523 18420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7c8f59b32e6492ab7b2a8bd13046f5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:35:33.662823 18635 tablet_bootstrap.cc:492] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d: Bootstrap starting.
I20260926 23:35:33.666481 18635 tablet_bootstrap.cc:654] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d: Neither blocks nor log segments found. Creating new log.
I20260926 23:35:33.667690 18635 log.cc:824] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d: Log is configured to *not* fsync() on all Append() calls
I20260926 23:35:33.670773 18637 tablet_bootstrap.cc:492] T a7c8f59b32e6492ab7b2a8bd13046f5c P 35446c828eae4da29f41dbf9d80cb9a3: Bootstrap starting.
I20260926 23:35:33.673292 18638 tablet_bootstrap.cc:492] T a7c8f59b32e6492ab7b2a8bd13046f5c P 413bca71aa954f548861a1ead8b76ec4: Bootstrap starting.
I20260926 23:35:33.676283 18635 tablet_bootstrap.cc:492] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d: No bootstrap required, opened a new log
I20260926 23:35:33.676245 18637 tablet_bootstrap.cc:654] T a7c8f59b32e6492ab7b2a8bd13046f5c P 35446c828eae4da29f41dbf9d80cb9a3: Neither blocks nor log segments found. Creating new log.
I20260926 23:35:33.676608 18635 ts_tablet_manager.cc:1397] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20260926 23:35:33.677546 18638 tablet_bootstrap.cc:654] T a7c8f59b32e6492ab7b2a8bd13046f5c P 413bca71aa954f548861a1ead8b76ec4: Neither blocks nor log segments found. Creating new log.
I20260926 23:35:33.677660 18637 log.cc:824] T a7c8f59b32e6492ab7b2a8bd13046f5c P 35446c828eae4da29f41dbf9d80cb9a3: Log is configured to *not* fsync() on all Append() calls
I20260926 23:35:33.679227 18638 log.cc:824] T a7c8f59b32e6492ab7b2a8bd13046f5c P 413bca71aa954f548861a1ead8b76ec4: Log is configured to *not* fsync() on all Append() calls
I20260926 23:35:33.681748 18637 tablet_bootstrap.cc:492] T a7c8f59b32e6492ab7b2a8bd13046f5c P 35446c828eae4da29f41dbf9d80cb9a3: No bootstrap required, opened a new log
I20260926 23:35:33.682179 18637 ts_tablet_manager.cc:1397] T a7c8f59b32e6492ab7b2a8bd13046f5c P 35446c828eae4da29f41dbf9d80cb9a3: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260926 23:35:33.682777 18638 tablet_bootstrap.cc:492] T a7c8f59b32e6492ab7b2a8bd13046f5c P 413bca71aa954f548861a1ead8b76ec4: No bootstrap required, opened a new log
I20260926 23:35:33.683138 18638 ts_tablet_manager.cc:1397] T a7c8f59b32e6492ab7b2a8bd13046f5c P 413bca71aa954f548861a1ead8b76ec4: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260926 23:35:33.687163 18635 raft_consensus.cc:348] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecd07944d5e4272aca25d935174886d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39285 } } peers { permanent_uuid: "413bca71aa954f548861a1ead8b76ec4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39779 } } peers { permanent_uuid: "35446c828eae4da29f41dbf9d80cb9a3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46349 } }
I20260926 23:35:33.687698 18635 raft_consensus.cc:374] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:35:33.687799 18635 raft_consensus.cc:729] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ecd07944d5e4272aca25d935174886d, State: Initialized, Role: FOLLOWER
I20260926 23:35:33.688542 18635 consensus_queue.cc:260] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecd07944d5e4272aca25d935174886d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39285 } } peers { permanent_uuid: "413bca71aa954f548861a1ead8b76ec4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39779 } } peers { permanent_uuid: "35446c828eae4da29f41dbf9d80cb9a3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46349 } }
I20260926 23:35:33.690047 18616 heartbeater.cc:503] Master 127.0.97.62:41167 was elected leader, sending a full tablet report...
I20260926 23:35:33.690655 18635 ts_tablet_manager.cc:1428] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260926 23:35:33.692523 18637 raft_consensus.cc:348] T a7c8f59b32e6492ab7b2a8bd13046f5c P 35446c828eae4da29f41dbf9d80cb9a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecd07944d5e4272aca25d935174886d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39285 } } peers { permanent_uuid: "413bca71aa954f548861a1ead8b76ec4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39779 } } peers { permanent_uuid: "35446c828eae4da29f41dbf9d80cb9a3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46349 } }
I20260926 23:35:33.693099 18637 raft_consensus.cc:374] T a7c8f59b32e6492ab7b2a8bd13046f5c P 35446c828eae4da29f41dbf9d80cb9a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:35:33.693216 18637 raft_consensus.cc:729] T a7c8f59b32e6492ab7b2a8bd13046f5c P 35446c828eae4da29f41dbf9d80cb9a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35446c828eae4da29f41dbf9d80cb9a3, State: Initialized, Role: FOLLOWER
I20260926 23:35:33.693522 18638 raft_consensus.cc:348] T a7c8f59b32e6492ab7b2a8bd13046f5c P 413bca71aa954f548861a1ead8b76ec4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecd07944d5e4272aca25d935174886d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39285 } } peers { permanent_uuid: "413bca71aa954f548861a1ead8b76ec4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39779 } } peers { permanent_uuid: "35446c828eae4da29f41dbf9d80cb9a3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46349 } }
I20260926 23:35:33.694145 18638 raft_consensus.cc:374] T a7c8f59b32e6492ab7b2a8bd13046f5c P 413bca71aa954f548861a1ead8b76ec4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:35:33.694281 18638 raft_consensus.cc:729] T a7c8f59b32e6492ab7b2a8bd13046f5c P 413bca71aa954f548861a1ead8b76ec4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 413bca71aa954f548861a1ead8b76ec4, State: Initialized, Role: FOLLOWER
I20260926 23:35:33.694598 18637 consensus_queue.cc:260] T a7c8f59b32e6492ab7b2a8bd13046f5c P 35446c828eae4da29f41dbf9d80cb9a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecd07944d5e4272aca25d935174886d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39285 } } peers { permanent_uuid: "413bca71aa954f548861a1ead8b76ec4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39779 } } peers { permanent_uuid: "35446c828eae4da29f41dbf9d80cb9a3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46349 } }
I20260926 23:35:33.695115 18638 consensus_queue.cc:260] T a7c8f59b32e6492ab7b2a8bd13046f5c P 413bca71aa954f548861a1ead8b76ec4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecd07944d5e4272aca25d935174886d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39285 } } peers { permanent_uuid: "413bca71aa954f548861a1ead8b76ec4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39779 } } peers { permanent_uuid: "35446c828eae4da29f41dbf9d80cb9a3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46349 } }
I20260926 23:35:33.696882 18637 ts_tablet_manager.cc:1428] T a7c8f59b32e6492ab7b2a8bd13046f5c P 35446c828eae4da29f41dbf9d80cb9a3: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260926 23:35:33.698776 18485 heartbeater.cc:503] Master 127.0.97.62:41167 was elected leader, sending a full tablet report...
I20260926 23:35:33.699436 18638 ts_tablet_manager.cc:1428] T a7c8f59b32e6492ab7b2a8bd13046f5c P 413bca71aa954f548861a1ead8b76ec4: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
W20260926 23:35:33.716512 18486 tablet.cc:2369] T a7c8f59b32e6492ab7b2a8bd13046f5c P 413bca71aa954f548861a1ead8b76ec4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:35:33.789615 18617 tablet.cc:2369] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:35:33.895515 18641 raft_consensus.cc:482] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:35:33.895919 18641 raft_consensus.cc:504] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecd07944d5e4272aca25d935174886d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39285 } } peers { permanent_uuid: "413bca71aa954f548861a1ead8b76ec4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39779 } } peers { permanent_uuid: "35446c828eae4da29f41dbf9d80cb9a3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46349 } }
I20260926 23:35:33.898048 18641 leader_election.cc:288] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 413bca71aa954f548861a1ead8b76ec4 (127.0.97.2:39779), 35446c828eae4da29f41dbf9d80cb9a3 (127.0.97.1:46349)
I20260926 23:35:33.905305 18642 raft_consensus.cc:482] T a7c8f59b32e6492ab7b2a8bd13046f5c P 35446c828eae4da29f41dbf9d80cb9a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:35:33.906033 18642 raft_consensus.cc:504] T a7c8f59b32e6492ab7b2a8bd13046f5c P 35446c828eae4da29f41dbf9d80cb9a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecd07944d5e4272aca25d935174886d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39285 } } peers { permanent_uuid: "413bca71aa954f548861a1ead8b76ec4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39779 } } peers { permanent_uuid: "35446c828eae4da29f41dbf9d80cb9a3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46349 } }
I20260926 23:35:33.906719 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:33.898801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46377 (local address 127.0.97.2:39779)
0926 23:35:33.899152 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:33.899157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:33.899183 (+ 26us) server_negotiation.cc:407] Connection header received
0926 23:35:33.899414 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:33.899422 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:33.899551 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:33.899722 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:35:33.900350 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.901376 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.903723 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.904825 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.904971 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:33.905083 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:33.906119 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0926 23:35:33.906285 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260926 23:35:33.906869 18619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:33.898468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39779 (local address 127.0.97.3:46377)
0926 23:35:33.898976 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:33.899004 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:33.899165 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:33.899866 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:33.899872 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:33.899924 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:35:33.900147 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:33.900161 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.901661 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.901668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:33.903476 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:33.903497 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.905051 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.905059 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:33.905271 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:33.905805 (+ 534us) client_negotiation.cc:769] Sending connection context
0926 23:35:33.905942 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 23:35:33.906146 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359}
I20260926 23:35:33.908285 18644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:33.898541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46349 (local address 127.0.97.3:45089)
0926 23:35:33.899607 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:33.899645 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:33.899809 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:33.900341 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:33.900347 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:33.900380 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:35:33.900791 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:33.900805 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.902338 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.902344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:33.904281 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:33.904300 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.907418 (+ 3118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.907425 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:33.907585 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:33.907623 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 23:35:33.907796 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 23:35:33.907972 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":908,"thread_start_us":620,"threads_started":1}
I20260926 23:35:33.908310 18632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:33.898803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45089 (local address 127.0.97.1:46349)
0926 23:35:33.899098 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:33.899106 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:33.899802 (+ 696us) server_negotiation.cc:407] Connection header received
0926 23:35:33.899995 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:33.900001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:33.900107 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:33.900279 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:35:33.901050 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.902123 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.904865 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.907123 (+ 2258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.907266 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:33.907403 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:33.907879 (+ 476us) server_negotiation.cc:299] Negotiation successful
0926 23:35:33.908054 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260926 23:35:33.908318 18440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c8f59b32e6492ab7b2a8bd13046f5c" candidate_uuid: "2ecd07944d5e4272aca25d935174886d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "413bca71aa954f548861a1ead8b76ec4" is_pre_election: true
I20260926 23:35:33.909017 18440 raft_consensus.cc:2454] T a7c8f59b32e6492ab7b2a8bd13046f5c P 413bca71aa954f548861a1ead8b76ec4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ecd07944d5e4272aca25d935174886d in term 0.
I20260926 23:35:33.909891 18504 leader_election.cc:302] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecd07944d5e4272aca25d935174886d, 413bca71aa954f548861a1ead8b76ec4; no voters:
I20260926 23:35:33.910014 18642 leader_election.cc:288] T a7c8f59b32e6492ab7b2a8bd13046f5c P 35446c828eae4da29f41dbf9d80cb9a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ecd07944d5e4272aca25d935174886d (127.0.97.3:39285), 413bca71aa954f548861a1ead8b76ec4 (127.0.97.2:39779)
I20260926 23:35:33.910383 18641 raft_consensus.cc:2781] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:35:33.910516 18641 raft_consensus.cc:482] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:35:33.910612 18641 raft_consensus.cc:3037] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:35:33.910293 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c8f59b32e6492ab7b2a8bd13046f5c" candidate_uuid: "2ecd07944d5e4272aca25d935174886d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35446c828eae4da29f41dbf9d80cb9a3" is_pre_election: true
I20260926 23:35:33.910989 18309 raft_consensus.cc:2454] T a7c8f59b32e6492ab7b2a8bd13046f5c P 35446c828eae4da29f41dbf9d80cb9a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ecd07944d5e4272aca25d935174886d in term 0.
I20260926 23:35:33.913875 18641 raft_consensus.cc:504] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecd07944d5e4272aca25d935174886d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39285 } } peers { permanent_uuid: "413bca71aa954f548861a1ead8b76ec4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39779 } } peers { permanent_uuid: "35446c828eae4da29f41dbf9d80cb9a3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46349 } }
I20260926 23:35:33.914738 18641 leader_election.cc:288] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d [CANDIDATE]: Term 1 election: Requested vote from peers 413bca71aa954f548861a1ead8b76ec4 (127.0.97.2:39779), 35446c828eae4da29f41dbf9d80cb9a3 (127.0.97.1:46349)
I20260926 23:35:33.915340 18440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c8f59b32e6492ab7b2a8bd13046f5c" candidate_uuid: "2ecd07944d5e4272aca25d935174886d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "413bca71aa954f548861a1ead8b76ec4"
I20260926 23:35:33.915639 18440 raft_consensus.cc:3037] T a7c8f59b32e6492ab7b2a8bd13046f5c P 413bca71aa954f548861a1ead8b76ec4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:35:33.915938 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c8f59b32e6492ab7b2a8bd13046f5c" candidate_uuid: "2ecd07944d5e4272aca25d935174886d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35446c828eae4da29f41dbf9d80cb9a3"
I20260926 23:35:33.916198 18309 raft_consensus.cc:3037] T a7c8f59b32e6492ab7b2a8bd13046f5c P 35446c828eae4da29f41dbf9d80cb9a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:35:33.917268 18630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:33.909178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40995 (local address 127.0.97.3:39285)
0926 23:35:33.909377 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:33.909383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:33.909787 (+ 404us) server_negotiation.cc:407] Connection header received
0926 23:35:33.910122 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:33.910128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:33.910218 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:33.910289 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:35:33.911322 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.912597 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.914804 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.915978 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.916108 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:33.916216 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:33.916846 (+ 630us) server_negotiation.cc:299] Negotiation successful
0926 23:35:33.917002 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 23:35:33.918083 18645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:33.908989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39285 (local address 127.0.97.1:40995)
0926 23:35:33.909671 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:33.909691 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:33.909941 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:33.910516 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:33.910522 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:33.910549 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:35:33.911127 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:33.911147 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.912769 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.912789 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:33.914543 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:33.914560 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.916369 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.916376 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:33.916621 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:33.916664 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 23:35:33.917468 (+ 804us) client_negotiation.cc:240] Negotiation successful
0926 23:35:33.917653 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":3,"thread_start_us":328,"threads_started":1}
I20260926 23:35:33.919539 18440 raft_consensus.cc:2454] T a7c8f59b32e6492ab7b2a8bd13046f5c P 413bca71aa954f548861a1ead8b76ec4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ecd07944d5e4272aca25d935174886d in term 1.
W20260926 23:35:33.919683 18355 tablet.cc:2369] T a7c8f59b32e6492ab7b2a8bd13046f5c P 35446c828eae4da29f41dbf9d80cb9a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:35:33.919562 18571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c8f59b32e6492ab7b2a8bd13046f5c" candidate_uuid: "35446c828eae4da29f41dbf9d80cb9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ecd07944d5e4272aca25d935174886d" is_pre_election: true
I20260926 23:35:33.919916 18309 raft_consensus.cc:2454] T a7c8f59b32e6492ab7b2a8bd13046f5c P 35446c828eae4da29f41dbf9d80cb9a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ecd07944d5e4272aca25d935174886d in term 1.
I20260926 23:35:33.920084 18504 leader_election.cc:302] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecd07944d5e4272aca25d935174886d, 413bca71aa954f548861a1ead8b76ec4; no voters:
I20260926 23:35:33.920178 18571 raft_consensus.cc:2379] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 35446c828eae4da29f41dbf9d80cb9a3 in current term 1: Already voted for candidate 2ecd07944d5e4272aca25d935174886d in this term.
I20260926 23:35:33.921227 18641 raft_consensus.cc:2781] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:35:33.921718 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:33.911800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35191 (local address 127.0.97.2:39779)
0926 23:35:33.912007 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:33.912014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:33.912156 (+ 142us) server_negotiation.cc:407] Connection header received
0926 23:35:33.912428 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:33.912434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:33.912517 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:33.912581 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:35:33.913441 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.914539 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.917674 (+ 3135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.919060 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.919222 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:33.919349 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:33.921110 (+ 1761us) server_negotiation.cc:299] Negotiation successful
0926 23:35:33.921273 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 23:35:33.922271 18641 raft_consensus.cc:686] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d [term 1 LEADER]: Becoming Leader. State: Replica: 2ecd07944d5e4272aca25d935174886d, State: Running, Role: LEADER
I20260926 23:35:33.922355 18646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:33.911043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39779 (local address 127.0.97.1:35191)
0926 23:35:33.912037 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:33.912065 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:33.912261 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:33.912769 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:33.912774 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:33.912800 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:35:33.913217 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:33.913228 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.915004 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.915011 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:33.917192 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:33.917211 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:33.920685 (+ 3474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:33.920693 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:33.920881 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:33.920922 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 23:35:33.921803 (+ 881us) client_negotiation.cc:240] Negotiation successful
0926 23:35:33.921977 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"mutex_wait_us":158,"thread_start_us":942,"threads_started":1}
I20260926 23:35:33.922981 18641 consensus_queue.cc:237] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecd07944d5e4272aca25d935174886d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39285 } } peers { permanent_uuid: "413bca71aa954f548861a1ead8b76ec4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39779 } } peers { permanent_uuid: "35446c828eae4da29f41dbf9d80cb9a3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46349 } }
I20260926 23:35:33.923127 18440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c8f59b32e6492ab7b2a8bd13046f5c" candidate_uuid: "35446c828eae4da29f41dbf9d80cb9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "413bca71aa954f548861a1ead8b76ec4" is_pre_election: true
I20260926 23:35:33.923480 18440 raft_consensus.cc:2379] T a7c8f59b32e6492ab7b2a8bd13046f5c P 413bca71aa954f548861a1ead8b76ec4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 35446c828eae4da29f41dbf9d80cb9a3 in current term 1: Already voted for candidate 2ecd07944d5e4272aca25d935174886d in this term.
I20260926 23:35:33.924127 18242 leader_election.cc:302] T a7c8f59b32e6492ab7b2a8bd13046f5c P 35446c828eae4da29f41dbf9d80cb9a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35446c828eae4da29f41dbf9d80cb9a3; no voters: 2ecd07944d5e4272aca25d935174886d, 413bca71aa954f548861a1ead8b76ec4
I20260926 23:35:33.924538 18642 raft_consensus.cc:2726] T a7c8f59b32e6492ab7b2a8bd13046f5c P 35446c828eae4da29f41dbf9d80cb9a3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 23:35:33.929152 18167 catalog_manager.cc:5495] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ecd07944d5e4272aca25d935174886d (127.0.97.3). New cstate: current_term: 1 leader_uuid: "2ecd07944d5e4272aca25d935174886d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecd07944d5e4272aca25d935174886d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "413bca71aa954f548861a1ead8b76ec4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35446c828eae4da29f41dbf9d80cb9a3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46349 } health_report { overall_health: UNKNOWN } } }
I20260926 23:35:34.014567 18630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:34.005007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41880 (local address 127.0.97.3:39285)
0926 23:35:34.005292 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:34.005300 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:34.005390 (+ 90us) server_negotiation.cc:407] Connection header received
0926 23:35:34.005744 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:34.005751 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:34.005951 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:34.006146 (+ 195us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:35:34.006641 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:34.007581 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:34.009902 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:34.010709 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:34.010874 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:34.011994 (+ 1120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:35:34.012987 (+ 993us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:35:34.013154 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:34.014053 (+ 899us) server_negotiation.cc:299] Negotiation successful
0926 23:35:34.014201 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 23:35:34.027901 18440 raft_consensus.cc:1261] T a7c8f59b32e6492ab7b2a8bd13046f5c P 413bca71aa954f548861a1ead8b76ec4 [term 1 FOLLOWER]: Refusing update from remote peer 2ecd07944d5e4272aca25d935174886d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:35:34.027954 18309 raft_consensus.cc:1261] T a7c8f59b32e6492ab7b2a8bd13046f5c P 35446c828eae4da29f41dbf9d80cb9a3 [term 1 FOLLOWER]: Refusing update from remote peer 2ecd07944d5e4272aca25d935174886d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:35:34.029107 18647 consensus_queue.cc:1035] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d [LEADER]: Connected to new peer: Peer: permanent_uuid: "413bca71aa954f548861a1ead8b76ec4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:35:34.030023 18647 consensus_queue.cc:1035] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d [LEADER]: Connected to new peer: Peer: permanent_uuid: "35446c828eae4da29f41dbf9d80cb9a3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:35:34.046161 18650 mvcc.cc:205] Tried to move back new op lower bound from 7333747646565097472 to 7333747646155075584. Current Snapshot: MvccSnapshot[applied={T|T < 7333747646565097472}]
I20260926 23:35:34.048619 18652 mvcc.cc:205] Tried to move back new op lower bound from 7333747646565097472 to 7333747646155075584. Current Snapshot: MvccSnapshot[applied={T|T < 7333747646565097472}]
I20260926 23:35:34.050159 18651 mvcc.cc:205] Tried to move back new op lower bound from 7333747646565097472 to 7333747646155075584. Current Snapshot: MvccSnapshot[applied={T|T < 7333747646565097472}]
I20260926 23:35:34.068231 18167 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:35:34.068624 18167 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:35:34.073654 18167 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c103125deedd4843b32505809ef7bea7: Sending DeleteTablet for 3 replicas of tablet a7c8f59b32e6492ab7b2a8bd13046f5c
I20260926 23:35:34.075515 18551 tablet_service.cc:1514] Processing DeleteTablet for tablet a7c8f59b32e6492ab7b2a8bd13046f5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:35:34 UTC) from {username='slave'} at 127.0.0.1:41870
I20260926 23:35:34.075954 18289 tablet_service.cc:1514] Processing DeleteTablet for tablet a7c8f59b32e6492ab7b2a8bd13046f5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:35:34 UTC) from {username='slave'} at 127.0.0.1:51300
I20260926 23:35:34.076314 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 18227
I20260926 23:35:34.077646 18420 tablet_service.cc:1514] Processing DeleteTablet for tablet a7c8f59b32e6492ab7b2a8bd13046f5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:35:34 UTC) from {username='slave'} at 127.0.0.1:40224
I20260926 23:35:34.086808 18663 tablet_replica.cc:330] T a7c8f59b32e6492ab7b2a8bd13046f5c P 413bca71aa954f548861a1ead8b76ec4: stopping tablet replica
I20260926 23:35:34.087679 18663 raft_consensus.cc:2229] T a7c8f59b32e6492ab7b2a8bd13046f5c P 413bca71aa954f548861a1ead8b76ec4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:35:34.088526 18663 raft_consensus.cc:2258] T a7c8f59b32e6492ab7b2a8bd13046f5c P 413bca71aa954f548861a1ead8b76ec4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:35:34.088749 18660 tablet_replica.cc:330] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d: stopping tablet replica
I20260926 23:35:34.089527 18660 raft_consensus.cc:2229] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:35:34.090494 18660 raft_consensus.cc:2258] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:35:34.091825 18663 ts_tablet_manager.cc:1905] T a7c8f59b32e6492ab7b2a8bd13046f5c P 413bca71aa954f548861a1ead8b76ec4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:35:34.093307 18660 ts_tablet_manager.cc:1905] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:35:34.095409 18661 tablet_replica.cc:330] T a7c8f59b32e6492ab7b2a8bd13046f5c P 35446c828eae4da29f41dbf9d80cb9a3: stopping tablet replica
I20260926 23:35:34.095997 18661 raft_consensus.cc:2229] T a7c8f59b32e6492ab7b2a8bd13046f5c P 35446c828eae4da29f41dbf9d80cb9a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:35:34.096657 18661 raft_consensus.cc:2258] T a7c8f59b32e6492ab7b2a8bd13046f5c P 35446c828eae4da29f41dbf9d80cb9a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:35:34.100687 18663 ts_tablet_manager.cc:1918] T a7c8f59b32e6492ab7b2a8bd13046f5c P 413bca71aa954f548861a1ead8b76ec4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:35:34.100988 18663 log.cc:1196] T a7c8f59b32e6492ab7b2a8bd13046f5c P 413bca71aa954f548861a1ead8b76ec4: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-1/wal/wals/a7c8f59b32e6492ab7b2a8bd13046f5c
I20260926 23:35:34.101683 18663 ts_tablet_manager.cc:1939] T a7c8f59b32e6492ab7b2a8bd13046f5c P 413bca71aa954f548861a1ead8b76ec4: Deleting consensus metadata
I20260926 23:35:34.102490 18660 ts_tablet_manager.cc:1918] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:35:34.102779 18660 log.cc:1196] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790465400694755-388-0/minicluster-data/ts-2/wal/wals/a7c8f59b32e6492ab7b2a8bd13046f5c
I20260926 23:35:34.103122 18661 ts_tablet_manager.cc:1905] T a7c8f59b32e6492ab7b2a8bd13046f5c P 35446c828eae4da29f41dbf9d80cb9a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:35:34.103387 18660 ts_tablet_manager.cc:1939] T a7c8f59b32e6492ab7b2a8bd13046f5c P 2ecd07944d5e4272aca25d935174886d: Deleting consensus metadata
I20260926 23:35:34.104938 18152 catalog_manager.cc:4841] TS 413bca71aa954f548861a1ead8b76ec4 (127.0.97.2:39779): tablet a7c8f59b32e6492ab7b2a8bd13046f5c (table test-table [id=5351769161b244d8a5c2ea8da460dc47]) successfully deleted
I20260926 23:35:34.105659 18152 catalog_manager.cc:4841] TS 2ecd07944d5e4272aca25d935174886d (127.0.97.3:39285): tablet a7c8f59b32e6492ab7b2a8bd13046f5c (table test-table [id=5351769161b244d8a5c2ea8da460dc47]) successfully deleted
I20260926 23:35:34.108393 18632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:34.078466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51302 (local address 127.0.97.1:46349)
0926 23:35:34.078813 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:34.078821 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:34.089129 (+ 10308us) server_negotiation.cc:407] Connection header received
0926 23:35:34.090383 (+ 1254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:34.090390 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:34.090540 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:34.090643 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:34.091528 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:34.092687 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:34.097215 (+ 4528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:34.098101 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:34.098269 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:34.106174 (+ 7905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:34.106192 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:34.106219 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:35:34.107281 (+ 1062us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:34.107385 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:35:34.107418 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:34.107578 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:34.107804 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 23:35:34.107967 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260926 23:35:34.109555 18349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:35:34.243080 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 18227
I20260926 23:35:34.262526 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 18358
W20260926 23:35:34.262669 18153 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:46349 (error 108)
I20260926 23:35:34.267753 18666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:34.264364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46349 (local address 127.0.0.1:51314)
0926 23:35:34.267265 (+ 2901us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:34.267446 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2745,"thread_start_us":2627,"threads_started":1}
W20260926 23:35:34.268147 18153 catalog_manager.cc:4584] TS 35446c828eae4da29f41dbf9d80cb9a3 (127.0.97.1:46349): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a7c8f59b32e6492ab7b2a8bd13046f5c: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46349: connect: Connection refused (error 111)
I20260926 23:35:34.272374 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:34.263969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40228 (local address 127.0.97.2:39779)
0926 23:35:34.264188 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:34.264194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:34.264215 (+ 21us) server_negotiation.cc:407] Connection header received
0926 23:35:34.264298 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:34.264304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:34.264456 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:34.264580 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:34.265403 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:34.266662 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:34.268344 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:34.268856 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:34.268978 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:34.270321 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:34.270331 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:34.270350 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:35:34.271492 (+ 1142us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:34.271578 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:35:34.271620 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:34.271737 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:34.271925 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 23:35:34.272041 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 23:35:34.273156 18480 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:35:34.322569 18666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:34.321866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46349 (local address 127.0.0.1:51330)
0926 23:35:34.322199 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:34.322328 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260926 23:35:34.324126 18666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:34.323586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46349 (local address 127.0.0.1:51340)
0926 23:35:34.323796 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:34.323904 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 23:35:34.375028 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 18358
I20260926 23:35:34.385432 18666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:34.384741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46349 (local address 127.0.0.1:51350)
0926 23:35:34.385053 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:34.385180 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260926 23:35:34.387096 18666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:34.386457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46349 (local address 127.0.0.1:51366)
0926 23:35:34.386739 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:34.386862 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 23:35:34.395555 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 18489
I20260926 23:35:34.406342 18630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:34.396603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41890 (local address 127.0.97.3:39285)
0926 23:35:34.396862 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:34.396869 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:34.396895 (+ 26us) server_negotiation.cc:407] Connection header received
0926 23:35:34.397098 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:34.397106 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:34.397271 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:34.397423 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:34.398029 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:34.399414 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:34.401393 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:34.402163 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:34.402335 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:34.404140 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:34.404154 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:34.404186 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:35:34.405234 (+ 1048us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:34.405338 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:35:34.405374 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:34.405507 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:34.405779 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 23:35:34.405930 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260926 23:35:34.415544 18611 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:35:34.468196 18666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:34.467643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46349 (local address 127.0.0.1:51368)
0926 23:35:34.467899 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:34.468000 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260926 23:35:34.469537 18666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:34.469004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46349 (local address 127.0.0.1:51372)
0926 23:35:34.469250 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:34.469332 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260926 23:35:34.561357 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 18489
I20260926 23:35:34.580701 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 18137
I20260926 23:35:34.582114 18198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:35:34.706256 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 18137
2026-09-26T23:35:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3608 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:35:34 dist-test-slave-gwfx krb5kdc[18672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:35:34 dist-test-slave-gwfx krb5kdc[18672](info): set up 2 sockets
Sep 26 23:35:34 dist-test-slave-gwfx krb5kdc[18672](info): commencing operation
krb5kdc: starting...
W20260926 23:35:35.787415 388 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:35:35 dist-test-slave-gwfx krb5kdc[18672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:35:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:35:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:35:36.101691 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38639
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:35:36.529670 18685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:35:36.530082 18685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:35:36.530205 18685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:35:36.538547 18685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:35:36.538769 18685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:35:36.538877 18685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:35:36.538959 18685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:35:36.539005 18685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:35:36.550585 18685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:35:36.552467 18685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:35:36.554188 18685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:35:36.569902 18691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:36.569957 18690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:36.569995 18693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:35:36.571946 18685 server_base.cc:1034] running on GCE node
I20260926 23:35:36.573302 18685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:35:36.575488 18685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:35:36.576761 18685 hybrid_clock.cc:648] HybridClock initialized: now 1790465736576680 us; error 82 us; skew 500 ppm
I20260926 23:35:36.577210 18685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:35:36 dist-test-slave-gwfx krb5kdc[18672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:35:36.581542 18685 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:35:36.583948 18685 webserver.cc:460] Webserver started at http://127.0.97.62:45397/ using document root <none> and password file <none>
I20260926 23:35:36.584914 18685 fs_manager.cc:359] Metadata directory not provided
I20260926 23:35:36.585062 18685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:35:36.585450 18685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:35:36.589188 18685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "e0c4aca58afa42439b8d22b3277b7eae"
format_stamp: "Formatted at 2026-09-26 23:35:36 on dist-test-slave-gwfx"
server_key: "4ce14ab5fdf385e6acbf168c7c1d7e52"
server_key_iv: "b427813de643e4c9e6336602e2ab7474"
server_key_version: "encryptionkey@0"
I20260926 23:35:36.590050 18685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "e0c4aca58afa42439b8d22b3277b7eae"
format_stamp: "Formatted at 2026-09-26 23:35:36 on dist-test-slave-gwfx"
server_key: "4ce14ab5fdf385e6acbf168c7c1d7e52"
server_key_iv: "b427813de643e4c9e6336602e2ab7474"
server_key_version: "encryptionkey@0"
I20260926 23:35:36.596823 18685 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260926 23:35:36.601661 18700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:36.603829 18685 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 23:35:36.604101 18685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "e0c4aca58afa42439b8d22b3277b7eae"
format_stamp: "Formatted at 2026-09-26 23:35:36 on dist-test-slave-gwfx"
server_key: "4ce14ab5fdf385e6acbf168c7c1d7e52"
server_key_iv: "b427813de643e4c9e6336602e2ab7474"
server_key_version: "encryptionkey@0"
I20260926 23:35:36.604358 18685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:35:36.646883 18685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:35:36.653709 18685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:35:36.654111 18685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:35:36.678583 18685 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38639
I20260926 23:35:36.678638 18751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38639 every 8 connection(s)
I20260926 23:35:36.680424 18685 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:35:36.685885 18752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:35:36.688566 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 18685
I20260926 23:35:36.688795 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:35:36.689337 388 external_mini_cluster.cc:1428] Setting key 66cb609fd7d9afcc86953ca656375478
I20260926 23:35:36.701390 18752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0c4aca58afa42439b8d22b3277b7eae: Bootstrap starting.
I20260926 23:35:36.705864 18752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0c4aca58afa42439b8d22b3277b7eae: Neither blocks nor log segments found. Creating new log.
I20260926 23:35:36.707404 18752 log.cc:824] T 00000000000000000000000000000000 P e0c4aca58afa42439b8d22b3277b7eae: Log is configured to *not* fsync() on all Append() calls
Sep 26 23:35:36 dist-test-slave-gwfx krb5kdc[18672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:35:36.711591 18752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0c4aca58afa42439b8d22b3277b7eae: No bootstrap required, opened a new log
I20260926 23:35:36.723934 18752 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0c4aca58afa42439b8d22b3277b7eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c4aca58afa42439b8d22b3277b7eae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38639 } }
I20260926 23:35:36.724804 18755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:36.692336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42682 (local address 127.0.97.62:38639)
0926 23:35:36.693683 (+ 1347us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:36.693706 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:36.694019 (+ 313us) server_negotiation.cc:407] Connection header received
0926 23:35:36.696603 (+ 2584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:36.696642 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:36.697351 (+ 709us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:36.697873 (+ 522us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:36.700030 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:36.701953 (+ 1923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:36.704584 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:36.705597 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:36.705821 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:36.711719 (+ 5898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:36.711733 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:36.711757 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:35:36.711815 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:36.717379 (+ 5564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:36.718715 (+ 1336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:36.718727 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:36.718737 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:36.719015 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:36.719602 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:36.719607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:36.719611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:36.720455 (+ 844us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:36.720803 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:36.721428 (+ 625us) server_negotiation.cc:299] Negotiation successful
0926 23:35:36.722513 (+ 1085us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":941,"thread_start_us":514,"threads_started":1}
I20260926 23:35:36.724831 18752 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0c4aca58afa42439b8d22b3277b7eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:35:36.724989 18752 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0c4aca58afa42439b8d22b3277b7eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0c4aca58afa42439b8d22b3277b7eae, State: Initialized, Role: FOLLOWER
I20260926 23:35:36.725788 18752 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0c4aca58afa42439b8d22b3277b7eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c4aca58afa42439b8d22b3277b7eae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38639 } }
I20260926 23:35:36.726099 18752 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0c4aca58afa42439b8d22b3277b7eae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:35:36.726265 18752 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0c4aca58afa42439b8d22b3277b7eae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:35:36.726462 18752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0c4aca58afa42439b8d22b3277b7eae [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:35:36.729038 18752 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0c4aca58afa42439b8d22b3277b7eae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c4aca58afa42439b8d22b3277b7eae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38639 } }
I20260926 23:35:36.729653 18752 leader_election.cc:302] T 00000000000000000000000000000000 P e0c4aca58afa42439b8d22b3277b7eae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0c4aca58afa42439b8d22b3277b7eae; no voters:
I20260926 23:35:36.730322 18752 leader_election.cc:288] T 00000000000000000000000000000000 P e0c4aca58afa42439b8d22b3277b7eae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:35:36.730571 18757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0c4aca58afa42439b8d22b3277b7eae [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:35:36.731540 18757 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0c4aca58afa42439b8d22b3277b7eae [term 1 LEADER]: Becoming Leader. State: Replica: e0c4aca58afa42439b8d22b3277b7eae, State: Running, Role: LEADER
I20260926 23:35:36.732239 18757 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0c4aca58afa42439b8d22b3277b7eae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c4aca58afa42439b8d22b3277b7eae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38639 } }
I20260926 23:35:36.732985 18752 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0c4aca58afa42439b8d22b3277b7eae [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:35:36.735692 18759 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0c4aca58afa42439b8d22b3277b7eae [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0c4aca58afa42439b8d22b3277b7eae. Latest consensus state: current_term: 1 leader_uuid: "e0c4aca58afa42439b8d22b3277b7eae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c4aca58afa42439b8d22b3277b7eae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38639 } } }
I20260926 23:35:36.736255 18759 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0c4aca58afa42439b8d22b3277b7eae [sys.catalog]: This master's current role is: LEADER
I20260926 23:35:36.737098 18758 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0c4aca58afa42439b8d22b3277b7eae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0c4aca58afa42439b8d22b3277b7eae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c4aca58afa42439b8d22b3277b7eae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38639 } } }
I20260926 23:35:36.737568 18758 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0c4aca58afa42439b8d22b3277b7eae [sys.catalog]: This master's current role is: LEADER
I20260926 23:35:36.740890 18764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:35:36.750350 18764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:35:36.763235 18764 catalog_manager.cc:1347] Generated new cluster ID: 04cbb354a22c4c61b47ad585782ffd88
I20260926 23:35:36.763367 18764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:35:36.790045 18764 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:35:36.791716 18764 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:35:36.801098 18764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0c4aca58afa42439b8d22b3277b7eae: Generated new TSK 0
I20260926 23:35:36.802667 18764 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:35:36.874904 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38639
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:35:37.275007 18778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:35:37.275429 18778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:35:37.275599 18778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:35:37.284621 18778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:35:37.284809 18778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:35:37.284992 18778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:35:37.297688 18778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:35:37.299639 18778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:35:37.301824 18778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:35:37.320514 18784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:37.320497 18783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:37.320884 18786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:35:37.321584 18778 server_base.cc:1034] running on GCE node
I20260926 23:35:37.322477 18778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:35:37.324057 18778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:35:37.325430 18778 hybrid_clock.cc:648] HybridClock initialized: now 1790465737325294 us; error 108 us; skew 500 ppm
I20260926 23:35:37.326035 18778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:35:37 dist-test-slave-gwfx krb5kdc[18672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:35:37.331125 18778 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:35:37.333684 18778 webserver.cc:460] Webserver started at http://127.0.97.1:37775/ using document root <none> and password file <none>
I20260926 23:35:37.334764 18778 fs_manager.cc:359] Metadata directory not provided
I20260926 23:35:37.334941 18778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:35:37.335352 18778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:35:37.340078 18778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-0/data/instance:
uuid: "8726ea4e193140e0a4bf2d59ddee1b0c"
format_stamp: "Formatted at 2026-09-26 23:35:37 on dist-test-slave-gwfx"
server_key: "97090f50002fb2d358e21e172ebcc34e"
server_key_iv: "eeed14984f3521d2d39951dd4011761f"
server_key_version: "encryptionkey@0"
I20260926 23:35:37.341394 18778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8726ea4e193140e0a4bf2d59ddee1b0c"
format_stamp: "Formatted at 2026-09-26 23:35:37 on dist-test-slave-gwfx"
server_key: "97090f50002fb2d358e21e172ebcc34e"
server_key_iv: "eeed14984f3521d2d39951dd4011761f"
server_key_version: "encryptionkey@0"
I20260926 23:35:37.349041 18778 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260926 23:35:37.354513 18793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:37.356961 18778 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260926 23:35:37.357246 18778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "8726ea4e193140e0a4bf2d59ddee1b0c"
format_stamp: "Formatted at 2026-09-26 23:35:37 on dist-test-slave-gwfx"
server_key: "97090f50002fb2d358e21e172ebcc34e"
server_key_iv: "eeed14984f3521d2d39951dd4011761f"
server_key_version: "encryptionkey@0"
I20260926 23:35:37.357517 18778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:35:37.380158 18778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:35:37.385747 18778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:35:37.386098 18778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:35:37.387413 18778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:35:37.389182 18778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:35:37.389276 18778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:37.389420 18778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:35:37.389477 18778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:37.428759 18778 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44763
I20260926 23:35:37.428862 18905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44763 every 8 connection(s)
I20260926 23:35:37.430377 18778 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:35:37.435581 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 18778
I20260926 23:35:37.435775 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:35:37.436309 388 external_mini_cluster.cc:1428] Setting key bd23257a2a0598f972c8343d0496e964
Sep 26 23:35:37 dist-test-slave-gwfx krb5kdc[18672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:35:37.458940 18908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:37.433629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46911 (local address 127.0.97.62:38639)
0926 23:35:37.434188 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:37.434193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:37.435515 (+ 1322us) server_negotiation.cc:407] Connection header received
0926 23:35:37.438175 (+ 2660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:37.438180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:37.438275 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:37.438398 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:37.441121 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:37.442072 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:37.444644 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:37.445167 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:37.445313 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:37.451850 (+ 6537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:37.451857 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:37.451861 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:35:37.451912 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:37.455112 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:37.456128 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:37.456133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:37.456136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:37.456279 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:37.456765 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:37.456769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:37.456771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:37.457113 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:37.457263 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:37.458173 (+ 910us) server_negotiation.cc:299] Negotiation successful
0926 23:35:37.458486 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":262,"threads_started":1}
I20260926 23:35:37.460232 18909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:37.434206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38639 (local address 127.0.97.1:46911)
0926 23:35:37.435246 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:37.435319 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:37.437762 (+ 2443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:37.438785 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:37.438795 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:37.439812 (+ 1017us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:35:37.440820 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:37.440854 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:37.442345 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:37.442350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:37.444385 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:37.444411 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:37.445499 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:37.445508 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:37.446209 (+ 701us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:37.448083 (+ 1874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:35:37.448132 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:35:37.451454 (+ 3322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:35:37.455347 (+ 3893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:35:37.455358 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:35:37.455379 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:35:37.455963 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:35:37.456401 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:35:37.456406 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:35:37.456409 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:35:37.456636 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:35:37.457251 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:35:37.457260 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:35:37.457721 (+ 461us) client_negotiation.cc:769] Sending connection context
0926 23:35:37.458025 (+ 304us) client_negotiation.cc:240] Negotiation successful
0926 23:35:37.458790 (+ 765us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":402,"threads_started":1}
I20260926 23:35:37.462862 18906 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38639
I20260926 23:35:37.463341 18906 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 23:35:37.464443 18906 heartbeater.cc:511] Master 127.0.97.62:38639 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 23:35:37.467576 18717 ts_manager.cc:194] Registered new tserver with Master: 8726ea4e193140e0a4bf2d59ddee1b0c (127.0.97.1:44763)
I20260926 23:35:37.469334 18717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46911
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:35:37.500053 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38639
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:35:37.854123 18912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:35:37.854445 18912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:35:37.854544 18912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:35:37.862314 18912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:35:37.862453 18912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:35:37.862587 18912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:35:37.872944 18912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:35:37.874405 18912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:35:37.875960 18912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:35:37.892591 18918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:37.892666 18920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:37.893303 18917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:35:37.893622 18912 server_base.cc:1034] running on GCE node
I20260926 23:35:37.894519 18912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:35:37.895856 18912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:35:37.897233 18912 hybrid_clock.cc:648] HybridClock initialized: now 1790465737897145 us; error 59 us; skew 500 ppm
I20260926 23:35:37.897861 18912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:35:37 dist-test-slave-gwfx krb5kdc[18672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:35:37.902441 18912 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:35:37.904651 18912 webserver.cc:460] Webserver started at http://127.0.97.2:37767/ using document root <none> and password file <none>
I20260926 23:35:37.905527 18912 fs_manager.cc:359] Metadata directory not provided
I20260926 23:35:37.905676 18912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:35:37.906059 18912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:35:37.909439 18912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-1/data/instance:
uuid: "f5efbb39a5374eba87b14ec37ccf9220"
format_stamp: "Formatted at 2026-09-26 23:35:37 on dist-test-slave-gwfx"
server_key: "24962d4516fd1335e019eab4fce5dbb9"
server_key_iv: "f181731b7bfd6e588482f46c92fa843e"
server_key_version: "encryptionkey@0"
I20260926 23:35:37.910485 18912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f5efbb39a5374eba87b14ec37ccf9220"
format_stamp: "Formatted at 2026-09-26 23:35:37 on dist-test-slave-gwfx"
server_key: "24962d4516fd1335e019eab4fce5dbb9"
server_key_iv: "f181731b7bfd6e588482f46c92fa843e"
server_key_version: "encryptionkey@0"
I20260926 23:35:37.917001 18912 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260926 23:35:37.922032 18927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:37.924062 18912 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 23:35:37.924304 18912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "f5efbb39a5374eba87b14ec37ccf9220"
format_stamp: "Formatted at 2026-09-26 23:35:37 on dist-test-slave-gwfx"
server_key: "24962d4516fd1335e019eab4fce5dbb9"
server_key_iv: "f181731b7bfd6e588482f46c92fa843e"
server_key_version: "encryptionkey@0"
I20260926 23:35:37.924552 18912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:35:37.953626 18912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:35:37.960668 18912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:35:37.961004 18912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:35:37.962182 18912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:35:37.963963 18912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:35:37.964051 18912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:37.964179 18912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:35:37.964231 18912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:38.005450 18912 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36317
I20260926 23:35:38.005549 19039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36317 every 8 connection(s)
I20260926 23:35:38.006963 18912 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:35:38.008718 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 18912
I20260926 23:35:38.008971 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:35:38.009907 388 external_mini_cluster.cc:1428] Setting key 0ebc076f3cd7391fca33c09ed6cff193
Sep 26 23:35:38 dist-test-slave-gwfx krb5kdc[18672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:35:38.036001 19042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:38.011250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59643 (local address 127.0.97.62:38639)
0926 23:35:38.011839 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:38.011847 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:38.013069 (+ 1222us) server_negotiation.cc:407] Connection header received
0926 23:35:38.016322 (+ 3253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:38.016330 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:38.016512 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:38.016699 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:38.019515 (+ 2816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.020660 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.022613 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.023321 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.023505 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:38.028635 (+ 5130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:38.028642 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:38.028646 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:35:38.028694 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:38.032033 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:38.033057 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:38.033062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:38.033064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:38.033208 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:38.033760 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:38.033764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:38.033766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:38.034118 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:38.034281 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:38.035272 (+ 991us) server_negotiation.cc:299] Negotiation successful
0926 23:35:38.035591 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":265,"threads_started":1}
I20260926 23:35:38.037442 19043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:38.011947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38639 (local address 127.0.97.2:59643)
0926 23:35:38.012860 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:38.012924 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:38.015603 (+ 2679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:38.017333 (+ 1730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:38.017347 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:38.018348 (+ 1001us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:35:38.019207 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:38.019233 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.020940 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.020945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:38.022388 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:38.022409 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.023665 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.023672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:38.023984 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:38.024921 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:35:38.024957 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:35:38.028225 (+ 3268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:35:38.032270 (+ 4045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:35:38.032281 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:35:38.032303 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:35:38.032897 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:35:38.033326 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:35:38.033330 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:35:38.033333 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:35:38.033632 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:35:38.034270 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:35:38.034280 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:35:38.034799 (+ 519us) client_negotiation.cc:769] Sending connection context
0926 23:35:38.035125 (+ 326us) client_negotiation.cc:240] Negotiation successful
0926 23:35:38.035870 (+ 745us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":382,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 23:35:38.039853 19040 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38639
I20260926 23:35:38.040320 19040 heartbeater.cc:464] Registering TS with master...
I20260926 23:35:38.041333 19040 heartbeater.cc:511] Master 127.0.97.62:38639 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 23:35:38.043701 18717 ts_manager.cc:194] Registered new tserver with Master: f5efbb39a5374eba87b14ec37ccf9220 (127.0.97.2:36317)
I20260926 23:35:38.045022 18717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:35:38.089125 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38639
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:35:38.473949 18906 heartbeater.cc:503] Master 127.0.97.62:38639 was elected leader, sending a full tablet report...
W20260926 23:35:38.498354 19046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:35:38.498895 19046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:35:38.499090 19046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:35:38.508263 19046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:35:38.508433 19046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:35:38.508594 19046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:35:38.521128 19046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:35:38.523068 19046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:35:38.524744 19046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:35:38.541025 19051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:38.541077 19054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:38.540984 19052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:35:38.547876 19046 server_base.cc:1034] running on GCE node
I20260926 23:35:38.548992 19046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:35:38.550413 19046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:35:38.551679 19046 hybrid_clock.cc:648] HybridClock initialized: now 1790465738551581 us; error 87 us; skew 500 ppm
I20260926 23:35:38.552183 19046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:35:38 dist-test-slave-gwfx krb5kdc[18672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:35:38.556102 19046 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:35:38.558327 19046 webserver.cc:460] Webserver started at http://127.0.97.3:42649/ using document root <none> and password file <none>
I20260926 23:35:38.559367 19046 fs_manager.cc:359] Metadata directory not provided
I20260926 23:35:38.559499 19046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:35:38.559823 19046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:35:38.563643 19046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-2/data/instance:
uuid: "88fe30aee6e04f01a91662cd462743a3"
format_stamp: "Formatted at 2026-09-26 23:35:38 on dist-test-slave-gwfx"
server_key: "5f2dae07ffc0f899d37784ec69932201"
server_key_iv: "3c6d4d29f3d963b0dfec7927298b3868"
server_key_version: "encryptionkey@0"
I20260926 23:35:38.564599 19046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-2/wal/instance:
uuid: "88fe30aee6e04f01a91662cd462743a3"
format_stamp: "Formatted at 2026-09-26 23:35:38 on dist-test-slave-gwfx"
server_key: "5f2dae07ffc0f899d37784ec69932201"
server_key_iv: "3c6d4d29f3d963b0dfec7927298b3868"
server_key_version: "encryptionkey@0"
I20260926 23:35:38.571300 19046 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260926 23:35:38.576081 19061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:38.578094 19046 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 23:35:38.578351 19046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "88fe30aee6e04f01a91662cd462743a3"
format_stamp: "Formatted at 2026-09-26 23:35:38 on dist-test-slave-gwfx"
server_key: "5f2dae07ffc0f899d37784ec69932201"
server_key_iv: "3c6d4d29f3d963b0dfec7927298b3868"
server_key_version: "encryptionkey@0"
I20260926 23:35:38.578655 19046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:35:38.609534 19046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:35:38.615075 19046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:35:38.615427 19046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:35:38.616606 19046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:35:38.618278 19046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:35:38.618356 19046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:38.618484 19046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:35:38.618556 19046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:38.656652 19046 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33933
I20260926 23:35:38.656697 19173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33933 every 8 connection(s)
I20260926 23:35:38.658622 19046 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:35:38.664794 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 19046
I20260926 23:35:38.665021 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:35:38.665691 388 external_mini_cluster.cc:1428] Setting key 7507842dd5ead2b3f95daec643b9082b
Sep 26 23:35:38 dist-test-slave-gwfx krb5kdc[18672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:35:38.687258 19176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:38.662409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59495 (local address 127.0.97.62:38639)
0926 23:35:38.663090 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:38.663096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:38.664752 (+ 1656us) server_negotiation.cc:407] Connection header received
0926 23:35:38.667700 (+ 2948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:38.667706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:38.667853 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:38.667957 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:38.670450 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.671493 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.673588 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.674099 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.674218 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:38.678865 (+ 4647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:38.678871 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:38.678875 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:35:38.678935 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:38.682787 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:38.683946 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:38.683953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:38.683958 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:38.684122 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:38.684844 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:38.684850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:38.684854 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:38.685273 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:38.685414 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:38.686588 (+ 1174us) server_negotiation.cc:299] Negotiation successful
0926 23:35:38.686883 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":382,"threads_started":1}
I20260926 23:35:38.688869 19177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:38.663321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38639 (local address 127.0.97.3:59495)
0926 23:35:38.664454 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:38.664541 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:38.667328 (+ 2787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:38.668289 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:38.668301 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:38.669267 (+ 966us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:35:38.670216 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:38.670243 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.671722 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.671728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:38.673349 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:38.673370 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.674385 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.674391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:38.674794 (+ 403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:38.675656 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:35:38.675686 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:35:38.678483 (+ 2797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:35:38.682981 (+ 4498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:35:38.682993 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:35:38.683141 (+ 148us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:35:38.683810 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:35:38.684260 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:35:38.684267 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:35:38.684272 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:35:38.684650 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:35:38.685422 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:35:38.685433 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:35:38.686041 (+ 608us) client_negotiation.cc:769] Sending connection context
0926 23:35:38.686457 (+ 416us) client_negotiation.cc:240] Negotiation successful
0926 23:35:38.687269 (+ 812us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":433,"threads_started":1}
I20260926 23:35:38.690770 19174 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38639
I20260926 23:35:38.691151 19174 heartbeater.cc:464] Registering TS with master...
I20260926 23:35:38.692037 19174 heartbeater.cc:511] Master 127.0.97.62:38639 requested a full tablet report, sending...
I20260926 23:35:38.693907 18717 ts_manager.cc:194] Registered new tserver with Master: 88fe30aee6e04f01a91662cd462743a3 (127.0.97.3:33933)
I20260926 23:35:38.695053 18717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59495
I20260926 23:35:38.696091 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:35:38.716990 19176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:38.701223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42688 (local address 127.0.97.62:38639)
0926 23:35:38.701497 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:38.701503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:38.701946 (+ 443us) server_negotiation.cc:407] Connection header received
0926 23:35:38.702191 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:38.702196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:38.702280 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:38.702392 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:38.704236 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.705077 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.706511 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.707100 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.707235 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:38.710532 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:38.710539 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:38.710543 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:35:38.710593 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:38.713778 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:38.714850 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:38.714854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:38.714857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:38.715007 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:38.715465 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:38.715468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:38.715470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:38.715835 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:38.716065 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:38.716506 (+ 441us) server_negotiation.cc:299] Negotiation successful
0926 23:35:38.716677 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 23:35:38.722564 18716 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:35:38.731374 18716 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:35:38.774006 19189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:38.749008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47250 (local address 127.0.97.2:36317)
0926 23:35:38.761702 (+ 12694us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:38.761717 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:38.763683 (+ 1966us) server_negotiation.cc:407] Connection header received
0926 23:35:38.763816 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:38.763827 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:38.764228 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:38.764463 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:35:38.766288 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.767979 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.770841 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.772215 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.772356 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:38.772643 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:38.773433 (+ 790us) server_negotiation.cc:299] Negotiation successful
0926 23:35:38.773684 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12419,"thread_start_us":12833,"threads_started":1}
I20260926 23:35:38.774739 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:38.748872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36317 (local address 127.0.0.1:47250)
0926 23:35:38.762875 (+ 14003us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:38.762952 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:38.763399 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:38.764817 (+ 1418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:38.764826 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:38.764868 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:35:38.766031 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:38.766052 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.768190 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.768195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:38.770544 (+ 2349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:38.770559 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.772836 (+ 2277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.772844 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:38.773115 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:38.773172 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 23:35:38.774255 (+ 1083us) client_negotiation.cc:240] Negotiation successful
0926 23:35:38.774386 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12341,"mutex_wait_us":555,"thread_start_us":14128,"threads_started":1}
I20260926 23:35:38.775168 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:38.748890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44763 (local address 127.0.0.1:52300)
0926 23:35:38.763394 (+ 14504us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:38.763414 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:38.763537 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:38.764817 (+ 1280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:38.764826 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:38.764898 (+ 72us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:35:38.765591 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:38.765615 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.768418 (+ 2803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.768423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:38.771003 (+ 2580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:38.771017 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.773396 (+ 2379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.773403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:38.773583 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:38.773620 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 23:35:38.773712 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 23:35:38.774001 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14361,"mutex_wait_us":92,"spinlock_wait_cycles":7424,"thread_start_us":12813,"threads_started":1}
I20260926 23:35:38.776180 19192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:38.764623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33933 (local address 127.0.0.1:33552)
0926 23:35:38.765824 (+ 1201us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:38.765847 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:38.765988 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:38.767809 (+ 1821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:38.767815 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:38.767842 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:35:38.768413 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:38.768426 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.770726 (+ 2300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.770732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:38.772959 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:38.772974 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.775422 (+ 2448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.775430 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:38.775606 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:38.775638 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 23:35:38.775724 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 23:35:38.775876 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1038,"thread_start_us":784,"threads_started":1}
I20260926 23:35:38.776310 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:38.763910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33552 (local address 127.0.97.3:33933)
0926 23:35:38.766724 (+ 2814us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:38.766735 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:38.766779 (+ 44us) server_negotiation.cc:407] Connection header received
0926 23:35:38.766899 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:38.766909 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:38.767255 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:38.767500 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:35:38.768732 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.770530 (+ 1798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.773185 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.774845 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.774982 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:38.775249 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:38.775806 (+ 557us) server_negotiation.cc:299] Negotiation successful
0926 23:35:38.775963 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2563,"thread_start_us":2360,"threads_started":1}
I20260926 23:35:38.776780 19190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:38.749247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52300 (local address 127.0.97.1:44763)
0926 23:35:38.761853 (+ 12606us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:38.761861 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:38.763682 (+ 1821us) server_negotiation.cc:407] Connection header received
0926 23:35:38.763816 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:38.763827 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:38.764228 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:38.764463 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:35:38.765918 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.768294 (+ 2376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.771203 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.772865 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.773016 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:38.773267 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:38.774520 (+ 1253us) server_negotiation.cc:299] Negotiation successful
0926 23:35:38.774777 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12396,"thread_start_us":12476,"threads_started":1}
I20260926 23:35:38.780301 18841 tablet_service.cc:1467] Processing CreateTablet for tablet 3e0b8bd74d8644adbf5c0b6d3dd11ef1 (DEFAULT_TABLE table=test-table [id=b123d236a5d240f59303a3e4017856fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:35:38.780323 18975 tablet_service.cc:1467] Processing CreateTablet for tablet 3e0b8bd74d8644adbf5c0b6d3dd11ef1 (DEFAULT_TABLE table=test-table [id=b123d236a5d240f59303a3e4017856fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:35:38.781888 18841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e0b8bd74d8644adbf5c0b6d3dd11ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:35:38.781922 18975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e0b8bd74d8644adbf5c0b6d3dd11ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:35:38.781862 19109 tablet_service.cc:1467] Processing CreateTablet for tablet 3e0b8bd74d8644adbf5c0b6d3dd11ef1 (DEFAULT_TABLE table=test-table [id=b123d236a5d240f59303a3e4017856fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:35:38.783865 19109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e0b8bd74d8644adbf5c0b6d3dd11ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:35:38.797181 19193 tablet_bootstrap.cc:492] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c: Bootstrap starting.
I20260926 23:35:38.801395 19193 tablet_bootstrap.cc:654] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c: Neither blocks nor log segments found. Creating new log.
I20260926 23:35:38.802532 19194 tablet_bootstrap.cc:492] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P f5efbb39a5374eba87b14ec37ccf9220: Bootstrap starting.
I20260926 23:35:38.803035 19193 log.cc:824] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c: Log is configured to *not* fsync() on all Append() calls
I20260926 23:35:38.804677 19195 tablet_bootstrap.cc:492] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 88fe30aee6e04f01a91662cd462743a3: Bootstrap starting.
I20260926 23:35:38.806864 19194 tablet_bootstrap.cc:654] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P f5efbb39a5374eba87b14ec37ccf9220: Neither blocks nor log segments found. Creating new log.
I20260926 23:35:38.808103 19194 log.cc:824] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P f5efbb39a5374eba87b14ec37ccf9220: Log is configured to *not* fsync() on all Append() calls
I20260926 23:35:38.808491 19195 tablet_bootstrap.cc:654] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 88fe30aee6e04f01a91662cd462743a3: Neither blocks nor log segments found. Creating new log.
I20260926 23:35:38.809258 19193 tablet_bootstrap.cc:492] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c: No bootstrap required, opened a new log
I20260926 23:35:38.809629 19193 ts_tablet_manager.cc:1397] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20260926 23:35:38.809916 19195 log.cc:824] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 88fe30aee6e04f01a91662cd462743a3: Log is configured to *not* fsync() on all Append() calls
I20260926 23:35:38.811748 19194 tablet_bootstrap.cc:492] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P f5efbb39a5374eba87b14ec37ccf9220: No bootstrap required, opened a new log
I20260926 23:35:38.812042 19194 ts_tablet_manager.cc:1397] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P f5efbb39a5374eba87b14ec37ccf9220: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260926 23:35:38.813573 19195 tablet_bootstrap.cc:492] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 88fe30aee6e04f01a91662cd462743a3: No bootstrap required, opened a new log
I20260926 23:35:38.813932 19195 ts_tablet_manager.cc:1397] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 88fe30aee6e04f01a91662cd462743a3: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 23:35:38.821655 19194 raft_consensus.cc:348] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P f5efbb39a5374eba87b14ec37ccf9220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5efbb39a5374eba87b14ec37ccf9220" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36317 } } peers { permanent_uuid: "8726ea4e193140e0a4bf2d59ddee1b0c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44763 } } peers { permanent_uuid: "88fe30aee6e04f01a91662cd462743a3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33933 } }
I20260926 23:35:38.821710 19193 raft_consensus.cc:348] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5efbb39a5374eba87b14ec37ccf9220" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36317 } } peers { permanent_uuid: "8726ea4e193140e0a4bf2d59ddee1b0c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44763 } } peers { permanent_uuid: "88fe30aee6e04f01a91662cd462743a3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33933 } }
I20260926 23:35:38.822331 19193 raft_consensus.cc:374] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:35:38.822335 19194 raft_consensus.cc:374] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P f5efbb39a5374eba87b14ec37ccf9220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:35:38.822475 19194 raft_consensus.cc:729] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P f5efbb39a5374eba87b14ec37ccf9220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5efbb39a5374eba87b14ec37ccf9220, State: Initialized, Role: FOLLOWER
I20260926 23:35:38.822507 19193 raft_consensus.cc:729] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8726ea4e193140e0a4bf2d59ddee1b0c, State: Initialized, Role: FOLLOWER
I20260926 23:35:38.823387 19194 consensus_queue.cc:260] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P f5efbb39a5374eba87b14ec37ccf9220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5efbb39a5374eba87b14ec37ccf9220" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36317 } } peers { permanent_uuid: "8726ea4e193140e0a4bf2d59ddee1b0c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44763 } } peers { permanent_uuid: "88fe30aee6e04f01a91662cd462743a3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33933 } }
I20260926 23:35:38.823395 19193 consensus_queue.cc:260] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5efbb39a5374eba87b14ec37ccf9220" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36317 } } peers { permanent_uuid: "8726ea4e193140e0a4bf2d59ddee1b0c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44763 } } peers { permanent_uuid: "88fe30aee6e04f01a91662cd462743a3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33933 } }
I20260926 23:35:38.824870 19040 heartbeater.cc:503] Master 127.0.97.62:38639 was elected leader, sending a full tablet report...
I20260926 23:35:38.825872 19194 ts_tablet_manager.cc:1428] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P f5efbb39a5374eba87b14ec37ccf9220: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260926 23:35:38.827221 19195 raft_consensus.cc:348] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 88fe30aee6e04f01a91662cd462743a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5efbb39a5374eba87b14ec37ccf9220" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36317 } } peers { permanent_uuid: "8726ea4e193140e0a4bf2d59ddee1b0c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44763 } } peers { permanent_uuid: "88fe30aee6e04f01a91662cd462743a3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33933 } }
I20260926 23:35:38.828070 19195 raft_consensus.cc:374] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 88fe30aee6e04f01a91662cd462743a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:35:38.828354 19193 ts_tablet_manager.cc:1428] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c: Time spent starting tablet: real 0.019s user 0.020s sys 0.000s
I20260926 23:35:38.828339 19195 raft_consensus.cc:729] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 88fe30aee6e04f01a91662cd462743a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88fe30aee6e04f01a91662cd462743a3, State: Initialized, Role: FOLLOWER
I20260926 23:35:38.829572 19195 consensus_queue.cc:260] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 88fe30aee6e04f01a91662cd462743a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5efbb39a5374eba87b14ec37ccf9220" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36317 } } peers { permanent_uuid: "8726ea4e193140e0a4bf2d59ddee1b0c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44763 } } peers { permanent_uuid: "88fe30aee6e04f01a91662cd462743a3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33933 } }
I20260926 23:35:38.831271 19174 heartbeater.cc:503] Master 127.0.97.62:38639 was elected leader, sending a full tablet report...
I20260926 23:35:38.832191 19195 ts_tablet_manager.cc:1428] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 88fe30aee6e04f01a91662cd462743a3: Time spent starting tablet: real 0.018s user 0.018s sys 0.000s
I20260926 23:35:38.858022 19200 raft_consensus.cc:482] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:35:38.858450 19200 raft_consensus.cc:504] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5efbb39a5374eba87b14ec37ccf9220" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36317 } } peers { permanent_uuid: "8726ea4e193140e0a4bf2d59ddee1b0c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44763 } } peers { permanent_uuid: "88fe30aee6e04f01a91662cd462743a3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33933 } }
I20260926 23:35:38.860680 19200 leader_election.cc:288] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5efbb39a5374eba87b14ec37ccf9220 (127.0.97.2:36317), 88fe30aee6e04f01a91662cd462743a3 (127.0.97.3:33933)
I20260926 23:35:38.868512 19189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:38.861384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49889 (local address 127.0.97.2:36317)
0926 23:35:38.861604 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:38.861609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:38.862267 (+ 658us) server_negotiation.cc:407] Connection header received
0926 23:35:38.862534 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:38.862540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:38.862633 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:38.862779 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:35:38.863571 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.864377 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.866412 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.867455 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.867586 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:38.867699 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:38.868096 (+ 397us) server_negotiation.cc:299] Negotiation successful
0926 23:35:38.868242 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 23:35:38.868604 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:38.862609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37515 (local address 127.0.97.3:33933)
0926 23:35:38.862848 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:38.862854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:38.863123 (+ 269us) server_negotiation.cc:407] Connection header received
0926 23:35:38.863331 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:38.863335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:38.863399 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:38.863480 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:35:38.863997 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.864774 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.866547 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.867505 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.867622 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:38.867728 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:38.868201 (+ 473us) server_negotiation.cc:299] Negotiation successful
0926 23:35:38.868348 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 23:35:38.868646 19202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:38.861150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36317 (local address 127.0.97.1:49889)
0926 23:35:38.862116 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:38.862137 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:38.862313 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:38.862808 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:38.862812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:38.862848 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:35:38.863420 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:38.863433 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.864499 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.864504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:38.866245 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:38.866259 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.867676 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.867682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:38.867890 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:38.867928 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 23:35:38.868048 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 23:35:38.868204 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":51,"spinlock_wait_cycles":101760,"thread_start_us":521,"threads_started":1}
I20260926 23:35:38.869393 19203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:38.862314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33933 (local address 127.0.97.1:37515)
0926 23:35:38.863017 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:38.863036 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:38.863194 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:38.863514 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:38.863519 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:38.863542 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:35:38.863856 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:38.863866 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.864914 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.864918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:38.866394 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:38.866406 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.867868 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.867874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:38.868035 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:38.868065 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 23:35:38.868862 (+ 797us) client_negotiation.cc:240] Negotiation successful
0926 23:35:38.869017 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":28,"thread_start_us":538,"threads_started":1}
I20260926 23:35:38.869936 18995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e0b8bd74d8644adbf5c0b6d3dd11ef1" candidate_uuid: "8726ea4e193140e0a4bf2d59ddee1b0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5efbb39a5374eba87b14ec37ccf9220" is_pre_election: true
I20260926 23:35:38.870512 19129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e0b8bd74d8644adbf5c0b6d3dd11ef1" candidate_uuid: "8726ea4e193140e0a4bf2d59ddee1b0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88fe30aee6e04f01a91662cd462743a3" is_pre_election: true
I20260926 23:35:38.870723 18995 raft_consensus.cc:2454] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P f5efbb39a5374eba87b14ec37ccf9220 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8726ea4e193140e0a4bf2d59ddee1b0c in term 0.
I20260926 23:35:38.871212 19129 raft_consensus.cc:2454] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 88fe30aee6e04f01a91662cd462743a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8726ea4e193140e0a4bf2d59ddee1b0c in term 0.
I20260926 23:35:38.871654 18795 leader_election.cc:302] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8726ea4e193140e0a4bf2d59ddee1b0c, f5efbb39a5374eba87b14ec37ccf9220; no voters:
I20260926 23:35:38.872128 19200 raft_consensus.cc:2781] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:35:38.872283 19200 raft_consensus.cc:482] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:35:38.872394 19200 raft_consensus.cc:3037] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:35:38.875133 19200 raft_consensus.cc:504] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5efbb39a5374eba87b14ec37ccf9220" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36317 } } peers { permanent_uuid: "8726ea4e193140e0a4bf2d59ddee1b0c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44763 } } peers { permanent_uuid: "88fe30aee6e04f01a91662cd462743a3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33933 } }
I20260926 23:35:38.875854 19200 leader_election.cc:288] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c [CANDIDATE]: Term 1 election: Requested vote from peers f5efbb39a5374eba87b14ec37ccf9220 (127.0.97.2:36317), 88fe30aee6e04f01a91662cd462743a3 (127.0.97.3:33933)
I20260926 23:35:38.876488 18995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e0b8bd74d8644adbf5c0b6d3dd11ef1" candidate_uuid: "8726ea4e193140e0a4bf2d59ddee1b0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5efbb39a5374eba87b14ec37ccf9220"
I20260926 23:35:38.876520 19129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e0b8bd74d8644adbf5c0b6d3dd11ef1" candidate_uuid: "8726ea4e193140e0a4bf2d59ddee1b0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88fe30aee6e04f01a91662cd462743a3"
I20260926 23:35:38.876749 19129 raft_consensus.cc:3037] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 88fe30aee6e04f01a91662cd462743a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:35:38.876722 18995 raft_consensus.cc:3037] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P f5efbb39a5374eba87b14ec37ccf9220 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:35:38.879460 18995 raft_consensus.cc:2454] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P f5efbb39a5374eba87b14ec37ccf9220 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8726ea4e193140e0a4bf2d59ddee1b0c in term 1.
I20260926 23:35:38.879472 19129 raft_consensus.cc:2454] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 88fe30aee6e04f01a91662cd462743a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8726ea4e193140e0a4bf2d59ddee1b0c in term 1.
I20260926 23:35:38.880049 18795 leader_election.cc:302] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8726ea4e193140e0a4bf2d59ddee1b0c, f5efbb39a5374eba87b14ec37ccf9220; no voters:
I20260926 23:35:38.880434 19200 raft_consensus.cc:2781] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:35:38.880975 19200 raft_consensus.cc:686] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c [term 1 LEADER]: Becoming Leader. State: Replica: 8726ea4e193140e0a4bf2d59ddee1b0c, State: Running, Role: LEADER
I20260926 23:35:38.881654 19200 consensus_queue.cc:237] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5efbb39a5374eba87b14ec37ccf9220" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36317 } } peers { permanent_uuid: "8726ea4e193140e0a4bf2d59ddee1b0c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44763 } } peers { permanent_uuid: "88fe30aee6e04f01a91662cd462743a3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33933 } }
I20260926 23:35:38.887560 18717 catalog_manager.cc:5495] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8726ea4e193140e0a4bf2d59ddee1b0c (127.0.97.1). New cstate: current_term: 1 leader_uuid: "8726ea4e193140e0a4bf2d59ddee1b0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5efbb39a5374eba87b14ec37ccf9220" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8726ea4e193140e0a4bf2d59ddee1b0c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88fe30aee6e04f01a91662cd462743a3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33933 } health_report { overall_health: UNKNOWN } } }
I20260926 23:35:38.908314 19190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:38.901664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52310 (local address 127.0.97.1:44763)
0926 23:35:38.901872 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:38.901877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:38.901896 (+ 19us) server_negotiation.cc:407] Connection header received
0926 23:35:38.902022 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:38.902026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:38.902153 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:38.902294 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:35:38.902875 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.903727 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.905395 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.905899 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.906015 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:38.906637 (+ 622us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:35:38.907442 (+ 805us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:35:38.907615 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:38.907901 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 23:35:38.907969 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 23:35:38.910854 19175 tablet.cc:2369] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 88fe30aee6e04f01a91662cd462743a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:35:38.919324 18995 raft_consensus.cc:1261] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P f5efbb39a5374eba87b14ec37ccf9220 [term 1 FOLLOWER]: Refusing update from remote peer 8726ea4e193140e0a4bf2d59ddee1b0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:35:38.919324 19129 raft_consensus.cc:1261] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 88fe30aee6e04f01a91662cd462743a3 [term 1 FOLLOWER]: Refusing update from remote peer 8726ea4e193140e0a4bf2d59ddee1b0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:35:38.920395 19204 consensus_queue.cc:1035] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5efbb39a5374eba87b14ec37ccf9220" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:35:38.920799 19200 consensus_queue.cc:1035] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "88fe30aee6e04f01a91662cd462743a3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 23:35:38.933532 18907 tablet.cc:2369] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:35:38.952498 18715 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:35:38.952941 18715 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:35:38.957816 18715 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0c4aca58afa42439b8d22b3277b7eae: Sending DeleteTablet for 3 replicas of tablet 3e0b8bd74d8644adbf5c0b6d3dd11ef1
I20260926 23:35:38.959403 18841 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e0b8bd74d8644adbf5c0b6d3dd11ef1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:35:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52300
I20260926 23:35:38.959386 18975 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e0b8bd74d8644adbf5c0b6d3dd11ef1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:35:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47250
I20260926 23:35:38.959860 19109 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e0b8bd74d8644adbf5c0b6d3dd11ef1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:35:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33552
I20260926 23:35:38.962448 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 18778
I20260926 23:35:38.978216 19217 tablet_replica.cc:330] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c: stopping tablet replica
I20260926 23:35:38.979281 19217 raft_consensus.cc:2229] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:35:38.980321 19217 raft_consensus.cc:2258] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:35:38.983389 19217 ts_tablet_manager.cc:1905] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:35:38.984442 19218 tablet_replica.cc:330] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P f5efbb39a5374eba87b14ec37ccf9220: stopping tablet replica
I20260926 23:35:38.985149 19219 tablet_replica.cc:330] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 88fe30aee6e04f01a91662cd462743a3: stopping tablet replica
I20260926 23:35:38.985865 19219 raft_consensus.cc:2229] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 88fe30aee6e04f01a91662cd462743a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:35:38.986537 19218 raft_consensus.cc:2229] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P f5efbb39a5374eba87b14ec37ccf9220 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:35:38.987316 19218 raft_consensus.cc:2258] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P f5efbb39a5374eba87b14ec37ccf9220 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:35:38.987931 19219 raft_consensus.cc:2258] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 88fe30aee6e04f01a91662cd462743a3 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 26 23:35:38 dist-test-slave-gwfx krb5kdc[18672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:35:38.991281 19218 ts_tablet_manager.cc:1905] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P f5efbb39a5374eba87b14ec37ccf9220: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:35:38.994046 19217 ts_tablet_manager.cc:1918] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:35:38.994333 19217 log.cc:1196] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-0/wal/wals/3e0b8bd74d8644adbf5c0b6d3dd11ef1
I20260926 23:35:38.995051 19217 ts_tablet_manager.cc:1939] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 8726ea4e193140e0a4bf2d59ddee1b0c: Deleting consensus metadata
I20260926 23:35:38.997709 18701 catalog_manager.cc:4841] TS 8726ea4e193140e0a4bf2d59ddee1b0c (127.0.97.1:44763): tablet 3e0b8bd74d8644adbf5c0b6d3dd11ef1 (table test-table [id=b123d236a5d240f59303a3e4017856fe]) successfully deleted
I20260926 23:35:38.998996 19219 ts_tablet_manager.cc:1905] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 88fe30aee6e04f01a91662cd462743a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:35:38.999727 19218 ts_tablet_manager.cc:1918] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P f5efbb39a5374eba87b14ec37ccf9220: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:35:38.999989 19218 log.cc:1196] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P f5efbb39a5374eba87b14ec37ccf9220: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-1/wal/wals/3e0b8bd74d8644adbf5c0b6d3dd11ef1
I20260926 23:35:39.000581 19218 ts_tablet_manager.cc:1939] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P f5efbb39a5374eba87b14ec37ccf9220: Deleting consensus metadata
I20260926 23:35:39.003170 19190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:38.964193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52326 (local address 127.0.97.1:44763)
0926 23:35:38.964416 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:38.964423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:38.967180 (+ 2757us) server_negotiation.cc:407] Connection header received
0926 23:35:38.968438 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:38.968446 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:38.968563 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:38.968653 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:38.978951 (+ 10298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.980286 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.982431 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:38.983119 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:38.983225 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:38.991786 (+ 8561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:38.991805 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:38.991828 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:35:38.991901 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:38.996946 (+ 5045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:38.998923 (+ 1977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:38.998937 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:38.998941 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:38.999189 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:38.999812 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:38.999818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:38.999822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:39.000416 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:39.000799 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:39.002344 (+ 1545us) server_negotiation.cc:299] Negotiation successful
0926 23:35:39.002731 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 23:35:39.003245 18702 catalog_manager.cc:4841] TS f5efbb39a5374eba87b14ec37ccf9220 (127.0.97.2:36317): tablet 3e0b8bd74d8644adbf5c0b6d3dd11ef1 (table test-table [id=b123d236a5d240f59303a3e4017856fe]) successfully deleted
I20260926 23:35:39.004366 18901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:35:39.008141 19219 ts_tablet_manager.cc:1918] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 88fe30aee6e04f01a91662cd462743a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:35:39.008466 19219 log.cc:1196] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 88fe30aee6e04f01a91662cd462743a3: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790465400694755-388-0/minicluster-data/ts-2/wal/wals/3e0b8bd74d8644adbf5c0b6d3dd11ef1
I20260926 23:35:39.009311 19219 ts_tablet_manager.cc:1939] T 3e0b8bd74d8644adbf5c0b6d3dd11ef1 P 88fe30aee6e04f01a91662cd462743a3: Deleting consensus metadata
I20260926 23:35:39.012147 18702 catalog_manager.cc:4841] TS 88fe30aee6e04f01a91662cd462743a3 (127.0.97.3:33933): tablet 3e0b8bd74d8644adbf5c0b6d3dd11ef1 (table test-table [id=b123d236a5d240f59303a3e4017856fe]) successfully deleted
I20260926 23:35:39.118014 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 18778
I20260926 23:35:39.139267 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 18912
Sep 26 23:35:39 dist-test-slave-gwfx krb5kdc[18672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:35:39.157482 19189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:39.140501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47260 (local address 127.0.97.2:36317)
0926 23:35:39.140693 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:39.140698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:39.140719 (+ 21us) server_negotiation.cc:407] Connection header received
0926 23:35:39.140907 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:39.140911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:39.141008 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:39.141098 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:39.142838 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:39.143696 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:39.145092 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:39.145527 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:39.145630 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:39.150460 (+ 4830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:39.150484 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:39.150506 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:35:39.150563 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:39.154175 (+ 3612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:39.155184 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:39.155195 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:39.155199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:39.155368 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:39.155749 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:39.155752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:39.155755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:39.156306 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:39.156596 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:39.156920 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 23:35:39.157142 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 23:35:39.158329 19035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:35:39.257231 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 18912
I20260926 23:35:39.275076 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 19046
Sep 26 23:35:39 dist-test-slave-gwfx krb5kdc[18672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:35:39.294068 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:39.276076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33560 (local address 127.0.97.3:33933)
0926 23:35:39.276263 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:39.276268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:39.276290 (+ 22us) server_negotiation.cc:407] Connection header received
0926 23:35:39.276475 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:39.276481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:39.276591 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:39.276691 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:39.278158 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:39.279054 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:39.280518 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:39.280954 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:39.281060 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:39.286474 (+ 5414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:39.286496 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:39.286518 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:35:39.286577 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:39.289958 (+ 3381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:39.291293 (+ 1335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:39.291303 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:39.291305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:39.291523 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:39.292054 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:39.292059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:39.292061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:39.292565 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:39.292841 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:39.293339 (+ 498us) server_negotiation.cc:299] Negotiation successful
0926 23:35:39.293613 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 23:35:39.294970 19169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:35:39.392886 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 19046
I20260926 23:35:39.409355 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 18685
I20260926 23:35:39.410794 18747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:35:39.539512 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 18685
2026-09-26T23:35:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4834 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T23:35:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:35:39Z Disabled control of system clock
I20260926 23:35:39.629431 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42263
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 23:35:40.064729 19228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:35:40.065178 19228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:35:40.065284 19228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:35:40.074007 19228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:35:40.074191 19228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:35:40.074239 19228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:35:40.074316 19228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:35:40.074378 19228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:35:40.085677 19228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42263
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:35:40.087749 19228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:35:40.090457 19228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:35:40.105762 19233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:40.105772 19234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:40.105918 19236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:35:40.106778 19228 server_base.cc:1034] running on GCE node
I20260926 23:35:40.107853 19228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:35:40.110255 19228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:35:40.111547 19228 hybrid_clock.cc:648] HybridClock initialized: now 1790465740111465 us; error 78 us; skew 500 ppm
I20260926 23:35:40.112109 19228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:35:40.122210 19228 webserver.cc:460] Webserver started at http://127.0.97.62:38769/ using document root <none> and password file <none>
I20260926 23:35:40.123420 19228 fs_manager.cc:359] Metadata directory not provided
I20260926 23:35:40.123569 19228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:35:40.123996 19228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:35:40.128180 19228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "5528d7b2bf0d4988b34eff65c340180c"
format_stamp: "Formatted at 2026-09-26 23:35:40 on dist-test-slave-gwfx"
server_key: "e7a5edce785aa1be2450db9431de3450"
server_key_iv: "d362f09fb6da06bdb410e6c92ffb144e"
server_key_version: "encryptionkey@0"
I20260926 23:35:40.129433 19228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "5528d7b2bf0d4988b34eff65c340180c"
format_stamp: "Formatted at 2026-09-26 23:35:40 on dist-test-slave-gwfx"
server_key: "e7a5edce785aa1be2450db9431de3450"
server_key_iv: "d362f09fb6da06bdb410e6c92ffb144e"
server_key_version: "encryptionkey@0"
I20260926 23:35:40.137281 19228 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260926 23:35:40.142877 19242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:40.145490 19228 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260926 23:35:40.145927 19228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "5528d7b2bf0d4988b34eff65c340180c"
format_stamp: "Formatted at 2026-09-26 23:35:40 on dist-test-slave-gwfx"
server_key: "e7a5edce785aa1be2450db9431de3450"
server_key_iv: "d362f09fb6da06bdb410e6c92ffb144e"
server_key_version: "encryptionkey@0"
I20260926 23:35:40.146314 19228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:35:40.184731 19228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:35:40.185889 19228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:35:40.186216 19228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:35:40.207789 19228 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42263
I20260926 23:35:40.207814 19293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42263 every 8 connection(s)
I20260926 23:35:40.209707 19228 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:35:40.215668 19294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:35:40.215756 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 19228
I20260926 23:35:40.215948 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:35:40.216463 388 external_mini_cluster.cc:1428] Setting key cd8fc7e452708b940e7af1be1bf41e7a
I20260926 23:35:40.232259 19294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5528d7b2bf0d4988b34eff65c340180c: Bootstrap starting.
I20260926 23:35:40.235186 19297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:40.219108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50310 (local address 127.0.97.62:42263)
0926 23:35:40.220775 (+ 1667us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:40.220791 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:40.220860 (+ 69us) server_negotiation.cc:407] Connection header received
0926 23:35:40.223840 (+ 2980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:40.223895 (+ 55us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:40.224727 (+ 832us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:40.225284 (+ 557us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:40.226098 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:40.227730 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:40.229368 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:40.230279 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:40.230493 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:40.232157 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:40.232167 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:40.232189 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:35:40.232954 (+ 765us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:40.233066 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:35:40.233099 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:40.233222 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:40.233492 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 23:35:40.233906 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":991,"thread_start_us":650,"threads_started":1}
I20260926 23:35:40.237120 19294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5528d7b2bf0d4988b34eff65c340180c: Neither blocks nor log segments found. Creating new log.
I20260926 23:35:40.238445 19294 log.cc:824] T 00000000000000000000000000000000 P 5528d7b2bf0d4988b34eff65c340180c: Log is configured to *not* fsync() on all Append() calls
I20260926 23:35:40.242619 19294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5528d7b2bf0d4988b34eff65c340180c: No bootstrap required, opened a new log
I20260926 23:35:40.252206 19294 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5528d7b2bf0d4988b34eff65c340180c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5528d7b2bf0d4988b34eff65c340180c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42263 } }
I20260926 23:35:40.252789 19294 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5528d7b2bf0d4988b34eff65c340180c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:35:40.252943 19294 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5528d7b2bf0d4988b34eff65c340180c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5528d7b2bf0d4988b34eff65c340180c, State: Initialized, Role: FOLLOWER
I20260926 23:35:40.253695 19294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5528d7b2bf0d4988b34eff65c340180c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5528d7b2bf0d4988b34eff65c340180c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42263 } }
I20260926 23:35:40.253961 19294 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5528d7b2bf0d4988b34eff65c340180c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:35:40.254110 19294 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5528d7b2bf0d4988b34eff65c340180c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:35:40.254298 19294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5528d7b2bf0d4988b34eff65c340180c [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:35:40.257071 19294 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5528d7b2bf0d4988b34eff65c340180c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5528d7b2bf0d4988b34eff65c340180c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42263 } }
I20260926 23:35:40.257786 19294 leader_election.cc:302] T 00000000000000000000000000000000 P 5528d7b2bf0d4988b34eff65c340180c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5528d7b2bf0d4988b34eff65c340180c; no voters:
I20260926 23:35:40.258512 19294 leader_election.cc:288] T 00000000000000000000000000000000 P 5528d7b2bf0d4988b34eff65c340180c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:35:40.258671 19299 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5528d7b2bf0d4988b34eff65c340180c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:35:40.259856 19299 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5528d7b2bf0d4988b34eff65c340180c [term 1 LEADER]: Becoming Leader. State: Replica: 5528d7b2bf0d4988b34eff65c340180c, State: Running, Role: LEADER
I20260926 23:35:40.260540 19299 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5528d7b2bf0d4988b34eff65c340180c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5528d7b2bf0d4988b34eff65c340180c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42263 } }
I20260926 23:35:40.261317 19294 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5528d7b2bf0d4988b34eff65c340180c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:35:40.263887 19301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5528d7b2bf0d4988b34eff65c340180c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5528d7b2bf0d4988b34eff65c340180c. Latest consensus state: current_term: 1 leader_uuid: "5528d7b2bf0d4988b34eff65c340180c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5528d7b2bf0d4988b34eff65c340180c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42263 } } }
I20260926 23:35:40.263898 19300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5528d7b2bf0d4988b34eff65c340180c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5528d7b2bf0d4988b34eff65c340180c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5528d7b2bf0d4988b34eff65c340180c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42263 } } }
I20260926 23:35:40.264668 19300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5528d7b2bf0d4988b34eff65c340180c [sys.catalog]: This master's current role is: LEADER
I20260926 23:35:40.264642 19301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5528d7b2bf0d4988b34eff65c340180c [sys.catalog]: This master's current role is: LEADER
I20260926 23:35:40.265735 19308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:35:40.272888 19308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:35:40.285462 19308 catalog_manager.cc:1347] Generated new cluster ID: 0e58e0314fe0433e96693a02620c0a7f
I20260926 23:35:40.285635 19308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:35:40.301736 19308 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:35:40.303511 19308 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:35:40.311791 19308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5528d7b2bf0d4988b34eff65c340180c: Generated new TSK 0
I20260926 23:35:40.313251 19308 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:35:40.330536 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42263
--builtin_ntp_servers=127.0.97.20:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 23:35:40.758301 19318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:35:40.758754 19318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:35:40.758891 19318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:35:40.767694 19318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:35:40.767850 19318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:35:40.768003 19318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:35:40.778838 19318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:35:40.780638 19318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:35:40.783018 19318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:35:40.799690 19324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:40.799615 19326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:40.799705 19323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:35:40.801466 19318 server_base.cc:1034] running on GCE node
I20260926 23:35:40.802883 19318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:35:40.804559 19318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:35:40.805884 19318 hybrid_clock.cc:648] HybridClock initialized: now 1790465740805815 us; error 77 us; skew 500 ppm
I20260926 23:35:40.806499 19318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:35:40.809917 19318 webserver.cc:460] Webserver started at http://127.0.97.1:35727/ using document root <none> and password file <none>
I20260926 23:35:40.811056 19318 fs_manager.cc:359] Metadata directory not provided
I20260926 23:35:40.811213 19318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:35:40.811623 19318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:35:40.815474 19318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-0/data/instance:
uuid: "eb5c497d820d40719fa0bf2b559dece2"
format_stamp: "Formatted at 2026-09-26 23:35:40 on dist-test-slave-gwfx"
server_key: "20a775374162e21bb9b2db339f467bb6"
server_key_iv: "784cbcd2a57c2d777a38c298fd5873b0"
server_key_version: "encryptionkey@0"
I20260926 23:35:40.816493 19318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-0/wal/instance:
uuid: "eb5c497d820d40719fa0bf2b559dece2"
format_stamp: "Formatted at 2026-09-26 23:35:40 on dist-test-slave-gwfx"
server_key: "20a775374162e21bb9b2db339f467bb6"
server_key_iv: "784cbcd2a57c2d777a38c298fd5873b0"
server_key_version: "encryptionkey@0"
I20260926 23:35:40.824038 19318 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260926 23:35:40.829504 19332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:40.831643 19318 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260926 23:35:40.831892 19318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "eb5c497d820d40719fa0bf2b559dece2"
format_stamp: "Formatted at 2026-09-26 23:35:40 on dist-test-slave-gwfx"
server_key: "20a775374162e21bb9b2db339f467bb6"
server_key_iv: "784cbcd2a57c2d777a38c298fd5873b0"
server_key_version: "encryptionkey@0"
I20260926 23:35:40.832207 19318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:35:40.850194 19318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:35:40.851542 19318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:35:40.851908 19318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:35:40.853286 19318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:35:40.855309 19318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:35:40.855437 19318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:40.855571 19318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:35:40.855643 19318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:40.891814 19318 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35083
I20260926 23:35:40.891839 19444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35083 every 8 connection(s)
I20260926 23:35:40.893602 19318 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:35:40.895435 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 19318
I20260926 23:35:40.895632 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:35:40.896180 388 external_mini_cluster.cc:1428] Setting key 0a8d5f1d6b48c8319398f119b56c519c
I20260926 23:35:40.900489 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42263
--builtin_ntp_servers=127.0.97.20:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 23:35:40.913846 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:40.899248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47587 (local address 127.0.97.62:42263)
0926 23:35:40.899853 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:40.899859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:40.902318 (+ 2459us) server_negotiation.cc:407] Connection header received
0926 23:35:40.904775 (+ 2457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:40.904782 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:40.904909 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:40.905024 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:40.906626 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:40.907537 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:40.909715 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:40.910255 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:40.910421 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:40.912657 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:40.912664 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:40.912668 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:35:40.912699 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:40.912758 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:35:40.912779 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:40.912876 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:40.913232 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 23:35:40.913388 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":338,"threads_started":1}
I20260926 23:35:40.915473 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:40.900497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42263 (local address 127.0.97.1:47587)
0926 23:35:40.902051 (+ 1554us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:40.902116 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:40.904352 (+ 2236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:40.905387 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:40.905400 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:40.905537 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:35:40.906364 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:40.906390 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:40.907781 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:40.907786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:40.909481 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:40.909503 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:40.910598 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:40.910607 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:40.910992 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:40.912059 (+ 1067us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:35:40.912062 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:35:40.912139 (+ 77us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:35:40.912278 (+ 139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:35:40.912899 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:35:40.912927 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:35:40.912936 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 23:35:40.913186 (+ 250us) client_negotiation.cc:240] Negotiation successful
0926 23:35:40.913847 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"thread_start_us":439,"threads_started":1}
I20260926 23:35:40.917724 19445 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42263
I20260926 23:35:40.918169 19445 heartbeater.cc:464] Registering TS with master...
I20260926 23:35:40.919379 19445 heartbeater.cc:511] Master 127.0.97.62:42263 requested a full tablet report, sending...
I20260926 23:35:40.922420 19259 ts_manager.cc:194] Registered new tserver with Master: eb5c497d820d40719fa0bf2b559dece2 (127.0.97.1:35083)
I20260926 23:35:40.924362 19259 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:47587
W20260926 23:35:41.331499 19449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:35:41.331933 19449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:35:41.332075 19449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:35:41.340965 19449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:35:41.341144 19449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:35:41.341323 19449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:35:41.353368 19449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:35:41.355305 19449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:35:41.358039 19449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:35:41.374639 19454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:41.374614 19455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:41.374603 19457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:35:41.375823 19449 server_base.cc:1034] running on GCE node
I20260926 23:35:41.376643 19449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:35:41.377986 19449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:35:41.379256 19449 hybrid_clock.cc:648] HybridClock initialized: now 1790465741379190 us; error 56 us; skew 500 ppm
I20260926 23:35:41.379702 19449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:35:41.382826 19449 webserver.cc:460] Webserver started at http://127.0.97.2:36999/ using document root <none> and password file <none>
I20260926 23:35:41.383761 19449 fs_manager.cc:359] Metadata directory not provided
I20260926 23:35:41.383889 19449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:35:41.384264 19449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:35:41.388204 19449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-1/data/instance:
uuid: "a464f9afb3a14f63bd6c1b1eaa5dc065"
format_stamp: "Formatted at 2026-09-26 23:35:41 on dist-test-slave-gwfx"
server_key: "2af481a0229ab7907c71bd434f8a9c18"
server_key_iv: "4f41fa99668b4b96647e62f460ad98b8"
server_key_version: "encryptionkey@0"
I20260926 23:35:41.389319 19449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a464f9afb3a14f63bd6c1b1eaa5dc065"
format_stamp: "Formatted at 2026-09-26 23:35:41 on dist-test-slave-gwfx"
server_key: "2af481a0229ab7907c71bd434f8a9c18"
server_key_iv: "4f41fa99668b4b96647e62f460ad98b8"
server_key_version: "encryptionkey@0"
I20260926 23:35:41.396026 19449 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260926 23:35:41.400372 19463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:41.402316 19449 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 23:35:41.402547 19449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "a464f9afb3a14f63bd6c1b1eaa5dc065"
format_stamp: "Formatted at 2026-09-26 23:35:41 on dist-test-slave-gwfx"
server_key: "2af481a0229ab7907c71bd434f8a9c18"
server_key_iv: "4f41fa99668b4b96647e62f460ad98b8"
server_key_version: "encryptionkey@0"
I20260926 23:35:41.402904 19449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:35:41.435053 19449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:35:41.436281 19449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:35:41.436612 19449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:35:41.437961 19449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:35:41.440217 19449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:35:41.440368 19449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:41.440502 19449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:35:41.440575 19449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:41.478706 19449 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44283
I20260926 23:35:41.478826 19575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44283 every 8 connection(s)
I20260926 23:35:41.480280 19449 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:35:41.488500 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 19449
I20260926 23:35:41.488713 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:35:41.489254 388 external_mini_cluster.cc:1428] Setting key 00deab8a08b09dba565b976965a0b632
I20260926 23:35:41.492774 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42263
--builtin_ntp_servers=127.0.97.20:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 23:35:41.495898 19578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:41.483333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35209 (local address 127.0.97.62:42263)
0926 23:35:41.483871 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:41.483877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:41.485636 (+ 1759us) server_negotiation.cc:407] Connection header received
0926 23:35:41.487935 (+ 2299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:41.487942 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:41.488070 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:41.488188 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:41.489736 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:41.490672 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:41.492627 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:41.493140 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:41.493262 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:41.494889 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:41.494894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:41.494897 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:35:41.494921 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:41.494969 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:35:41.494983 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:41.495060 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:41.495368 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 23:35:41.495501 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":283,"threads_started":1}
I20260926 23:35:41.497226 19579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:41.484090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42263 (local address 127.0.97.2:35209)
0926 23:35:41.485264 (+ 1174us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:41.485453 (+ 189us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:41.487544 (+ 2091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:41.488521 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:41.488532 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:41.488672 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:35:41.489510 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:41.489538 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:41.490877 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:41.490882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:41.492464 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:41.492479 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:41.493398 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:41.493407 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:41.493665 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:41.494473 (+ 808us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:35:41.494475 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:35:41.494537 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:35:41.494647 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:35:41.495071 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:35:41.495123 (+ 52us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:35:41.495131 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 23:35:41.495328 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 23:35:41.495868 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":316,"threads_started":1}
I20260926 23:35:41.505137 19576 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42263
I20260926 23:35:41.505626 19576 heartbeater.cc:464] Registering TS with master...
I20260926 23:35:41.506965 19576 heartbeater.cc:511] Master 127.0.97.62:42263 requested a full tablet report, sending...
I20260926 23:35:41.509243 19259 ts_manager.cc:194] Registered new tserver with Master: a464f9afb3a14f63bd6c1b1eaa5dc065 (127.0.97.2:44283)
I20260926 23:35:41.510239 19259 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:35209
W20260926 23:35:41.902990 19580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:35:41.903390 19580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:35:41.903553 19580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:35:41.912211 19580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:35:41.912369 19580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:35:41.912533 19580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:35:41.923345 19580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:35:41.925135 19580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:35:41.927695 19580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 23:35:41.929075 19445 heartbeater.cc:503] Master 127.0.97.62:42263 was elected leader, sending a full tablet report...
W20260926 23:35:41.945950 19585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:41.945925 19586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:41.946018 19588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:35:41.946942 19580 server_base.cc:1034] running on GCE node
I20260926 23:35:41.947825 19580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:35:41.949247 19580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:35:41.950527 19580 hybrid_clock.cc:648] HybridClock initialized: now 1790465741950461 us; error 65 us; skew 500 ppm
I20260926 23:35:41.950959 19580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:35:41.954001 19580 webserver.cc:460] Webserver started at http://127.0.97.3:36125/ using document root <none> and password file <none>
I20260926 23:35:41.955030 19580 fs_manager.cc:359] Metadata directory not provided
I20260926 23:35:41.955159 19580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:35:41.955557 19580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:35:41.959906 19580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-2/data/instance:
uuid: "b3264018c1384f80ac934f20c9526da5"
format_stamp: "Formatted at 2026-09-26 23:35:41 on dist-test-slave-gwfx"
server_key: "a90cebd1c999b37ec44a457aee3985c2"
server_key_iv: "89b1531d95ebca0ae48befcbbdfb4f68"
server_key_version: "encryptionkey@0"
I20260926 23:35:41.961064 19580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b3264018c1384f80ac934f20c9526da5"
format_stamp: "Formatted at 2026-09-26 23:35:41 on dist-test-slave-gwfx"
server_key: "a90cebd1c999b37ec44a457aee3985c2"
server_key_iv: "89b1531d95ebca0ae48befcbbdfb4f68"
server_key_version: "encryptionkey@0"
I20260926 23:35:41.968003 19580 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 23:35:41.973126 19594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:41.975584 19580 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260926 23:35:41.975844 19580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "b3264018c1384f80ac934f20c9526da5"
format_stamp: "Formatted at 2026-09-26 23:35:41 on dist-test-slave-gwfx"
server_key: "a90cebd1c999b37ec44a457aee3985c2"
server_key_iv: "89b1531d95ebca0ae48befcbbdfb4f68"
server_key_version: "encryptionkey@0"
I20260926 23:35:41.976153 19580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:35:41.994048 19580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:35:41.995291 19580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:35:41.995612 19580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:35:41.996920 19580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:35:41.998974 19580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:35:41.999099 19580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:41.999233 19580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:35:41.999307 19580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:42.036742 19580 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39871
I20260926 23:35:42.036839 19706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39871 every 8 connection(s)
I20260926 23:35:42.038496 19580 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:35:42.040709 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 19580
I20260926 23:35:42.040915 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:35:42.041512 388 external_mini_cluster.cc:1428] Setting key 8326c1fbe3b39954ee606f50c413afe8
I20260926 23:35:42.055630 19709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:42.042352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52255 (local address 127.0.97.62:42263)
0926 23:35:42.042892 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:42.042899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:42.044652 (+ 1753us) server_negotiation.cc:407] Connection header received
0926 23:35:42.046984 (+ 2332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:42.046989 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:42.047096 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:42.047203 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:42.048735 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.049623 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.052059 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.052553 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.052697 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:42.054540 (+ 1843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:42.054546 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:42.054549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:35:42.054576 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:42.054625 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:35:42.054639 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:42.054756 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:42.055028 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 23:35:42.055169 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":271,"threads_started":1}
I20260926 23:35:42.057138 19710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:42.042930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42263 (local address 127.0.97.3:52255)
0926 23:35:42.044417 (+ 1487us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:42.044485 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:42.046580 (+ 2095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:42.047499 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:42.047511 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:42.047646 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:35:42.048524 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:42.048546 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.049852 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.049863 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:42.051790 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:42.051812 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.052859 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.052868 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:42.053174 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:42.054093 (+ 919us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:35:42.054096 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:35:42.054155 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:35:42.054275 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:35:42.054772 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:35:42.054786 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:35:42.054796 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 23:35:42.055000 (+ 204us) client_negotiation.cc:240] Negotiation successful
0926 23:35:42.055564 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":385,"threads_started":1}
I20260926 23:35:42.059043 19707 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42263
I20260926 23:35:42.059442 19707 heartbeater.cc:464] Registering TS with master...
I20260926 23:35:42.060612 19707 heartbeater.cc:511] Master 127.0.97.62:42263 requested a full tablet report, sending...
I20260926 23:35:42.062994 19259 ts_manager.cc:194] Registered new tserver with Master: b3264018c1384f80ac934f20c9526da5 (127.0.97.3:39871)
I20260926 23:35:42.063989 19259 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:52255
I20260926 23:35:42.071910 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:35:42.084378 19709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:42.076834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50316 (local address 127.0.97.62:42263)
0926 23:35:42.077217 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:42.077224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:42.077635 (+ 411us) server_negotiation.cc:407] Connection header received
0926 23:35:42.077857 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:42.077863 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:42.077948 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:42.078046 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:42.078642 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.079515 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.081238 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.082011 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.082143 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:42.083526 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:42.083531 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:42.083535 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:35:42.083562 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:42.083607 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:35:42.083621 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:42.083819 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:42.083982 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 23:35:42.084052 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260926 23:35:42.089499 19259 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:35:42.097828 19259 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:35:42.141466 19721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:42.115413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35083 (local address 127.0.0.1:59078)
0926 23:35:42.129260 (+ 13847us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:42.129350 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:42.129682 (+ 332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:42.133033 (+ 3351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:42.133043 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:42.133078 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:35:42.133618 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:42.133637 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.135820 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.135826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:42.137756 (+ 1930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:42.137778 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.140272 (+ 2494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.140280 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:42.140523 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:42.140587 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 23:35:42.140760 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 23:35:42.141024 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13372,"thread_start_us":18637,"threads_started":1}
I20260926 23:35:42.142946 19722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:42.115717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59078 (local address 127.0.97.1:35083)
0926 23:35:42.130541 (+ 14824us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:42.130554 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:42.130601 (+ 47us) server_negotiation.cc:407] Connection header received
0926 23:35:42.131623 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:42.131636 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:42.132072 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:42.132388 (+ 316us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:35:42.133926 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.135636 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.138044 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.139583 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.139757 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:42.140123 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:42.140975 (+ 852us) server_negotiation.cc:299] Negotiation successful
0926 23:35:42.141224 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14567,"thread_start_us":14385,"threads_started":1}
I20260926 23:35:42.144317 19723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:42.115657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39871 (local address 127.0.0.1:34102)
0926 23:35:42.132515 (+ 16858us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:42.132544 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:42.132720 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:42.134835 (+ 2115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:42.134840 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:42.134869 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:35:42.135347 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:42.135363 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.138098 (+ 2735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.138104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:42.140910 (+ 2806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:42.140928 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.143282 (+ 2354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.143288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:42.143513 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:42.143556 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 23:35:42.143643 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 23:35:42.143909 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16674,"mutex_wait_us":163,"thread_start_us":12917,"threads_started":1}
I20260926 23:35:42.144763 19724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:42.116407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34102 (local address 127.0.97.3:39871)
0926 23:35:42.133973 (+ 17566us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:42.133983 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:42.134036 (+ 53us) server_negotiation.cc:407] Connection header received
0926 23:35:42.134161 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:42.134171 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:42.134533 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:42.135589 (+ 1056us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:35:42.135943 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.137922 (+ 1979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.141211 (+ 3289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.142910 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.143687 (+ 777us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:42.143987 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:42.144080 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 23:35:42.144342 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17386,"thread_start_us":17204,"threads_started":1}
I20260926 23:35:42.144340 19720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:42.116428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44283 (local address 127.0.0.1:39890)
0926 23:35:42.134360 (+ 17932us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:42.134381 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:42.134522 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:42.135630 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:42.135634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:42.135659 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:35:42.136145 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:42.136158 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.138098 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.138103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:42.141388 (+ 3285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:42.141405 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.143428 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.143435 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:42.143612 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:42.143642 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 23:35:42.143718 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 23:35:42.143856 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17800,"mutex_wait_us":737,"thread_start_us":16214,"threads_started":1}
I20260926 23:35:42.145007 19725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:42.116597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39890 (local address 127.0.97.2:44283)
0926 23:35:42.133386 (+ 16789us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:42.133399 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:42.134670 (+ 1271us) server_negotiation.cc:407] Connection header received
0926 23:35:42.134836 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:42.134844 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:42.135066 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:42.135312 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:35:42.136370 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.137920 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.141670 (+ 3750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.143201 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.143364 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:42.143639 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:42.144459 (+ 820us) server_negotiation.cc:299] Negotiation successful
0926 23:35:42.144643 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16569,"thread_start_us":16407,"threads_started":1}
I20260926 23:35:42.149084 19511 tablet_service.cc:1467] Processing CreateTablet for tablet 0f321370a1dc4962a137b0466dd89693 (DEFAULT_TABLE table=test-table [id=0f93c823cd36450a829bd8b190ef00d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:35:42.149099 19380 tablet_service.cc:1467] Processing CreateTablet for tablet 0f321370a1dc4962a137b0466dd89693 (DEFAULT_TABLE table=test-table [id=0f93c823cd36450a829bd8b190ef00d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:35:42.149524 19642 tablet_service.cc:1467] Processing CreateTablet for tablet 0f321370a1dc4962a137b0466dd89693 (DEFAULT_TABLE table=test-table [id=0f93c823cd36450a829bd8b190ef00d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:35:42.150988 19511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f321370a1dc4962a137b0466dd89693. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:35:42.151046 19380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f321370a1dc4962a137b0466dd89693. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:35:42.151068 19642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f321370a1dc4962a137b0466dd89693. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:35:42.167299 19726 tablet_bootstrap.cc:492] T 0f321370a1dc4962a137b0466dd89693 P a464f9afb3a14f63bd6c1b1eaa5dc065: Bootstrap starting.
I20260926 23:35:42.171370 19728 tablet_bootstrap.cc:492] T 0f321370a1dc4962a137b0466dd89693 P b3264018c1384f80ac934f20c9526da5: Bootstrap starting.
I20260926 23:35:42.170610 19727 tablet_bootstrap.cc:492] T 0f321370a1dc4962a137b0466dd89693 P eb5c497d820d40719fa0bf2b559dece2: Bootstrap starting.
I20260926 23:35:42.173722 19726 tablet_bootstrap.cc:654] T 0f321370a1dc4962a137b0466dd89693 P a464f9afb3a14f63bd6c1b1eaa5dc065: Neither blocks nor log segments found. Creating new log.
I20260926 23:35:42.175388 19727 tablet_bootstrap.cc:654] T 0f321370a1dc4962a137b0466dd89693 P eb5c497d820d40719fa0bf2b559dece2: Neither blocks nor log segments found. Creating new log.
I20260926 23:35:42.175423 19728 tablet_bootstrap.cc:654] T 0f321370a1dc4962a137b0466dd89693 P b3264018c1384f80ac934f20c9526da5: Neither blocks nor log segments found. Creating new log.
I20260926 23:35:42.175388 19726 log.cc:824] T 0f321370a1dc4962a137b0466dd89693 P a464f9afb3a14f63bd6c1b1eaa5dc065: Log is configured to *not* fsync() on all Append() calls
I20260926 23:35:42.176635 19728 log.cc:824] T 0f321370a1dc4962a137b0466dd89693 P b3264018c1384f80ac934f20c9526da5: Log is configured to *not* fsync() on all Append() calls
I20260926 23:35:42.176663 19727 log.cc:824] T 0f321370a1dc4962a137b0466dd89693 P eb5c497d820d40719fa0bf2b559dece2: Log is configured to *not* fsync() on all Append() calls
I20260926 23:35:42.179584 19726 tablet_bootstrap.cc:492] T 0f321370a1dc4962a137b0466dd89693 P a464f9afb3a14f63bd6c1b1eaa5dc065: No bootstrap required, opened a new log
I20260926 23:35:42.179939 19726 ts_tablet_manager.cc:1397] T 0f321370a1dc4962a137b0466dd89693 P a464f9afb3a14f63bd6c1b1eaa5dc065: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260926 23:35:42.180003 19728 tablet_bootstrap.cc:492] T 0f321370a1dc4962a137b0466dd89693 P b3264018c1384f80ac934f20c9526da5: No bootstrap required, opened a new log
I20260926 23:35:42.180107 19727 tablet_bootstrap.cc:492] T 0f321370a1dc4962a137b0466dd89693 P eb5c497d820d40719fa0bf2b559dece2: No bootstrap required, opened a new log
I20260926 23:35:42.180311 19728 ts_tablet_manager.cc:1397] T 0f321370a1dc4962a137b0466dd89693 P b3264018c1384f80ac934f20c9526da5: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 23:35:42.180383 19727 ts_tablet_manager.cc:1397] T 0f321370a1dc4962a137b0466dd89693 P eb5c497d820d40719fa0bf2b559dece2: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260926 23:35:42.189244 19726 raft_consensus.cc:348] T 0f321370a1dc4962a137b0466dd89693 P a464f9afb3a14f63bd6c1b1eaa5dc065 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb5c497d820d40719fa0bf2b559dece2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35083 } } peers { permanent_uuid: "b3264018c1384f80ac934f20c9526da5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39871 } } peers { permanent_uuid: "a464f9afb3a14f63bd6c1b1eaa5dc065" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44283 } }
I20260926 23:35:42.189961 19726 raft_consensus.cc:374] T 0f321370a1dc4962a137b0466dd89693 P a464f9afb3a14f63bd6c1b1eaa5dc065 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:35:42.190147 19726 raft_consensus.cc:729] T 0f321370a1dc4962a137b0466dd89693 P a464f9afb3a14f63bd6c1b1eaa5dc065 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a464f9afb3a14f63bd6c1b1eaa5dc065, State: Initialized, Role: FOLLOWER
I20260926 23:35:42.190970 19726 consensus_queue.cc:260] T 0f321370a1dc4962a137b0466dd89693 P a464f9afb3a14f63bd6c1b1eaa5dc065 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb5c497d820d40719fa0bf2b559dece2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35083 } } peers { permanent_uuid: "b3264018c1384f80ac934f20c9526da5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39871 } } peers { permanent_uuid: "a464f9afb3a14f63bd6c1b1eaa5dc065" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44283 } }
I20260926 23:35:42.191988 19727 raft_consensus.cc:348] T 0f321370a1dc4962a137b0466dd89693 P eb5c497d820d40719fa0bf2b559dece2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb5c497d820d40719fa0bf2b559dece2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35083 } } peers { permanent_uuid: "b3264018c1384f80ac934f20c9526da5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39871 } } peers { permanent_uuid: "a464f9afb3a14f63bd6c1b1eaa5dc065" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44283 } }
I20260926 23:35:42.192456 19576 heartbeater.cc:503] Master 127.0.97.62:42263 was elected leader, sending a full tablet report...
I20260926 23:35:42.192556 19727 raft_consensus.cc:374] T 0f321370a1dc4962a137b0466dd89693 P eb5c497d820d40719fa0bf2b559dece2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:35:42.192660 19727 raft_consensus.cc:729] T 0f321370a1dc4962a137b0466dd89693 P eb5c497d820d40719fa0bf2b559dece2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb5c497d820d40719fa0bf2b559dece2, State: Initialized, Role: FOLLOWER
I20260926 23:35:42.193290 19726 ts_tablet_manager.cc:1428] T 0f321370a1dc4962a137b0466dd89693 P a464f9afb3a14f63bd6c1b1eaa5dc065: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260926 23:35:42.193378 19727 consensus_queue.cc:260] T 0f321370a1dc4962a137b0466dd89693 P eb5c497d820d40719fa0bf2b559dece2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb5c497d820d40719fa0bf2b559dece2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35083 } } peers { permanent_uuid: "b3264018c1384f80ac934f20c9526da5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39871 } } peers { permanent_uuid: "a464f9afb3a14f63bd6c1b1eaa5dc065" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44283 } }
I20260926 23:35:42.191984 19728 raft_consensus.cc:348] T 0f321370a1dc4962a137b0466dd89693 P b3264018c1384f80ac934f20c9526da5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb5c497d820d40719fa0bf2b559dece2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35083 } } peers { permanent_uuid: "b3264018c1384f80ac934f20c9526da5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39871 } } peers { permanent_uuid: "a464f9afb3a14f63bd6c1b1eaa5dc065" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44283 } }
I20260926 23:35:42.194008 19728 raft_consensus.cc:374] T 0f321370a1dc4962a137b0466dd89693 P b3264018c1384f80ac934f20c9526da5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:35:42.194289 19728 raft_consensus.cc:729] T 0f321370a1dc4962a137b0466dd89693 P b3264018c1384f80ac934f20c9526da5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3264018c1384f80ac934f20c9526da5, State: Initialized, Role: FOLLOWER
I20260926 23:35:42.195194 19728 consensus_queue.cc:260] T 0f321370a1dc4962a137b0466dd89693 P b3264018c1384f80ac934f20c9526da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb5c497d820d40719fa0bf2b559dece2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35083 } } peers { permanent_uuid: "b3264018c1384f80ac934f20c9526da5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39871 } } peers { permanent_uuid: "a464f9afb3a14f63bd6c1b1eaa5dc065" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44283 } }
I20260926 23:35:42.195590 19727 ts_tablet_manager.cc:1428] T 0f321370a1dc4962a137b0466dd89693 P eb5c497d820d40719fa0bf2b559dece2: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260926 23:35:42.196623 19707 heartbeater.cc:503] Master 127.0.97.62:42263 was elected leader, sending a full tablet report...
I20260926 23:35:42.197325 19728 ts_tablet_manager.cc:1428] T 0f321370a1dc4962a137b0466dd89693 P b3264018c1384f80ac934f20c9526da5: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
W20260926 23:35:42.232805 19577 tablet.cc:2369] T 0f321370a1dc4962a137b0466dd89693 P a464f9afb3a14f63bd6c1b1eaa5dc065: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:35:42.290753 19708 tablet.cc:2369] T 0f321370a1dc4962a137b0466dd89693 P b3264018c1384f80ac934f20c9526da5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 23:35:42.400868 19446 tablet.cc:2369] T 0f321370a1dc4962a137b0466dd89693 P eb5c497d820d40719fa0bf2b559dece2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:35:42.439594 19733 raft_consensus.cc:482] T 0f321370a1dc4962a137b0466dd89693 P eb5c497d820d40719fa0bf2b559dece2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:35:42.439970 19733 raft_consensus.cc:504] T 0f321370a1dc4962a137b0466dd89693 P eb5c497d820d40719fa0bf2b559dece2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb5c497d820d40719fa0bf2b559dece2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35083 } } peers { permanent_uuid: "b3264018c1384f80ac934f20c9526da5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39871 } } peers { permanent_uuid: "a464f9afb3a14f63bd6c1b1eaa5dc065" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44283 } }
I20260926 23:35:42.442193 19733 leader_election.cc:288] T 0f321370a1dc4962a137b0466dd89693 P eb5c497d820d40719fa0bf2b559dece2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3264018c1384f80ac934f20c9526da5 (127.0.97.3:39871), a464f9afb3a14f63bd6c1b1eaa5dc065 (127.0.97.2:44283)
I20260926 23:35:42.450860 19724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:42.443070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37965 (local address 127.0.97.3:39871)
0926 23:35:42.443259 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:42.443265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:42.444203 (+ 938us) server_negotiation.cc:407] Connection header received
0926 23:35:42.444339 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:42.444345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:42.444453 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:42.444724 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:35:42.445703 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.446453 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.448766 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.449702 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.449827 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:42.449928 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:42.450344 (+ 416us) server_negotiation.cc:299] Negotiation successful
0926 23:35:42.450504 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 23:35:42.450913 19725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:42.442815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35177 (local address 127.0.97.2:44283)
0926 23:35:42.443066 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:42.443072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:42.444088 (+ 1016us) server_negotiation.cc:407] Connection header received
0926 23:35:42.444432 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:42.444437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:42.444529 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:42.444629 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:35:42.445655 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.446445 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.448968 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.449820 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.449977 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:42.450084 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:42.450457 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 23:35:42.450618 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 23:35:42.451020 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:42.442778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39871 (local address 127.0.97.1:37965)
0926 23:35:42.443894 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:42.443910 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:42.444131 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:42.444741 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:42.444749 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:42.444834 (+ 85us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:35:42.445479 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:42.445491 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.446625 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.446630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:42.448455 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:42.448476 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.449952 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.449959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:42.450146 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:42.450182 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 23:35:42.450268 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 23:35:42.450423 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1013,"mutex_wait_us":108,"spinlock_wait_cycles":89984,"thread_start_us":526,"threads_started":1}
I20260926 23:35:42.451078 19735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:42.442787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44283 (local address 127.0.97.1:35177)
0926 23:35:42.443861 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:42.443889 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:42.444189 (+ 300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:42.444744 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:42.444751 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:42.444828 (+ 77us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:35:42.445439 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:42.445459 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.446614 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.446619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:42.448619 (+ 2000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:42.448647 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.450073 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.450079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:42.450235 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:42.450273 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 23:35:42.450380 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 23:35:42.450533 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"mutex_wait_us":3,"spinlock_wait_cycles":36992,"thread_start_us":651,"threads_started":1}
I20260926 23:35:42.452581 19531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f321370a1dc4962a137b0466dd89693" candidate_uuid: "eb5c497d820d40719fa0bf2b559dece2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a464f9afb3a14f63bd6c1b1eaa5dc065" is_pre_election: true
I20260926 23:35:42.452961 19662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f321370a1dc4962a137b0466dd89693" candidate_uuid: "eb5c497d820d40719fa0bf2b559dece2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3264018c1384f80ac934f20c9526da5" is_pre_election: true
I20260926 23:35:42.453258 19531 raft_consensus.cc:2454] T 0f321370a1dc4962a137b0466dd89693 P a464f9afb3a14f63bd6c1b1eaa5dc065 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb5c497d820d40719fa0bf2b559dece2 in term 0.
I20260926 23:35:42.453649 19662 raft_consensus.cc:2454] T 0f321370a1dc4962a137b0466dd89693 P b3264018c1384f80ac934f20c9526da5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb5c497d820d40719fa0bf2b559dece2 in term 0.
I20260926 23:35:42.454123 19334 leader_election.cc:302] T 0f321370a1dc4962a137b0466dd89693 P eb5c497d820d40719fa0bf2b559dece2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a464f9afb3a14f63bd6c1b1eaa5dc065, eb5c497d820d40719fa0bf2b559dece2; no voters:
I20260926 23:35:42.454727 19733 raft_consensus.cc:2781] T 0f321370a1dc4962a137b0466dd89693 P eb5c497d820d40719fa0bf2b559dece2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:35:42.454937 19733 raft_consensus.cc:482] T 0f321370a1dc4962a137b0466dd89693 P eb5c497d820d40719fa0bf2b559dece2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:35:42.455072 19733 raft_consensus.cc:3037] T 0f321370a1dc4962a137b0466dd89693 P eb5c497d820d40719fa0bf2b559dece2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:35:42.457777 19733 raft_consensus.cc:504] T 0f321370a1dc4962a137b0466dd89693 P eb5c497d820d40719fa0bf2b559dece2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb5c497d820d40719fa0bf2b559dece2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35083 } } peers { permanent_uuid: "b3264018c1384f80ac934f20c9526da5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39871 } } peers { permanent_uuid: "a464f9afb3a14f63bd6c1b1eaa5dc065" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44283 } }
I20260926 23:35:42.458670 19733 leader_election.cc:288] T 0f321370a1dc4962a137b0466dd89693 P eb5c497d820d40719fa0bf2b559dece2 [CANDIDATE]: Term 1 election: Requested vote from peers b3264018c1384f80ac934f20c9526da5 (127.0.97.3:39871), a464f9afb3a14f63bd6c1b1eaa5dc065 (127.0.97.2:44283)
I20260926 23:35:42.459546 19662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f321370a1dc4962a137b0466dd89693" candidate_uuid: "eb5c497d820d40719fa0bf2b559dece2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3264018c1384f80ac934f20c9526da5"
I20260926 23:35:42.459828 19662 raft_consensus.cc:3037] T 0f321370a1dc4962a137b0466dd89693 P b3264018c1384f80ac934f20c9526da5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:35:42.459710 19531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f321370a1dc4962a137b0466dd89693" candidate_uuid: "eb5c497d820d40719fa0bf2b559dece2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a464f9afb3a14f63bd6c1b1eaa5dc065"
I20260926 23:35:42.460105 19531 raft_consensus.cc:3037] T 0f321370a1dc4962a137b0466dd89693 P a464f9afb3a14f63bd6c1b1eaa5dc065 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:35:42.462997 19662 raft_consensus.cc:2454] T 0f321370a1dc4962a137b0466dd89693 P b3264018c1384f80ac934f20c9526da5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb5c497d820d40719fa0bf2b559dece2 in term 1.
I20260926 23:35:42.463693 19531 raft_consensus.cc:2454] T 0f321370a1dc4962a137b0466dd89693 P a464f9afb3a14f63bd6c1b1eaa5dc065 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb5c497d820d40719fa0bf2b559dece2 in term 1.
I20260926 23:35:42.463806 19333 leader_election.cc:302] T 0f321370a1dc4962a137b0466dd89693 P eb5c497d820d40719fa0bf2b559dece2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3264018c1384f80ac934f20c9526da5, eb5c497d820d40719fa0bf2b559dece2; no voters:
I20260926 23:35:42.464327 19733 raft_consensus.cc:2781] T 0f321370a1dc4962a137b0466dd89693 P eb5c497d820d40719fa0bf2b559dece2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:35:42.465008 19733 raft_consensus.cc:686] T 0f321370a1dc4962a137b0466dd89693 P eb5c497d820d40719fa0bf2b559dece2 [term 1 LEADER]: Becoming Leader. State: Replica: eb5c497d820d40719fa0bf2b559dece2, State: Running, Role: LEADER
I20260926 23:35:42.465821 19733 consensus_queue.cc:237] T 0f321370a1dc4962a137b0466dd89693 P eb5c497d820d40719fa0bf2b559dece2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb5c497d820d40719fa0bf2b559dece2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35083 } } peers { permanent_uuid: "b3264018c1384f80ac934f20c9526da5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39871 } } peers { permanent_uuid: "a464f9afb3a14f63bd6c1b1eaa5dc065" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44283 } }
I20260926 23:35:42.472165 19258 catalog_manager.cc:5495] T 0f321370a1dc4962a137b0466dd89693 P eb5c497d820d40719fa0bf2b559dece2 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb5c497d820d40719fa0bf2b559dece2 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "eb5c497d820d40719fa0bf2b559dece2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb5c497d820d40719fa0bf2b559dece2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3264018c1384f80ac934f20c9526da5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a464f9afb3a14f63bd6c1b1eaa5dc065" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44283 } health_report { overall_health: UNKNOWN } } }
I20260926 23:35:42.523927 19722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:42.517368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59082 (local address 127.0.97.1:35083)
0926 23:35:42.517620 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:42.517627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:42.517652 (+ 25us) server_negotiation.cc:407] Connection header received
0926 23:35:42.517784 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:42.517788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:42.517903 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:42.518058 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:35:42.518516 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.519369 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.521255 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.521725 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.521860 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:42.522540 (+ 680us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:35:42.523263 (+ 723us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:35:42.523454 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:42.523584 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 23:35:42.523637 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 23:35:42.535073 19531 raft_consensus.cc:1261] T 0f321370a1dc4962a137b0466dd89693 P a464f9afb3a14f63bd6c1b1eaa5dc065 [term 1 FOLLOWER]: Refusing update from remote peer eb5c497d820d40719fa0bf2b559dece2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:35:42.535073 19662 raft_consensus.cc:1261] T 0f321370a1dc4962a137b0466dd89693 P b3264018c1384f80ac934f20c9526da5 [term 1 FOLLOWER]: Refusing update from remote peer eb5c497d820d40719fa0bf2b559dece2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:35:42.536792 19737 consensus_queue.cc:1035] T 0f321370a1dc4962a137b0466dd89693 P eb5c497d820d40719fa0bf2b559dece2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a464f9afb3a14f63bd6c1b1eaa5dc065" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:35:42.537446 19733 consensus_queue.cc:1035] T 0f321370a1dc4962a137b0466dd89693 P eb5c497d820d40719fa0bf2b559dece2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3264018c1384f80ac934f20c9526da5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:35:42.552958 19740 mvcc.cc:205] Tried to move back new op lower bound from 7333747681409753088 to 7333747681145724928. Current Snapshot: MvccSnapshot[applied={T|T < 7333747681409753088}]
I20260926 23:35:42.553282 19741 mvcc.cc:205] Tried to move back new op lower bound from 7333747681409753088 to 7333747681145724928. Current Snapshot: MvccSnapshot[applied={T|T < 7333747681409753088}]
I20260926 23:35:42.571461 19258 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:35:42.571885 19258 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:35:42.577359 19258 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5528d7b2bf0d4988b34eff65c340180c: Sending DeleteTablet for 3 replicas of tablet 0f321370a1dc4962a137b0466dd89693
I20260926 23:35:42.578946 19642 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f321370a1dc4962a137b0466dd89693 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:35:42 UTC) from {username='slave'} at 127.0.0.1:34102
I20260926 23:35:42.578946 19511 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f321370a1dc4962a137b0466dd89693 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:35:42 UTC) from {username='slave'} at 127.0.0.1:39890
I20260926 23:35:42.579097 19380 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f321370a1dc4962a137b0466dd89693 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:35:42 UTC) from {username='slave'} at 127.0.0.1:59078
I20260926 23:35:42.584685 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 19318
I20260926 23:35:42.593097 19750 tablet_replica.cc:330] T 0f321370a1dc4962a137b0466dd89693 P b3264018c1384f80ac934f20c9526da5: stopping tablet replica
I20260926 23:35:42.594507 19750 raft_consensus.cc:2229] T 0f321370a1dc4962a137b0466dd89693 P b3264018c1384f80ac934f20c9526da5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:35:42.596037 19750 raft_consensus.cc:2258] T 0f321370a1dc4962a137b0466dd89693 P b3264018c1384f80ac934f20c9526da5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:35:42.599887 19750 ts_tablet_manager.cc:1905] T 0f321370a1dc4962a137b0466dd89693 P b3264018c1384f80ac934f20c9526da5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:35:42.600907 19751 tablet_replica.cc:330] T 0f321370a1dc4962a137b0466dd89693 P a464f9afb3a14f63bd6c1b1eaa5dc065: stopping tablet replica
I20260926 23:35:42.601470 19751 raft_consensus.cc:2229] T 0f321370a1dc4962a137b0466dd89693 P a464f9afb3a14f63bd6c1b1eaa5dc065 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:35:42.602181 19751 raft_consensus.cc:2258] T 0f321370a1dc4962a137b0466dd89693 P a464f9afb3a14f63bd6c1b1eaa5dc065 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:35:42.606139 19752 tablet_replica.cc:330] T 0f321370a1dc4962a137b0466dd89693 P eb5c497d820d40719fa0bf2b559dece2: stopping tablet replica
I20260926 23:35:42.606745 19752 raft_consensus.cc:2229] T 0f321370a1dc4962a137b0466dd89693 P eb5c497d820d40719fa0bf2b559dece2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:35:42.607678 19752 raft_consensus.cc:2258] T 0f321370a1dc4962a137b0466dd89693 P eb5c497d820d40719fa0bf2b559dece2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:35:42.609287 19722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:42.586133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59098 (local address 127.0.97.1:35083)
0926 23:35:42.586332 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:42.586337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:42.598743 (+ 12406us) server_negotiation.cc:407] Connection header received
0926 23:35:42.599302 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:42.599309 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:42.599431 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:42.599514 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:42.600262 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.601449 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.603306 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.603963 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.604081 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:42.606720 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:42.606739 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:42.606767 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:35:42.607935 (+ 1168us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:42.608028 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:35:42.608066 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:42.608216 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:42.608640 (+ 424us) server_negotiation.cc:299] Negotiation successful
0926 23:35:42.608829 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 23:35:42.610186 19751 ts_tablet_manager.cc:1905] T 0f321370a1dc4962a137b0466dd89693 P a464f9afb3a14f63bd6c1b1eaa5dc065: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:35:42.610958 19440 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:35:42.610921 19752 ts_tablet_manager.cc:1905] T 0f321370a1dc4962a137b0466dd89693 P eb5c497d820d40719fa0bf2b559dece2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:35:42.613515 19750 ts_tablet_manager.cc:1918] T 0f321370a1dc4962a137b0466dd89693 P b3264018c1384f80ac934f20c9526da5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:35:42.613829 19750 log.cc:1196] T 0f321370a1dc4962a137b0466dd89693 P b3264018c1384f80ac934f20c9526da5: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-2/wal/wals/0f321370a1dc4962a137b0466dd89693
I20260926 23:35:42.614704 19750 ts_tablet_manager.cc:1939] T 0f321370a1dc4962a137b0466dd89693 P b3264018c1384f80ac934f20c9526da5: Deleting consensus metadata
I20260926 23:35:42.617502 19243 catalog_manager.cc:4841] TS b3264018c1384f80ac934f20c9526da5 (127.0.97.3:39871): tablet 0f321370a1dc4962a137b0466dd89693 (table test-table [id=0f93c823cd36450a829bd8b190ef00d5]) successfully deleted
I20260926 23:35:42.619208 19751 ts_tablet_manager.cc:1918] T 0f321370a1dc4962a137b0466dd89693 P a464f9afb3a14f63bd6c1b1eaa5dc065: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:35:42.619397 19751 log.cc:1196] T 0f321370a1dc4962a137b0466dd89693 P a464f9afb3a14f63bd6c1b1eaa5dc065: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790465400694755-388-0/minicluster-data/ts-1/wal/wals/0f321370a1dc4962a137b0466dd89693
I20260926 23:35:42.619892 19751 ts_tablet_manager.cc:1939] T 0f321370a1dc4962a137b0466dd89693 P a464f9afb3a14f63bd6c1b1eaa5dc065: Deleting consensus metadata
I20260926 23:35:42.622182 19244 catalog_manager.cc:4841] TS a464f9afb3a14f63bd6c1b1eaa5dc065 (127.0.97.2:44283): tablet 0f321370a1dc4962a137b0466dd89693 (table test-table [id=0f93c823cd36450a829bd8b190ef00d5]) successfully deleted
I20260926 23:35:42.714028 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 19318
W20260926 23:35:42.733891 19245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:35083 (error 108)
I20260926 23:35:42.734158 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 19449
I20260926 23:35:42.736510 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:42.735679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35083 (local address 127.0.0.1:59102)
0926 23:35:42.736195 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:42.736321 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":280,"threads_started":1}
W20260926 23:35:42.736881 19245 catalog_manager.cc:4584] TS eb5c497d820d40719fa0bf2b559dece2 (127.0.97.1:35083): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0f321370a1dc4962a137b0466dd89693: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35083: connect: Connection refused (error 111)
I20260926 23:35:42.743597 19725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:42.735180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39892 (local address 127.0.97.2:44283)
0926 23:35:42.735385 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:42.735391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:42.735585 (+ 194us) server_negotiation.cc:407] Connection header received
0926 23:35:42.735851 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:42.735859 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:42.736093 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:42.736233 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:42.736919 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.737770 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.739633 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.740145 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.740305 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:42.741907 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:42.741917 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:42.741934 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:35:42.742665 (+ 731us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:42.742776 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:35:42.742802 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:42.742916 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:42.743096 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 23:35:42.743231 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 23:35:42.744547 19571 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:35:42.791357 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:42.790623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35083 (local address 127.0.0.1:59112)
0926 23:35:42.790999 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:42.791160 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260926 23:35:42.792726 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:42.792186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35083 (local address 127.0.0.1:59126)
0926 23:35:42.792406 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:42.792520 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 23:35:42.844704 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 19449
I20260926 23:35:42.853626 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:42.852973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35083 (local address 127.0.0.1:59134)
0926 23:35:42.853280 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:42.853421 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260926 23:35:42.855119 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:42.854594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35083 (local address 127.0.0.1:59138)
0926 23:35:42.854800 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:42.854906 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 23:35:42.861344 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 19580
I20260926 23:35:42.869899 19724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:42.862671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34112 (local address 127.0.97.3:39871)
0926 23:35:42.862908 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:42.862913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:42.862937 (+ 24us) server_negotiation.cc:407] Connection header received
0926 23:35:42.863061 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:42.863066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:42.863198 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:42.863297 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:42.863767 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.864605 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.866221 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:42.866783 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:42.866927 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:42.868075 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:42.868085 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:42.868104 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:35:42.868903 (+ 799us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:42.868967 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:35:42.868993 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:42.869105 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:42.869309 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 23:35:42.869471 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 23:35:42.870824 19702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:35:42.935957 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:42.935279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35083 (local address 127.0.0.1:59154)
0926 23:35:42.935669 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:42.935794 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260926 23:35:42.937630 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:42.937001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35083 (local address 127.0.0.1:59170)
0926 23:35:42.937321 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:42.937448 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
I20260926 23:35:42.972263 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 19580
I20260926 23:35:42.988169 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 19228
I20260926 23:35:42.989677 19289 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:35:43.122268 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 19228
2026-09-26T23:35:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3579 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:35:43 dist-test-slave-gwfx krb5kdc[19762](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:35:43 dist-test-slave-gwfx krb5kdc[19762](info): set up 2 sockets
Sep 26 23:35:43 dist-test-slave-gwfx krb5kdc[19762](info): commencing operation
krb5kdc: starting...
W20260926 23:35:44.191920 388 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:35:44 dist-test-slave-gwfx krb5kdc[19762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:35:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:35:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:35:44.500046 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42891
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:35:44.911572 19775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:35:44.911943 19775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:35:44.912078 19775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:35:44.920244 19775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:35:44.920395 19775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:35:44.920478 19775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:35:44.920558 19775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:35:44.920609 19775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:35:44.932250 19775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:35:44.933969 19775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:35:44.935768 19775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:35:44.950563 19780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:44.950536 19783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:44.950589 19781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:35:44.951632 19775 server_base.cc:1034] running on GCE node
I20260926 23:35:44.952623 19775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:35:44.954481 19775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:35:44.955739 19775 hybrid_clock.cc:648] HybridClock initialized: now 1790465744955671 us; error 53 us; skew 500 ppm
I20260926 23:35:44.956205 19775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:35:44 dist-test-slave-gwfx krb5kdc[19762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:35:44.960223 19775 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:35:44.962246 19775 webserver.cc:460] Webserver started at http://127.0.97.62:34979/ using document root <none> and password file <none>
I20260926 23:35:44.963228 19775 fs_manager.cc:359] Metadata directory not provided
I20260926 23:35:44.963361 19775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:35:44.963723 19775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:35:44.967646 19775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "d83bbc1bfbbb4620a8958d351026e1ef"
format_stamp: "Formatted at 2026-09-26 23:35:44 on dist-test-slave-gwfx"
server_key: "050f704efac7df80ebbea0d0f98842f4"
server_key_iv: "2d4c69e9385b64f45971e048b49bede2"
server_key_version: "encryptionkey@0"
I20260926 23:35:44.968737 19775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "d83bbc1bfbbb4620a8958d351026e1ef"
format_stamp: "Formatted at 2026-09-26 23:35:44 on dist-test-slave-gwfx"
server_key: "050f704efac7df80ebbea0d0f98842f4"
server_key_iv: "2d4c69e9385b64f45971e048b49bede2"
server_key_version: "encryptionkey@0"
I20260926 23:35:44.975607 19775 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 23:35:44.980490 19790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:44.982788 19775 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.007s sys 0.000s
I20260926 23:35:44.983160 19775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "d83bbc1bfbbb4620a8958d351026e1ef"
format_stamp: "Formatted at 2026-09-26 23:35:44 on dist-test-slave-gwfx"
server_key: "050f704efac7df80ebbea0d0f98842f4"
server_key_iv: "2d4c69e9385b64f45971e048b49bede2"
server_key_version: "encryptionkey@0"
I20260926 23:35:44.983469 19775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:35:45.008572 19775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:35:45.014329 19775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:35:45.014827 19775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:35:45.035734 19775 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42891
I20260926 23:35:45.035777 19841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42891 every 8 connection(s)
I20260926 23:35:45.037474 19775 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:35:45.042517 19842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:35:45.043625 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 19775
I20260926 23:35:45.043834 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:35:45.044498 388 external_mini_cluster.cc:1428] Setting key 2f255a64d0edf5aac1948afad3a268de
I20260926 23:35:45.057842 19842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d83bbc1bfbbb4620a8958d351026e1ef: Bootstrap starting.
Sep 26 23:35:45 dist-test-slave-gwfx krb5kdc[19762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:35:45.062126 19842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d83bbc1bfbbb4620a8958d351026e1ef: Neither blocks nor log segments found. Creating new log.
I20260926 23:35:45.063730 19842 log.cc:824] T 00000000000000000000000000000000 P d83bbc1bfbbb4620a8958d351026e1ef: Log is configured to *not* fsync() on all Append() calls
I20260926 23:35:45.067708 19842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d83bbc1bfbbb4620a8958d351026e1ef: No bootstrap required, opened a new log
I20260926 23:35:45.070870 19845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:45.047063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52040 (local address 127.0.97.62:42891)
0926 23:35:45.048017 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:45.048032 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:45.048082 (+ 50us) server_negotiation.cc:407] Connection header received
0926 23:35:45.050107 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:45.050143 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:45.050734 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:45.051250 (+ 516us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:45.052903 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:45.054407 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:45.055923 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:45.056739 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:45.056942 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:45.061810 (+ 4868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:45.061822 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:45.061845 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:35:45.061909 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:45.065692 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:45.066749 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:45.066759 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:45.066768 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:45.067002 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:45.067409 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:45.067415 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:45.067417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:45.068114 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:45.068475 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:45.068956 (+ 481us) server_negotiation.cc:299] Negotiation successful
0926 23:35:45.069506 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":687,"thread_start_us":435,"threads_started":1}
I20260926 23:35:45.077410 19842 raft_consensus.cc:348] T 00000000000000000000000000000000 P d83bbc1bfbbb4620a8958d351026e1ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d83bbc1bfbbb4620a8958d351026e1ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42891 } }
I20260926 23:35:45.077942 19842 raft_consensus.cc:374] T 00000000000000000000000000000000 P d83bbc1bfbbb4620a8958d351026e1ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:35:45.078042 19842 raft_consensus.cc:729] T 00000000000000000000000000000000 P d83bbc1bfbbb4620a8958d351026e1ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d83bbc1bfbbb4620a8958d351026e1ef, State: Initialized, Role: FOLLOWER
I20260926 23:35:45.078765 19842 consensus_queue.cc:260] T 00000000000000000000000000000000 P d83bbc1bfbbb4620a8958d351026e1ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d83bbc1bfbbb4620a8958d351026e1ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42891 } }
I20260926 23:35:45.079051 19842 raft_consensus.cc:388] T 00000000000000000000000000000000 P d83bbc1bfbbb4620a8958d351026e1ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:35:45.079209 19842 raft_consensus.cc:482] T 00000000000000000000000000000000 P d83bbc1bfbbb4620a8958d351026e1ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:35:45.079416 19842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d83bbc1bfbbb4620a8958d351026e1ef [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:35:45.082154 19842 raft_consensus.cc:504] T 00000000000000000000000000000000 P d83bbc1bfbbb4620a8958d351026e1ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d83bbc1bfbbb4620a8958d351026e1ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42891 } }
I20260926 23:35:45.082820 19842 leader_election.cc:302] T 00000000000000000000000000000000 P d83bbc1bfbbb4620a8958d351026e1ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d83bbc1bfbbb4620a8958d351026e1ef; no voters:
I20260926 23:35:45.083505 19842 leader_election.cc:288] T 00000000000000000000000000000000 P d83bbc1bfbbb4620a8958d351026e1ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:35:45.083671 19847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d83bbc1bfbbb4620a8958d351026e1ef [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:35:45.084323 19847 raft_consensus.cc:686] T 00000000000000000000000000000000 P d83bbc1bfbbb4620a8958d351026e1ef [term 1 LEADER]: Becoming Leader. State: Replica: d83bbc1bfbbb4620a8958d351026e1ef, State: Running, Role: LEADER
I20260926 23:35:45.084964 19847 consensus_queue.cc:237] T 00000000000000000000000000000000 P d83bbc1bfbbb4620a8958d351026e1ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d83bbc1bfbbb4620a8958d351026e1ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42891 } }
I20260926 23:35:45.085817 19842 sys_catalog.cc:564] T 00000000000000000000000000000000 P d83bbc1bfbbb4620a8958d351026e1ef [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:35:45.088110 19848 sys_catalog.cc:455] T 00000000000000000000000000000000 P d83bbc1bfbbb4620a8958d351026e1ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d83bbc1bfbbb4620a8958d351026e1ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d83bbc1bfbbb4620a8958d351026e1ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42891 } } }
I20260926 23:35:45.088622 19848 sys_catalog.cc:458] T 00000000000000000000000000000000 P d83bbc1bfbbb4620a8958d351026e1ef [sys.catalog]: This master's current role is: LEADER
I20260926 23:35:45.089054 19849 sys_catalog.cc:455] T 00000000000000000000000000000000 P d83bbc1bfbbb4620a8958d351026e1ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader d83bbc1bfbbb4620a8958d351026e1ef. Latest consensus state: current_term: 1 leader_uuid: "d83bbc1bfbbb4620a8958d351026e1ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d83bbc1bfbbb4620a8958d351026e1ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42891 } } }
I20260926 23:35:45.089371 19849 sys_catalog.cc:458] T 00000000000000000000000000000000 P d83bbc1bfbbb4620a8958d351026e1ef [sys.catalog]: This master's current role is: LEADER
I20260926 23:35:45.091852 19854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:35:45.098544 19854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:35:45.110908 19854 catalog_manager.cc:1347] Generated new cluster ID: 6c84080c8ef94f9fa7e3bbaf9e3d9364
I20260926 23:35:45.111074 19854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:35:45.129752 19854 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:35:45.132074 19854 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:35:45.147925 19854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d83bbc1bfbbb4620a8958d351026e1ef: Generated new TSK 0
I20260926 23:35:45.149289 19854 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 23:35:45.239394 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42891
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:35:45.671082 19868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:35:45.671494 19868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:35:45.671610 19868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:35:45.680214 19868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:35:45.680373 19868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:35:45.680508 19868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:35:45.692387 19868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:35:45.694367 19868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:35:45.696707 19868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:35:45.714349 19873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:45.714409 19876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:45.714365 19874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:35:45.720947 19868 server_base.cc:1034] running on GCE node
I20260926 23:35:45.721839 19868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:35:45.723174 19868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:35:45.724395 19868 hybrid_clock.cc:648] HybridClock initialized: now 1790465745724334 us; error 51 us; skew 500 ppm
I20260926 23:35:45.724757 19868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:35:45 dist-test-slave-gwfx krb5kdc[19762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:35:45.733247 19868 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 23:35:45.735486 19868 webserver.cc:460] Webserver started at http://127.0.97.1:36051/ using document root <none> and password file <none>
I20260926 23:35:45.736433 19868 fs_manager.cc:359] Metadata directory not provided
I20260926 23:35:45.736563 19868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:35:45.736945 19868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:35:45.741328 19868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-0/data/instance:
uuid: "6002658560034a4dbdf0b4c16dc670ef"
format_stamp: "Formatted at 2026-09-26 23:35:45 on dist-test-slave-gwfx"
server_key: "d553f8b5b17771b2b44039627a5d3f73"
server_key_iv: "5d26702aac664420feabcc080228899a"
server_key_version: "encryptionkey@0"
I20260926 23:35:45.742334 19868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6002658560034a4dbdf0b4c16dc670ef"
format_stamp: "Formatted at 2026-09-26 23:35:45 on dist-test-slave-gwfx"
server_key: "d553f8b5b17771b2b44039627a5d3f73"
server_key_iv: "5d26702aac664420feabcc080228899a"
server_key_version: "encryptionkey@0"
I20260926 23:35:45.749315 19868 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 23:35:45.753926 19883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:45.755703 19868 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 23:35:45.755904 19868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "6002658560034a4dbdf0b4c16dc670ef"
format_stamp: "Formatted at 2026-09-26 23:35:45 on dist-test-slave-gwfx"
server_key: "d553f8b5b17771b2b44039627a5d3f73"
server_key_iv: "5d26702aac664420feabcc080228899a"
server_key_version: "encryptionkey@0"
I20260926 23:35:45.756140 19868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:35:45.781062 19868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:35:45.786655 19868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:35:45.787014 19868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:35:45.788260 19868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:35:45.789947 19868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:35:45.790041 19868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:45.790186 19868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:35:45.790252 19868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:45.825874 19868 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:32949
I20260926 23:35:45.825922 19995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:32949 every 8 connection(s)
I20260926 23:35:45.827550 19868 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:35:45.833388 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 19868
I20260926 23:35:45.833616 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:35:45.834252 388 external_mini_cluster.cc:1428] Setting key ff79d29f9b5d5b989e6a134850771559
Sep 26 23:35:45 dist-test-slave-gwfx krb5kdc[19762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:35:45.852030 19998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:45.830817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55501 (local address 127.0.97.62:42891)
0926 23:35:45.831422 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:45.831427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:45.832495 (+ 1068us) server_negotiation.cc:407] Connection header received
0926 23:35:45.835317 (+ 2822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:45.835323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:45.835440 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:45.835558 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:45.837812 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:45.838657 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:45.840530 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:45.841001 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:45.841125 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:45.845554 (+ 4429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:45.845560 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:45.845564 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:35:45.845607 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:45.848321 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:45.849247 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:45.849252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:45.849254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:45.849419 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:45.849899 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:45.849902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:45.849904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:45.850231 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:45.850394 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:45.851331 (+ 937us) server_negotiation.cc:299] Negotiation successful
0926 23:35:45.851630 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":264,"threads_started":1}
I20260926 23:35:45.853317 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:45.831333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42891 (local address 127.0.97.1:55501)
0926 23:35:45.832279 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:45.832331 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:45.834944 (+ 2613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:45.835966 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:45.835978 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:45.836815 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:35:45.837583 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:45.837606 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:45.838869 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:45.838874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:45.840360 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:45.840375 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:45.841266 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:45.841274 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:45.841535 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:45.842321 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:35:45.842348 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:35:45.845240 (+ 2892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:35:45.848534 (+ 3294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:35:45.848547 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:35:45.848567 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:35:45.849102 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:35:45.849539 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:35:45.849544 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:35:45.849548 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:35:45.849777 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:35:45.850369 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:35:45.850379 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:35:45.850882 (+ 503us) client_negotiation.cc:769] Sending connection context
0926 23:35:45.851183 (+ 301us) client_negotiation.cc:240] Negotiation successful
0926 23:35:45.851887 (+ 704us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":407,"threads_started":1}
I20260926 23:35:45.855279 19996 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42891
I20260926 23:35:45.855669 19996 heartbeater.cc:464] Registering TS with master...
I20260926 23:35:45.856627 19996 heartbeater.cc:511] Master 127.0.97.62:42891 requested a full tablet report, sending...
I20260926 23:35:45.859377 19807 ts_manager.cc:194] Registered new tserver with Master: 6002658560034a4dbdf0b4c16dc670ef (127.0.97.1:32949)
I20260926 23:35:45.861198 19807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55501
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 23:35:45.903136 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42891
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:35:46.335201 20002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:35:46.335593 20002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:35:46.335701 20002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:35:46.344340 20002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:35:46.344511 20002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:35:46.344702 20002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:35:46.356877 20002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:35:46.358737 20002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:35:46.360515 20002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:35:46.376500 20008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:46.376471 20010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:46.376551 20007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:35:46.382863 20002 server_base.cc:1034] running on GCE node
I20260926 23:35:46.383848 20002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:35:46.385227 20002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:35:46.386489 20002 hybrid_clock.cc:648] HybridClock initialized: now 1790465746386415 us; error 65 us; skew 500 ppm
I20260926 23:35:46.387064 20002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:35:46 dist-test-slave-gwfx krb5kdc[19762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:35:46.391871 20002 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 23:35:46.394304 20002 webserver.cc:460] Webserver started at http://127.0.97.2:46045/ using document root <none> and password file <none>
I20260926 23:35:46.395473 20002 fs_manager.cc:359] Metadata directory not provided
I20260926 23:35:46.395704 20002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:35:46.396106 20002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:35:46.400355 20002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-1/data/instance:
uuid: "de92c08649f045998f681c9aa6559bdb"
format_stamp: "Formatted at 2026-09-26 23:35:46 on dist-test-slave-gwfx"
server_key: "29bcdba473ae67b01672127945b1a552"
server_key_iv: "d1b74e3fd638ee6a08716ea6f8d91126"
server_key_version: "encryptionkey@0"
I20260926 23:35:46.401549 20002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-1/wal/instance:
uuid: "de92c08649f045998f681c9aa6559bdb"
format_stamp: "Formatted at 2026-09-26 23:35:46 on dist-test-slave-gwfx"
server_key: "29bcdba473ae67b01672127945b1a552"
server_key_iv: "d1b74e3fd638ee6a08716ea6f8d91126"
server_key_version: "encryptionkey@0"
I20260926 23:35:46.409895 20002 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.001s sys 0.008s
I20260926 23:35:46.415644 20017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:46.418130 20002 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20260926 23:35:46.418427 20002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "de92c08649f045998f681c9aa6559bdb"
format_stamp: "Formatted at 2026-09-26 23:35:46 on dist-test-slave-gwfx"
server_key: "29bcdba473ae67b01672127945b1a552"
server_key_iv: "d1b74e3fd638ee6a08716ea6f8d91126"
server_key_version: "encryptionkey@0"
I20260926 23:35:46.418843 20002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:35:46.442423 20002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:35:46.448366 20002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:35:46.448745 20002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:35:46.450058 20002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:35:46.452004 20002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:35:46.452101 20002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:46.452236 20002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:35:46.452296 20002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:46.490167 20002 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34003
I20260926 23:35:46.490224 20129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34003 every 8 connection(s)
I20260926 23:35:46.492097 20002 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:35:46.498963 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 20002
I20260926 23:35:46.499182 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:35:46.500311 388 external_mini_cluster.cc:1428] Setting key 0396f18e59844d9a3c5838536f9b8f78
Sep 26 23:35:46 dist-test-slave-gwfx krb5kdc[19762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:35:46.519089 20132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:46.496971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43385 (local address 127.0.97.62:42891)
0926 23:35:46.497574 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:46.497583 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:46.498891 (+ 1308us) server_negotiation.cc:407] Connection header received
0926 23:35:46.501871 (+ 2980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:46.501878 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:46.501999 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:46.502147 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:46.504442 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:46.505258 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:46.507165 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:46.507691 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:46.507819 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:46.512370 (+ 4551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:46.512377 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:46.512381 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:35:46.512425 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:46.515320 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:46.516262 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:46.516268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:46.516270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:46.516424 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:46.516919 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:46.516923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:46.516926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:46.517258 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:46.517418 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:46.518313 (+ 895us) server_negotiation.cc:299] Negotiation successful
0926 23:35:46.518612 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":899,"threads_started":1}
I20260926 23:35:46.520466 20133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:46.497508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42891 (local address 127.0.97.2:43385)
0926 23:35:46.498642 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:46.498734 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:46.501445 (+ 2711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:46.502702 (+ 1257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:46.502720 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:46.503487 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:35:46.504231 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:46.504255 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:46.505454 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:46.505459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:46.506998 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:46.507015 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:46.507969 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:46.507977 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:46.508248 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:46.509134 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:35:46.509170 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:35:46.512021 (+ 2851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:35:46.515521 (+ 3500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:35:46.515534 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:35:46.515558 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:35:46.516097 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:35:46.516548 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:35:46.516553 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:35:46.516557 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:35:46.516789 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:35:46.517431 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:35:46.517441 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:35:46.517899 (+ 458us) client_negotiation.cc:769] Sending connection context
0926 23:35:46.518192 (+ 293us) client_negotiation.cc:240] Negotiation successful
0926 23:35:46.518973 (+ 781us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"thread_start_us":446,"threads_started":1}
I20260926 23:35:46.522289 20130 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42891
I20260926 23:35:46.522639 20130 heartbeater.cc:464] Registering TS with master...
I20260926 23:35:46.523546 20130 heartbeater.cc:511] Master 127.0.97.62:42891 requested a full tablet report, sending...
I20260926 23:35:46.525566 19807 ts_manager.cc:194] Registered new tserver with Master: de92c08649f045998f681c9aa6559bdb (127.0.97.2:34003)
I20260926 23:35:46.526567 19807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43385
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 23:35:46.570878 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42891
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:35:46.865291 19996 heartbeater.cc:503] Master 127.0.97.62:42891 was elected leader, sending a full tablet report...
W20260926 23:35:46.975733 20136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:35:46.976122 20136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:35:46.976260 20136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:35:46.984369 20136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:35:46.984545 20136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:35:46.984721 20136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:35:46.996281 20136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:35:46.998159 20136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:35:46.999920 20136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:35:47.016103 20141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:47.016126 20142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:47.016064 20144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:35:47.017140 20136 server_base.cc:1034] running on GCE node
I20260926 23:35:47.018019 20136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:35:47.019358 20136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:35:47.020619 20136 hybrid_clock.cc:648] HybridClock initialized: now 1790465747020546 us; error 68 us; skew 500 ppm
I20260926 23:35:47.021071 20136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:35:47 dist-test-slave-gwfx krb5kdc[19762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:35:47.025180 20136 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 23:35:47.027211 20136 webserver.cc:460] Webserver started at http://127.0.97.3:45445/ using document root <none> and password file <none>
I20260926 23:35:47.028086 20136 fs_manager.cc:359] Metadata directory not provided
I20260926 23:35:47.028216 20136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:35:47.028568 20136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:35:47.032311 20136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-2/data/instance:
uuid: "93ea9f9ede994b9ba790110665a69d38"
format_stamp: "Formatted at 2026-09-26 23:35:47 on dist-test-slave-gwfx"
server_key: "d67fdb1dfeae9f67bba1dde00d9e2102"
server_key_iv: "9ad05c8c1cd993c0b621b891d1146684"
server_key_version: "encryptionkey@0"
I20260926 23:35:47.033254 20136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-2/wal/instance:
uuid: "93ea9f9ede994b9ba790110665a69d38"
format_stamp: "Formatted at 2026-09-26 23:35:47 on dist-test-slave-gwfx"
server_key: "d67fdb1dfeae9f67bba1dde00d9e2102"
server_key_iv: "9ad05c8c1cd993c0b621b891d1146684"
server_key_version: "encryptionkey@0"
I20260926 23:35:47.039721 20136 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260926 23:35:47.044143 20151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:47.046286 20136 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 23:35:47.046521 20136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "93ea9f9ede994b9ba790110665a69d38"
format_stamp: "Formatted at 2026-09-26 23:35:47 on dist-test-slave-gwfx"
server_key: "d67fdb1dfeae9f67bba1dde00d9e2102"
server_key_iv: "9ad05c8c1cd993c0b621b891d1146684"
server_key_version: "encryptionkey@0"
I20260926 23:35:47.046834 20136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:35:47.072701 20136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:35:47.080062 20136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:35:47.080540 20136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:35:47.081974 20136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:35:47.083989 20136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:35:47.084098 20136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:47.084254 20136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:35:47.084317 20136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:47.121212 20136 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40609
I20260926 23:35:47.121280 20263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40609 every 8 connection(s)
I20260926 23:35:47.123128 20136 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:35:47.124595 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 20136
I20260926 23:35:47.124841 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:35:47.125484 388 external_mini_cluster.cc:1428] Setting key fc55f137d484b54d918bf7ca27b40b28
Sep 26 23:35:47 dist-test-slave-gwfx krb5kdc[19762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:35:47.151997 20266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:47.126947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58203 (local address 127.0.97.62:42891)
0926 23:35:47.127536 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:47.127543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:47.129119 (+ 1576us) server_negotiation.cc:407] Connection header received
0926 23:35:47.131481 (+ 2362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:47.131486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:47.131573 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:47.131682 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:47.133701 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.134587 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.136559 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.137068 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.137253 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:47.142286 (+ 5033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:47.142294 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:47.142300 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:35:47.142355 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:47.146285 (+ 3930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:47.147968 (+ 1683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:47.147974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:47.147977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:47.148154 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:47.148860 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:47.148865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:47.148867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:47.149240 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:47.149474 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:47.150952 (+ 1478us) server_negotiation.cc:299] Negotiation successful
0926 23:35:47.151279 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":246,"threads_started":1}
I20260926 23:35:47.152875 20267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:47.127748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42891 (local address 127.0.97.3:58203)
0926 23:35:47.128907 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:47.128972 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:47.131145 (+ 2173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:47.132013 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:47.132024 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:47.132742 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:35:47.133483 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:47.133503 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.134839 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.134843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:47.136369 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:47.136386 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.137439 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.137450 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:47.137782 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:47.138780 (+ 998us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:35:47.138814 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:35:47.141922 (+ 3108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:35:47.146632 (+ 4710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:35:47.146648 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:35:47.146672 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:35:47.147753 (+ 1081us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:35:47.148371 (+ 618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:35:47.148375 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:35:47.148378 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:35:47.148667 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:35:47.149547 (+ 880us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:35:47.149570 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:35:47.150205 (+ 635us) client_negotiation.cc:769] Sending connection context
0926 23:35:47.150680 (+ 475us) client_negotiation.cc:240] Negotiation successful
0926 23:35:47.151388 (+ 708us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":412,"threads_started":1}
I20260926 23:35:47.154944 20264 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42891
I20260926 23:35:47.155428 20264 heartbeater.cc:464] Registering TS with master...
I20260926 23:35:47.156529 20264 heartbeater.cc:511] Master 127.0.97.62:42891 requested a full tablet report, sending...
I20260926 23:35:47.158852 19807 ts_manager.cc:194] Registered new tserver with Master: 93ea9f9ede994b9ba790110665a69d38 (127.0.97.3:40609)
I20260926 23:35:47.159947 19807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58203
I20260926 23:35:47.170557 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:35:47.191497 20266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:47.176043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33276 (local address 127.0.97.62:42891)
0926 23:35:47.176328 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:47.176347 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:47.176601 (+ 254us) server_negotiation.cc:407] Connection header received
0926 23:35:47.176798 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:47.176803 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:47.176907 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:47.177007 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:47.178817 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.179702 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.181272 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.181726 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.181885 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:47.184611 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:47.184617 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:47.184621 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:35:47.184669 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:47.188149 (+ 3480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:47.189047 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:47.189052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:47.189055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:47.189228 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:47.189687 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:47.189691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:47.189694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:47.190204 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:47.190438 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:47.190916 (+ 478us) server_negotiation.cc:299] Negotiation successful
0926 23:35:47.191155 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260926 23:35:47.196575 19807 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:35:47.204707 19807 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:35:47.249459 20282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:47.224737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55326 (local address 127.0.97.1:32949)
0926 23:35:47.230117 (+ 5380us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:47.230131 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:47.240952 (+ 10821us) server_negotiation.cc:407] Connection header received
0926 23:35:47.241368 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:47.241377 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:47.241626 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:47.241844 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:35:47.242756 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.244335 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.246466 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.248087 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.248213 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:47.248507 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:47.248829 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 23:35:47.249030 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5117,"thread_start_us":4868,"threads_started":1}
I20260926 23:35:47.249543 20278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:47.224189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32949 (local address 127.0.0.1:55326)
0926 23:35:47.240756 (+ 16567us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:47.240794 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:47.241105 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:47.242117 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:47.242121 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:47.242150 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:35:47.242539 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:47.242550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.244478 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.244482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:47.246255 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:47.246277 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.248286 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.248293 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:47.248535 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:47.248597 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 23:35:47.248752 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 23:35:47.249030 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16408,"mutex_wait_us":50,"thread_start_us":14182,"threads_started":1}
I20260926 23:35:47.249986 20277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:47.220318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40609 (local address 127.0.0.1:36194)
0926 23:35:47.236422 (+ 16104us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:47.236520 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:47.236817 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:47.240821 (+ 4004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:47.240831 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:47.240894 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:35:47.241993 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:47.242011 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.243787 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.243794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:47.246402 (+ 2608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:47.246416 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.248849 (+ 2433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.248856 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:47.249152 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:47.249220 (+ 68us) client_negotiation.cc:769] Sending connection context
0926 23:35:47.249310 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 23:35:47.249491 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15659,"mutex_wait_us":479,"spinlock_wait_cycles":9472,"thread_start_us":16823,"threads_started":1}
I20260926 23:35:47.250461 20279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:47.220700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36194 (local address 127.0.97.3:40609)
0926 23:35:47.239853 (+ 19153us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:47.239864 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:47.239914 (+ 50us) server_negotiation.cc:407] Connection header received
0926 23:35:47.240050 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:47.240060 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:47.240441 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:47.241025 (+ 584us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:35:47.242277 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.243631 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.246619 (+ 2988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.248224 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.248359 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:47.248604 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:47.249849 (+ 1245us) server_negotiation.cc:299] Negotiation successful
0926 23:35:47.250073 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18929,"thread_start_us":18733,"threads_started":1}
I20260926 23:35:47.250651 20281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:47.220726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34003 (local address 127.0.0.1:58796)
0926 23:35:47.238823 (+ 18097us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:47.238872 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:47.239096 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:47.240821 (+ 1725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:47.240831 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:47.240895 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:35:47.242148 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:47.242163 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.244189 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.244194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:47.246505 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:47.246517 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.248847 (+ 2330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.248856 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:47.249152 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:47.249227 (+ 75us) client_negotiation.cc:769] Sending connection context
0926 23:35:47.249310 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 23:35:47.249491 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17881,"mutex_wait_us":115,"spinlock_wait_cycles":15616,"thread_start_us":19696,"threads_started":1}
I20260926 23:35:47.251142 20280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:47.220871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58796 (local address 127.0.97.2:34003)
0926 23:35:47.240209 (+ 19338us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:47.240218 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:47.240245 (+ 27us) server_negotiation.cc:407] Connection header received
0926 23:35:47.240351 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:47.240359 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:47.240554 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:47.241583 (+ 1029us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:35:47.242400 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.244049 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.246709 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.248224 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.248369 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:47.248568 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:47.249630 (+ 1062us) server_negotiation.cc:299] Negotiation successful
0926 23:35:47.250253 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19164,"thread_start_us":19050,"threads_started":1}
I20260926 23:35:47.254731 20199 tablet_service.cc:1467] Processing CreateTablet for tablet a9ae0d779cab454fb213018f35a989a7 (DEFAULT_TABLE table=test-table [id=d28d5e4ab8d54959bfa4f32ab347c720]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:35:47.254746 19931 tablet_service.cc:1467] Processing CreateTablet for tablet a9ae0d779cab454fb213018f35a989a7 (DEFAULT_TABLE table=test-table [id=d28d5e4ab8d54959bfa4f32ab347c720]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:35:47.254742 20065 tablet_service.cc:1467] Processing CreateTablet for tablet a9ae0d779cab454fb213018f35a989a7 (DEFAULT_TABLE table=test-table [id=d28d5e4ab8d54959bfa4f32ab347c720]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:35:47.256629 20199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9ae0d779cab454fb213018f35a989a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:35:47.256652 19931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9ae0d779cab454fb213018f35a989a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:35:47.257392 20065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9ae0d779cab454fb213018f35a989a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:35:47.271483 20283 tablet_bootstrap.cc:492] T a9ae0d779cab454fb213018f35a989a7 P 93ea9f9ede994b9ba790110665a69d38: Bootstrap starting.
I20260926 23:35:47.275553 20284 tablet_bootstrap.cc:492] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef: Bootstrap starting.
I20260926 23:35:47.277107 20283 tablet_bootstrap.cc:654] T a9ae0d779cab454fb213018f35a989a7 P 93ea9f9ede994b9ba790110665a69d38: Neither blocks nor log segments found. Creating new log.
I20260926 23:35:47.278818 20283 log.cc:824] T a9ae0d779cab454fb213018f35a989a7 P 93ea9f9ede994b9ba790110665a69d38: Log is configured to *not* fsync() on all Append() calls
I20260926 23:35:47.279739 20285 tablet_bootstrap.cc:492] T a9ae0d779cab454fb213018f35a989a7 P de92c08649f045998f681c9aa6559bdb: Bootstrap starting.
I20260926 23:35:47.279989 20284 tablet_bootstrap.cc:654] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef: Neither blocks nor log segments found. Creating new log.
I20260926 23:35:47.281786 20284 log.cc:824] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef: Log is configured to *not* fsync() on all Append() calls
I20260926 23:35:47.284189 20285 tablet_bootstrap.cc:654] T a9ae0d779cab454fb213018f35a989a7 P de92c08649f045998f681c9aa6559bdb: Neither blocks nor log segments found. Creating new log.
I20260926 23:35:47.285784 20285 log.cc:824] T a9ae0d779cab454fb213018f35a989a7 P de92c08649f045998f681c9aa6559bdb: Log is configured to *not* fsync() on all Append() calls
I20260926 23:35:47.286967 20284 tablet_bootstrap.cc:492] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef: No bootstrap required, opened a new log
I20260926 23:35:47.287370 20284 ts_tablet_manager.cc:1397] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260926 23:35:47.289979 20283 tablet_bootstrap.cc:492] T a9ae0d779cab454fb213018f35a989a7 P 93ea9f9ede994b9ba790110665a69d38: No bootstrap required, opened a new log
I20260926 23:35:47.289976 20285 tablet_bootstrap.cc:492] T a9ae0d779cab454fb213018f35a989a7 P de92c08649f045998f681c9aa6559bdb: No bootstrap required, opened a new log
I20260926 23:35:47.290375 20283 ts_tablet_manager.cc:1397] T a9ae0d779cab454fb213018f35a989a7 P 93ea9f9ede994b9ba790110665a69d38: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.003s
I20260926 23:35:47.290375 20285 ts_tablet_manager.cc:1397] T a9ae0d779cab454fb213018f35a989a7 P de92c08649f045998f681c9aa6559bdb: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260926 23:35:47.296741 20284 raft_consensus.cc:348] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93ea9f9ede994b9ba790110665a69d38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40609 } } peers { permanent_uuid: "de92c08649f045998f681c9aa6559bdb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34003 } } peers { permanent_uuid: "6002658560034a4dbdf0b4c16dc670ef" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } }
I20260926 23:35:47.297698 20284 raft_consensus.cc:374] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:35:47.297844 20284 raft_consensus.cc:729] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6002658560034a4dbdf0b4c16dc670ef, State: Initialized, Role: FOLLOWER
I20260926 23:35:47.298899 20284 consensus_queue.cc:260] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93ea9f9ede994b9ba790110665a69d38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40609 } } peers { permanent_uuid: "de92c08649f045998f681c9aa6559bdb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34003 } } peers { permanent_uuid: "6002658560034a4dbdf0b4c16dc670ef" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } }
I20260926 23:35:47.301330 20284 ts_tablet_manager.cc:1428] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260926 23:35:47.303000 20283 raft_consensus.cc:348] T a9ae0d779cab454fb213018f35a989a7 P 93ea9f9ede994b9ba790110665a69d38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93ea9f9ede994b9ba790110665a69d38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40609 } } peers { permanent_uuid: "de92c08649f045998f681c9aa6559bdb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34003 } } peers { permanent_uuid: "6002658560034a4dbdf0b4c16dc670ef" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } }
I20260926 23:35:47.303000 20285 raft_consensus.cc:348] T a9ae0d779cab454fb213018f35a989a7 P de92c08649f045998f681c9aa6559bdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93ea9f9ede994b9ba790110665a69d38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40609 } } peers { permanent_uuid: "de92c08649f045998f681c9aa6559bdb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34003 } } peers { permanent_uuid: "6002658560034a4dbdf0b4c16dc670ef" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } }
I20260926 23:35:47.303745 20283 raft_consensus.cc:374] T a9ae0d779cab454fb213018f35a989a7 P 93ea9f9ede994b9ba790110665a69d38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:35:47.303810 20285 raft_consensus.cc:374] T a9ae0d779cab454fb213018f35a989a7 P de92c08649f045998f681c9aa6559bdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:35:47.303925 20283 raft_consensus.cc:729] T a9ae0d779cab454fb213018f35a989a7 P 93ea9f9ede994b9ba790110665a69d38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93ea9f9ede994b9ba790110665a69d38, State: Initialized, Role: FOLLOWER
I20260926 23:35:47.303928 20285 raft_consensus.cc:729] T a9ae0d779cab454fb213018f35a989a7 P de92c08649f045998f681c9aa6559bdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de92c08649f045998f681c9aa6559bdb, State: Initialized, Role: FOLLOWER
I20260926 23:35:47.304734 20285 consensus_queue.cc:260] T a9ae0d779cab454fb213018f35a989a7 P de92c08649f045998f681c9aa6559bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93ea9f9ede994b9ba790110665a69d38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40609 } } peers { permanent_uuid: "de92c08649f045998f681c9aa6559bdb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34003 } } peers { permanent_uuid: "6002658560034a4dbdf0b4c16dc670ef" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } }
I20260926 23:35:47.304728 20283 consensus_queue.cc:260] T a9ae0d779cab454fb213018f35a989a7 P 93ea9f9ede994b9ba790110665a69d38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93ea9f9ede994b9ba790110665a69d38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40609 } } peers { permanent_uuid: "de92c08649f045998f681c9aa6559bdb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34003 } } peers { permanent_uuid: "6002658560034a4dbdf0b4c16dc670ef" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } }
I20260926 23:35:47.306892 20130 heartbeater.cc:503] Master 127.0.97.62:42891 was elected leader, sending a full tablet report...
I20260926 23:35:47.307380 20283 ts_tablet_manager.cc:1428] T a9ae0d779cab454fb213018f35a989a7 P 93ea9f9ede994b9ba790110665a69d38: Time spent starting tablet: real 0.017s user 0.015s sys 0.005s
I20260926 23:35:47.307407 20285 ts_tablet_manager.cc:1428] T a9ae0d779cab454fb213018f35a989a7 P de92c08649f045998f681c9aa6559bdb: Time spent starting tablet: real 0.017s user 0.009s sys 0.010s
I20260926 23:35:47.308915 20264 heartbeater.cc:503] Master 127.0.97.62:42891 was elected leader, sending a full tablet report...
W20260926 23:35:47.330961 19997 tablet.cc:2369] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:35:47.331149 20289 raft_consensus.cc:482] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:35:47.331491 20289 raft_consensus.cc:504] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93ea9f9ede994b9ba790110665a69d38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40609 } } peers { permanent_uuid: "de92c08649f045998f681c9aa6559bdb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34003 } } peers { permanent_uuid: "6002658560034a4dbdf0b4c16dc670ef" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } }
I20260926 23:35:47.333773 20289 leader_election.cc:288] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93ea9f9ede994b9ba790110665a69d38 (127.0.97.3:40609), de92c08649f045998f681c9aa6559bdb (127.0.97.2:34003)
I20260926 23:35:47.342435 20280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:47.334826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47167 (local address 127.0.97.2:34003)
0926 23:35:47.335074 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:47.335081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:47.336133 (+ 1052us) server_negotiation.cc:407] Connection header received
0926 23:35:47.336449 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:47.336454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:47.336530 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:47.336610 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:35:47.337291 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.338095 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.340435 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.341316 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.341479 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:47.341605 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:47.341983 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 23:35:47.342132 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 23:35:47.342578 20293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:47.334251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34003 (local address 127.0.97.1:47167)
0926 23:35:47.336018 (+ 1767us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:47.336033 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:47.336252 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:47.336639 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:47.336644 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:47.336673 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:35:47.337113 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:47.337126 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.338236 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.338242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:47.340237 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:47.340252 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.341606 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.341613 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:47.341781 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:47.341834 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 23:35:47.341931 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 23:35:47.342098 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1647,"mutex_wait_us":22,"thread_start_us":1421,"threads_started":1}
I20260926 23:35:47.342752 20279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:47.334316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47763 (local address 127.0.97.3:40609)
0926 23:35:47.334599 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:47.334607 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:47.335859 (+ 1252us) server_negotiation.cc:407] Connection header received
0926 23:35:47.336206 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:47.336211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:47.336302 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:47.336373 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:35:47.337257 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.338076 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.340433 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.341460 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.341688 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:47.341903 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:47.342192 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 23:35:47.342435 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 23:35:47.343199 20292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:47.334136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40609 (local address 127.0.97.1:47763)
0926 23:35:47.335703 (+ 1567us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:47.335725 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:47.336013 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:47.336592 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:47.336595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:47.336618 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:35:47.337086 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:47.337100 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.338221 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.338226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:47.340225 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:47.340243 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.341786 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.341793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:47.342022 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:47.342052 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 23:35:47.342144 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 23:35:47.342421 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1393,"mutex_wait_us":87,"spinlock_wait_cycles":34176,"thread_start_us":1147,"threads_started":1}
I20260926 23:35:47.344092 20085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ae0d779cab454fb213018f35a989a7" candidate_uuid: "6002658560034a4dbdf0b4c16dc670ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de92c08649f045998f681c9aa6559bdb" is_pre_election: true
I20260926 23:35:47.344237 20219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ae0d779cab454fb213018f35a989a7" candidate_uuid: "6002658560034a4dbdf0b4c16dc670ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93ea9f9ede994b9ba790110665a69d38" is_pre_election: true
I20260926 23:35:47.344765 20085 raft_consensus.cc:2454] T a9ae0d779cab454fb213018f35a989a7 P de92c08649f045998f681c9aa6559bdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6002658560034a4dbdf0b4c16dc670ef in term 0.
I20260926 23:35:47.344873 20219 raft_consensus.cc:2454] T a9ae0d779cab454fb213018f35a989a7 P 93ea9f9ede994b9ba790110665a69d38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6002658560034a4dbdf0b4c16dc670ef in term 0.
I20260926 23:35:47.345696 19886 leader_election.cc:302] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6002658560034a4dbdf0b4c16dc670ef, 93ea9f9ede994b9ba790110665a69d38; no voters:
I20260926 23:35:47.346191 20289 raft_consensus.cc:2781] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:35:47.346379 20289 raft_consensus.cc:482] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:35:47.346500 20289 raft_consensus.cc:3037] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:35:47.349476 20289 raft_consensus.cc:504] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93ea9f9ede994b9ba790110665a69d38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40609 } } peers { permanent_uuid: "de92c08649f045998f681c9aa6559bdb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34003 } } peers { permanent_uuid: "6002658560034a4dbdf0b4c16dc670ef" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } }
I20260926 23:35:47.350389 20289 leader_election.cc:288] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef [CANDIDATE]: Term 1 election: Requested vote from peers 93ea9f9ede994b9ba790110665a69d38 (127.0.97.3:40609), de92c08649f045998f681c9aa6559bdb (127.0.97.2:34003)
I20260926 23:35:47.351151 20219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ae0d779cab454fb213018f35a989a7" candidate_uuid: "6002658560034a4dbdf0b4c16dc670ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93ea9f9ede994b9ba790110665a69d38"
I20260926 23:35:47.351147 20085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ae0d779cab454fb213018f35a989a7" candidate_uuid: "6002658560034a4dbdf0b4c16dc670ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de92c08649f045998f681c9aa6559bdb"
I20260926 23:35:47.351439 20219 raft_consensus.cc:3037] T a9ae0d779cab454fb213018f35a989a7 P 93ea9f9ede994b9ba790110665a69d38 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:35:47.351439 20085 raft_consensus.cc:3037] T a9ae0d779cab454fb213018f35a989a7 P de92c08649f045998f681c9aa6559bdb [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:35:47.355378 20219 raft_consensus.cc:2454] T a9ae0d779cab454fb213018f35a989a7 P 93ea9f9ede994b9ba790110665a69d38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6002658560034a4dbdf0b4c16dc670ef in term 1.
I20260926 23:35:47.355378 20085 raft_consensus.cc:2454] T a9ae0d779cab454fb213018f35a989a7 P de92c08649f045998f681c9aa6559bdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6002658560034a4dbdf0b4c16dc670ef in term 1.
I20260926 23:35:47.356117 19886 leader_election.cc:302] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6002658560034a4dbdf0b4c16dc670ef, 93ea9f9ede994b9ba790110665a69d38, de92c08649f045998f681c9aa6559bdb; no voters:
I20260926 23:35:47.356568 20289 raft_consensus.cc:2781] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:35:47.357120 20289 raft_consensus.cc:686] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef [term 1 LEADER]: Becoming Leader. State: Replica: 6002658560034a4dbdf0b4c16dc670ef, State: Running, Role: LEADER
I20260926 23:35:47.357784 20289 consensus_queue.cc:237] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93ea9f9ede994b9ba790110665a69d38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40609 } } peers { permanent_uuid: "de92c08649f045998f681c9aa6559bdb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34003 } } peers { permanent_uuid: "6002658560034a4dbdf0b4c16dc670ef" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } }
I20260926 23:35:47.363795 19806 catalog_manager.cc:5495] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 6002658560034a4dbdf0b4c16dc670ef (127.0.97.1). New cstate: current_term: 1 leader_uuid: "6002658560034a4dbdf0b4c16dc670ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93ea9f9ede994b9ba790110665a69d38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de92c08649f045998f681c9aa6559bdb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6002658560034a4dbdf0b4c16dc670ef" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32949 } health_report { overall_health: HEALTHY } } }
W20260926 23:35:47.375308 20265 tablet.cc:2369] T a9ae0d779cab454fb213018f35a989a7 P 93ea9f9ede994b9ba790110665a69d38: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:35:47.406620 20282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:47.399845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55336 (local address 127.0.97.1:32949)
0926 23:35:47.400098 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:47.400103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:47.400125 (+ 22us) server_negotiation.cc:407] Connection header received
0926 23:35:47.400407 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:47.400413 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:47.400526 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:47.400699 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:35:47.401187 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.402074 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.403789 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.404353 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.404488 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:47.405181 (+ 693us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:35:47.405929 (+ 748us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:35:47.406068 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:47.406186 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 23:35:47.406309 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 23:35:47.417605 20219 raft_consensus.cc:1261] T a9ae0d779cab454fb213018f35a989a7 P 93ea9f9ede994b9ba790110665a69d38 [term 1 FOLLOWER]: Refusing update from remote peer 6002658560034a4dbdf0b4c16dc670ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:35:47.417605 20085 raft_consensus.cc:1261] T a9ae0d779cab454fb213018f35a989a7 P de92c08649f045998f681c9aa6559bdb [term 1 FOLLOWER]: Refusing update from remote peer 6002658560034a4dbdf0b4c16dc670ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:35:47.419035 20289 consensus_queue.cc:1035] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "de92c08649f045998f681c9aa6559bdb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:35:47.419484 20294 consensus_queue.cc:1035] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "93ea9f9ede994b9ba790110665a69d38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:35:47.435796 20298 mvcc.cc:205] Tried to move back new op lower bound from 7333747701409177600 to 7333747701182853120. Current Snapshot: MvccSnapshot[applied={T|T < 7333747701409177600}]
I20260926 23:35:47.454576 19806 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:35:47.455178 19806 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:35:47.459851 19806 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d83bbc1bfbbb4620a8958d351026e1ef: Sending DeleteTablet for 3 replicas of tablet a9ae0d779cab454fb213018f35a989a7
I20260926 23:35:47.461786 20199 tablet_service.cc:1514] Processing DeleteTablet for tablet a9ae0d779cab454fb213018f35a989a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:35:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36194
I20260926 23:35:47.461748 20065 tablet_service.cc:1514] Processing DeleteTablet for tablet a9ae0d779cab454fb213018f35a989a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:35:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58796
I20260926 23:35:47.464036 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 19868
I20260926 23:35:47.464130 19931 tablet_service.cc:1514] Processing DeleteTablet for tablet a9ae0d779cab454fb213018f35a989a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:35:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55326
I20260926 23:35:47.465961 20307 tablet_replica.cc:330] T a9ae0d779cab454fb213018f35a989a7 P de92c08649f045998f681c9aa6559bdb: stopping tablet replica
I20260926 23:35:47.466621 20307 raft_consensus.cc:2229] T a9ae0d779cab454fb213018f35a989a7 P de92c08649f045998f681c9aa6559bdb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:35:47.467424 20307 raft_consensus.cc:2258] T a9ae0d779cab454fb213018f35a989a7 P de92c08649f045998f681c9aa6559bdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:35:47.468432 20306 tablet_replica.cc:330] T a9ae0d779cab454fb213018f35a989a7 P 93ea9f9ede994b9ba790110665a69d38: stopping tablet replica
I20260926 23:35:47.468904 20306 raft_consensus.cc:2229] T a9ae0d779cab454fb213018f35a989a7 P 93ea9f9ede994b9ba790110665a69d38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:35:47.469450 20306 raft_consensus.cc:2258] T a9ae0d779cab454fb213018f35a989a7 P 93ea9f9ede994b9ba790110665a69d38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:35:47.471298 20308 tablet_replica.cc:330] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef: stopping tablet replica
I20260926 23:35:47.471946 20308 raft_consensus.cc:2229] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:35:47.472316 20307 ts_tablet_manager.cc:1905] T a9ae0d779cab454fb213018f35a989a7 P de92c08649f045998f681c9aa6559bdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:35:47.472316 20306 ts_tablet_manager.cc:1905] T a9ae0d779cab454fb213018f35a989a7 P 93ea9f9ede994b9ba790110665a69d38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:35:47.472967 20308 raft_consensus.cc:2258] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:35:47.480696 20306 ts_tablet_manager.cc:1918] T a9ae0d779cab454fb213018f35a989a7 P 93ea9f9ede994b9ba790110665a69d38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:35:47.480687 20307 ts_tablet_manager.cc:1918] T a9ae0d779cab454fb213018f35a989a7 P de92c08649f045998f681c9aa6559bdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:35:47.480931 20307 log.cc:1196] T a9ae0d779cab454fb213018f35a989a7 P de92c08649f045998f681c9aa6559bdb: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-1/wal/wals/a9ae0d779cab454fb213018f35a989a7
I20260926 23:35:47.481172 20306 log.cc:1196] T a9ae0d779cab454fb213018f35a989a7 P 93ea9f9ede994b9ba790110665a69d38: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-2/wal/wals/a9ae0d779cab454fb213018f35a989a7
I20260926 23:35:47.481607 20307 ts_tablet_manager.cc:1939] T a9ae0d779cab454fb213018f35a989a7 P de92c08649f045998f681c9aa6559bdb: Deleting consensus metadata
I20260926 23:35:47.481940 20306 ts_tablet_manager.cc:1939] T a9ae0d779cab454fb213018f35a989a7 P 93ea9f9ede994b9ba790110665a69d38: Deleting consensus metadata
I20260926 23:35:47.484167 19791 catalog_manager.cc:4841] TS de92c08649f045998f681c9aa6559bdb (127.0.97.2:34003): tablet a9ae0d779cab454fb213018f35a989a7 (table test-table [id=d28d5e4ab8d54959bfa4f32ab347c720]) successfully deleted
I20260926 23:35:47.484226 19793 catalog_manager.cc:4841] TS 93ea9f9ede994b9ba790110665a69d38 (127.0.97.3:40609): tablet a9ae0d779cab454fb213018f35a989a7 (table test-table [id=d28d5e4ab8d54959bfa4f32ab347c720]) successfully deleted
I20260926 23:35:47.485442 20308 ts_tablet_manager.cc:1905] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 23:35:47 dist-test-slave-gwfx krb5kdc[19762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 23:35:47.495049 20308 ts_tablet_manager.cc:1918] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:35:47.495312 20308 log.cc:1196] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790465400694755-388-0/minicluster-data/ts-0/wal/wals/a9ae0d779cab454fb213018f35a989a7
I20260926 23:35:47.495978 20308 ts_tablet_manager.cc:1939] T a9ae0d779cab454fb213018f35a989a7 P 6002658560034a4dbdf0b4c16dc670ef: Deleting consensus metadata
I20260926 23:35:47.498744 19792 catalog_manager.cc:4841] TS 6002658560034a4dbdf0b4c16dc670ef (127.0.97.1:32949): tablet a9ae0d779cab454fb213018f35a989a7 (table test-table [id=d28d5e4ab8d54959bfa4f32ab347c720]) successfully deleted
I20260926 23:35:47.505717 20282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:47.466400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55340 (local address 127.0.97.1:32949)
0926 23:35:47.466630 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:47.466639 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:47.477764 (+ 11125us) server_negotiation.cc:407] Connection header received
0926 23:35:47.478456 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:47.478464 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:47.478581 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:47.478709 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:47.480806 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.482145 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.484780 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.485340 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.485436 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:47.496705 (+ 11269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:47.496727 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:47.496753 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:35:47.496825 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:47.501612 (+ 4787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:47.502897 (+ 1285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:47.502910 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:47.502914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:47.503121 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:47.503588 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:47.503593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:47.503596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:47.504142 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:47.504650 (+ 508us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:47.504903 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 23:35:47.505246 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 23:35:47.506645 19991 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:35:47.621047 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 19868
I20260926 23:35:47.640277 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 20002
Sep 26 23:35:47 dist-test-slave-gwfx krb5kdc[19762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 23:35:47.660044 20280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:47.641345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58802 (local address 127.0.97.2:34003)
0926 23:35:47.641598 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:47.641604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:47.641652 (+ 48us) server_negotiation.cc:407] Connection header received
0926 23:35:47.641801 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:47.641806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:47.641908 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:47.641999 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:47.643638 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.644514 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.646150 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.646670 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.646908 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:47.652216 (+ 5308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:47.652240 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:47.652261 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:35:47.652324 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:47.656359 (+ 4035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:47.657385 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:47.657397 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:47.657400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:47.657586 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:47.658019 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:47.658024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:47.658026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:47.658560 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:47.658970 (+ 410us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:47.659335 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 23:35:47.659613 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260926 23:35:47.660971 20125 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:35:47.761305 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 20002
I20260926 23:35:47.778738 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 20136
Sep 26 23:35:47 dist-test-slave-gwfx krb5kdc[19762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 23:35:47.796960 20279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:47.779654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36206 (local address 127.0.97.3:40609)
0926 23:35:47.779884 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:47.779889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:47.779912 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:35:47.779991 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:47.779995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:47.780071 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:47.780168 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:47.781613 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.782470 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.784299 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:47.784770 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:47.784896 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:47.790281 (+ 5385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:47.790302 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:47.790324 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:35:47.790384 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:47.793654 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:47.794535 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:47.794545 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:47.794548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:47.794823 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:47.795290 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:47.795295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:47.795297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:47.795802 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:47.796086 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:47.796389 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 23:35:47.796594 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 23:35:47.797766 20259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:35:47.895619 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 20136
I20260926 23:35:47.912590 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 19775
I20260926 23:35:47.914086 19837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:35:48.041165 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 19775
2026-09-26T23:35:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4922 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T23:35:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:35:48Z Disabled control of system clock
I20260926 23:35:48.131713 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35729
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 23:35:48.541548 20317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:35:48.541954 20317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:35:48.542068 20317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:35:48.550156 20317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:35:48.550329 20317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:35:48.550369 20317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:35:48.550434 20317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:35:48.550499 20317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:35:48.563423 20317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35729
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:35:48.565304 20317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:35:48.567842 20317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:35:48.584046 20322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:48.584113 20325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:48.584084 20323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:35:48.585398 20317 server_base.cc:1034] running on GCE node
I20260926 23:35:48.586566 20317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:35:48.589131 20317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:35:48.590451 20317 hybrid_clock.cc:648] HybridClock initialized: now 1790465748590393 us; error 58 us; skew 500 ppm
I20260926 23:35:48.591038 20317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:35:48.594149 20317 webserver.cc:460] Webserver started at http://127.0.97.62:34093/ using document root <none> and password file <none>
I20260926 23:35:48.595357 20317 fs_manager.cc:359] Metadata directory not provided
I20260926 23:35:48.595512 20317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:35:48.595983 20317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:35:48.600391 20317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "24c3c01d5ad2465ca41f84e739f974cc"
format_stamp: "Formatted at 2026-09-26 23:35:48 on dist-test-slave-gwfx"
server_key: "f75cef5a40041e6433d334d954da1eb0"
server_key_iv: "2cd10c1bbd10712d7c9e9b732385707e"
server_key_version: "encryptionkey@0"
I20260926 23:35:48.601643 20317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "24c3c01d5ad2465ca41f84e739f974cc"
format_stamp: "Formatted at 2026-09-26 23:35:48 on dist-test-slave-gwfx"
server_key: "f75cef5a40041e6433d334d954da1eb0"
server_key_iv: "2cd10c1bbd10712d7c9e9b732385707e"
server_key_version: "encryptionkey@0"
I20260926 23:35:48.608965 20317 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260926 23:35:48.613620 20331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:48.615716 20317 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:35:48.615996 20317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "24c3c01d5ad2465ca41f84e739f974cc"
format_stamp: "Formatted at 2026-09-26 23:35:48 on dist-test-slave-gwfx"
server_key: "f75cef5a40041e6433d334d954da1eb0"
server_key_iv: "2cd10c1bbd10712d7c9e9b732385707e"
server_key_version: "encryptionkey@0"
I20260926 23:35:48.616276 20317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:35:48.650296 20317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:35:48.651492 20317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:35:48.651829 20317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:35:48.674093 20317 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35729
I20260926 23:35:48.674108 20382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35729 every 8 connection(s)
I20260926 23:35:48.675956 20317 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:35:48.681527 20383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:35:48.685883 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 20317
I20260926 23:35:48.686091 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:35:48.686717 388 external_mini_cluster.cc:1428] Setting key dd76c5706a2e344e19f91ef37ef0349a
I20260926 23:35:48.697280 20383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24c3c01d5ad2465ca41f84e739f974cc: Bootstrap starting.
I20260926 23:35:48.701905 20383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24c3c01d5ad2465ca41f84e739f974cc: Neither blocks nor log segments found. Creating new log.
I20260926 23:35:48.703394 20383 log.cc:824] T 00000000000000000000000000000000 P 24c3c01d5ad2465ca41f84e739f974cc: Log is configured to *not* fsync() on all Append() calls
I20260926 23:35:48.703501 20386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:48.689305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47850 (local address 127.0.97.62:35729)
0926 23:35:48.690656 (+ 1351us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:48.690671 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:48.690773 (+ 102us) server_negotiation.cc:407] Connection header received
0926 23:35:48.692745 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:48.692779 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:48.693465 (+ 686us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:48.693954 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:48.694728 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:48.696206 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:48.697784 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:48.698674 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:48.698969 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:48.700410 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:48.700420 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:48.700441 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:35:48.701143 (+ 702us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:48.701262 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:35:48.701292 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:48.701407 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:48.701684 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 23:35:48.702126 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":799,"thread_start_us":494,"threads_started":1}
I20260926 23:35:48.706982 20383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24c3c01d5ad2465ca41f84e739f974cc: No bootstrap required, opened a new log
I20260926 23:35:48.715832 20383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24c3c01d5ad2465ca41f84e739f974cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c3c01d5ad2465ca41f84e739f974cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35729 } }
I20260926 23:35:48.716369 20383 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24c3c01d5ad2465ca41f84e739f974cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:35:48.716488 20383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24c3c01d5ad2465ca41f84e739f974cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24c3c01d5ad2465ca41f84e739f974cc, State: Initialized, Role: FOLLOWER
I20260926 23:35:48.717236 20383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24c3c01d5ad2465ca41f84e739f974cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c3c01d5ad2465ca41f84e739f974cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35729 } }
I20260926 23:35:48.717506 20383 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24c3c01d5ad2465ca41f84e739f974cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:35:48.717654 20383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24c3c01d5ad2465ca41f84e739f974cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:35:48.717823 20383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24c3c01d5ad2465ca41f84e739f974cc [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:35:48.720867 20383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24c3c01d5ad2465ca41f84e739f974cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c3c01d5ad2465ca41f84e739f974cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35729 } }
I20260926 23:35:48.721593 20383 leader_election.cc:302] T 00000000000000000000000000000000 P 24c3c01d5ad2465ca41f84e739f974cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24c3c01d5ad2465ca41f84e739f974cc; no voters:
I20260926 23:35:48.722630 20383 leader_election.cc:288] T 00000000000000000000000000000000 P 24c3c01d5ad2465ca41f84e739f974cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:35:48.722780 20388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24c3c01d5ad2465ca41f84e739f974cc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:35:48.723656 20388 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24c3c01d5ad2465ca41f84e739f974cc [term 1 LEADER]: Becoming Leader. State: Replica: 24c3c01d5ad2465ca41f84e739f974cc, State: Running, Role: LEADER
I20260926 23:35:48.724351 20388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24c3c01d5ad2465ca41f84e739f974cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c3c01d5ad2465ca41f84e739f974cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35729 } }
I20260926 23:35:48.725299 20383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24c3c01d5ad2465ca41f84e739f974cc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:35:48.727399 20389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c3c01d5ad2465ca41f84e739f974cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24c3c01d5ad2465ca41f84e739f974cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c3c01d5ad2465ca41f84e739f974cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35729 } } }
I20260926 23:35:48.727939 20389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c3c01d5ad2465ca41f84e739f974cc [sys.catalog]: This master's current role is: LEADER
I20260926 23:35:48.728402 20390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c3c01d5ad2465ca41f84e739f974cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24c3c01d5ad2465ca41f84e739f974cc. Latest consensus state: current_term: 1 leader_uuid: "24c3c01d5ad2465ca41f84e739f974cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c3c01d5ad2465ca41f84e739f974cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35729 } } }
I20260926 23:35:48.728847 20390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c3c01d5ad2465ca41f84e739f974cc [sys.catalog]: This master's current role is: LEADER
I20260926 23:35:48.730345 20394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:35:48.738205 20394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:35:48.750118 20394 catalog_manager.cc:1347] Generated new cluster ID: de36b91cf46c4440ad1297ce00e15aaf
I20260926 23:35:48.750285 20394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:35:48.779842 20394 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:35:48.781888 20394 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:35:48.794945 20394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24c3c01d5ad2465ca41f84e739f974cc: Generated new TSK 0
I20260926 23:35:48.796169 20394 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:35:48.810289 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35729
--builtin_ntp_servers=127.0.97.20:38783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 23:35:49.234349 20407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:35:49.234791 20407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:35:49.234942 20407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:35:49.243355 20407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:35:49.243503 20407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:35:49.243649 20407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 23:35:49.254652 20407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:35:49.256516 20407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:35:49.258993 20407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:35:49.274844 20412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:49.274942 20413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:49.274920 20415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:35:49.276104 20407 server_base.cc:1034] running on GCE node
I20260926 23:35:49.276925 20407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:35:49.278146 20407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:35:49.279420 20407 hybrid_clock.cc:648] HybridClock initialized: now 1790465749279354 us; error 46 us; skew 500 ppm
I20260926 23:35:49.279872 20407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:35:49.282930 20407 webserver.cc:460] Webserver started at http://127.0.97.1:40209/ using document root <none> and password file <none>
I20260926 23:35:49.283864 20407 fs_manager.cc:359] Metadata directory not provided
I20260926 23:35:49.283987 20407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:35:49.284404 20407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:35:49.288148 20407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-0/data/instance:
uuid: "2cd4d13141854299a73eb2016cd01f2b"
format_stamp: "Formatted at 2026-09-26 23:35:49 on dist-test-slave-gwfx"
server_key: "cbc8253123d4da6e09a6dfefe58d1819"
server_key_iv: "dea78186b2f994428ff0beead0822a7c"
server_key_version: "encryptionkey@0"
I20260926 23:35:49.289176 20407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2cd4d13141854299a73eb2016cd01f2b"
format_stamp: "Formatted at 2026-09-26 23:35:49 on dist-test-slave-gwfx"
server_key: "cbc8253123d4da6e09a6dfefe58d1819"
server_key_iv: "dea78186b2f994428ff0beead0822a7c"
server_key_version: "encryptionkey@0"
I20260926 23:35:49.296170 20407 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260926 23:35:49.300686 20421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:49.302517 20407 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260926 23:35:49.302758 20407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-0/wal
uuid: "2cd4d13141854299a73eb2016cd01f2b"
format_stamp: "Formatted at 2026-09-26 23:35:49 on dist-test-slave-gwfx"
server_key: "cbc8253123d4da6e09a6dfefe58d1819"
server_key_iv: "dea78186b2f994428ff0beead0822a7c"
server_key_version: "encryptionkey@0"
I20260926 23:35:49.303030 20407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:35:49.319805 20407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:35:49.320991 20407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:35:49.321316 20407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:35:49.322566 20407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:35:49.324618 20407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:35:49.324735 20407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:49.324837 20407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:35:49.324895 20407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:49.360389 20407 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41209
I20260926 23:35:49.360430 20533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41209 every 8 connection(s)
I20260926 23:35:49.362185 20407 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-0/data/info.pb
I20260926 23:35:49.368846 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 20407
I20260926 23:35:49.369019 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-0/wal/instance
I20260926 23:35:49.369495 388 external_mini_cluster.cc:1428] Setting key e1e20f1b09fef044238cf5c5cfa73233
I20260926 23:35:49.372603 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35729
--builtin_ntp_servers=127.0.97.20:38783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 23:35:49.377502 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:49.365603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54979 (local address 127.0.97.62:35729)
0926 23:35:49.366205 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:49.366210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:49.367546 (+ 1336us) server_negotiation.cc:407] Connection header received
0926 23:35:49.369829 (+ 2283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:49.369834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:49.369923 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:49.370032 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:49.371615 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:49.372406 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:49.374263 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:49.374775 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:49.374896 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:49.376533 (+ 1637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:49.376539 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:49.376542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:35:49.376566 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:49.376616 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:35:49.376629 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:49.376710 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:49.376960 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 23:35:49.377090 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":276,"threads_started":1}
I20260926 23:35:49.379058 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:49.366192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35729 (local address 127.0.97.1:54979)
0926 23:35:49.367345 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:49.367398 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:49.369367 (+ 1969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:49.370398 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:49.370411 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:49.370578 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:35:49.371428 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:49.371459 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:49.372575 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:49.372579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:49.374095 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:49.374110 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:49.375044 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:49.375051 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:49.375327 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:49.376128 (+ 801us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:35:49.376131 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:35:49.376186 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:35:49.376304 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:35:49.376721 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:35:49.376735 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:35:49.376745 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 23:35:49.376932 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 23:35:49.377521 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":389,"threads_started":1}
I20260926 23:35:49.381289 20534 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35729
I20260926 23:35:49.381669 20534 heartbeater.cc:464] Registering TS with master...
I20260926 23:35:49.382786 20534 heartbeater.cc:511] Master 127.0.97.62:35729 requested a full tablet report, sending...
I20260926 23:35:49.385629 20348 ts_manager.cc:194] Registered new tserver with Master: 2cd4d13141854299a73eb2016cd01f2b (127.0.97.1:41209)
I20260926 23:35:49.387527 20348 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:54979
W20260926 23:35:49.776263 20538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:35:49.776685 20538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:35:49.776821 20538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:35:49.785292 20538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:35:49.785455 20538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:35:49.785622 20538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 23:35:49.796437 20538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:35:49.798710 20538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:35:49.801344 20538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:35:49.819232 20546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:49.819239 20544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:49.819288 20543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:35:49.821044 20538 server_base.cc:1034] running on GCE node
I20260926 23:35:49.822026 20538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:35:49.823554 20538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:35:49.824903 20538 hybrid_clock.cc:648] HybridClock initialized: now 1790465749824812 us; error 72 us; skew 500 ppm
I20260926 23:35:49.825480 20538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:35:49.828938 20538 webserver.cc:460] Webserver started at http://127.0.97.2:41385/ using document root <none> and password file <none>
I20260926 23:35:49.829972 20538 fs_manager.cc:359] Metadata directory not provided
I20260926 23:35:49.830113 20538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:35:49.830523 20538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:35:49.834499 20538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-1/data/instance:
uuid: "fcb76f7ec09d4adfaf82530e1af647bf"
format_stamp: "Formatted at 2026-09-26 23:35:49 on dist-test-slave-gwfx"
server_key: "c1c2c2803259a6a7a920c543c74e5e00"
server_key_iv: "b8affe69f5190c2ddbb33b3382298949"
server_key_version: "encryptionkey@0"
I20260926 23:35:49.835554 20538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-1/wal/instance:
uuid: "fcb76f7ec09d4adfaf82530e1af647bf"
format_stamp: "Formatted at 2026-09-26 23:35:49 on dist-test-slave-gwfx"
server_key: "c1c2c2803259a6a7a920c543c74e5e00"
server_key_iv: "b8affe69f5190c2ddbb33b3382298949"
server_key_version: "encryptionkey@0"
I20260926 23:35:49.843206 20538 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260926 23:35:49.848403 20552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:49.850412 20538 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260926 23:35:49.850677 20538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-1/wal
uuid: "fcb76f7ec09d4adfaf82530e1af647bf"
format_stamp: "Formatted at 2026-09-26 23:35:49 on dist-test-slave-gwfx"
server_key: "c1c2c2803259a6a7a920c543c74e5e00"
server_key_iv: "b8affe69f5190c2ddbb33b3382298949"
server_key_version: "encryptionkey@0"
I20260926 23:35:49.851022 20538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:35:49.879511 20538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:35:49.880697 20538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:35:49.881033 20538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:35:49.882375 20538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:35:49.884523 20538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:35:49.884660 20538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:49.884800 20538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:35:49.884879 20538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:49.923470 20538 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43419
I20260926 23:35:49.923563 20664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43419 every 8 connection(s)
I20260926 23:35:49.924989 20538 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-1/data/info.pb
I20260926 23:35:49.926908 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 20538
I20260926 23:35:49.927078 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-1/wal/instance
I20260926 23:35:49.927556 388 external_mini_cluster.cc:1428] Setting key ebe8e8aa18738c8d830aef69ed64742a
I20260926 23:35:49.931406 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35729
--builtin_ntp_servers=127.0.97.20:38783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 23:35:49.944556 20668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:49.929857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59073 (local address 127.0.97.62:35729)
0926 23:35:49.930513 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:49.930520 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:49.931011 (+ 491us) server_negotiation.cc:407] Connection header received
0926 23:35:49.933814 (+ 2803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:49.933822 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:49.933926 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:49.934073 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:49.936188 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:49.937291 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:49.940353 (+ 3062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:49.940958 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:49.941097 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:49.943284 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:49.943291 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:49.943295 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:35:49.943325 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:49.943385 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:35:49.943400 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:49.943493 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:49.943754 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 23:35:49.943903 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":847,"threads_started":1}
I20260926 23:35:49.945858 20667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:49.929250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35729 (local address 127.0.97.2:59073)
0926 23:35:49.930789 (+ 1539us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:49.930857 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:49.933366 (+ 2509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:49.934414 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:49.934428 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:49.934589 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:35:49.935887 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:49.935920 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:49.937532 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:49.937538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:49.940090 (+ 2552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:49.940131 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:49.941267 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:49.941275 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:49.941625 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:49.942633 (+ 1008us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:35:49.942637 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:35:49.942734 (+ 97us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:35:49.942945 (+ 211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:35:49.943513 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:35:49.943525 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:35:49.943534 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 23:35:49.943707 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 23:35:49.944184 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":669,"threads_started":1}
I20260926 23:35:49.948027 20665 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35729
I20260926 23:35:49.948426 20665 heartbeater.cc:464] Registering TS with master...
I20260926 23:35:49.949450 20665 heartbeater.cc:511] Master 127.0.97.62:35729 requested a full tablet report, sending...
I20260926 23:35:49.951397 20348 ts_manager.cc:194] Registered new tserver with Master: fcb76f7ec09d4adfaf82530e1af647bf (127.0.97.2:43419)
I20260926 23:35:49.952349 20348 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:59073
W20260926 23:35:50.354600 20669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:35:50.355067 20669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:35:50.355221 20669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:35:50.363857 20669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:35:50.364010 20669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:35:50.364171 20669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 23:35:50.374540 20669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:35:50.376381 20669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:35:50.378810 20669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 23:35:50.392001 20534 heartbeater.cc:503] Master 127.0.97.62:35729 was elected leader, sending a full tablet report...
W20260926 23:35:50.396523 20675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:50.397113 20674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:50.396513 20677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:35:50.397934 20669 server_base.cc:1034] running on GCE node
I20260926 23:35:50.398947 20669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:35:50.400573 20669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:35:50.401876 20669 hybrid_clock.cc:648] HybridClock initialized: now 1790465750401807 us; error 66 us; skew 500 ppm
I20260926 23:35:50.402340 20669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 23:35:50.405761 20669 webserver.cc:460] Webserver started at http://127.0.97.3:38981/ using document root <none> and password file <none>
I20260926 23:35:50.406972 20669 fs_manager.cc:359] Metadata directory not provided
I20260926 23:35:50.407125 20669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:35:50.407543 20669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:35:50.411650 20669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-2/data/instance:
uuid: "39e259ff32bd460ba3082ca3f036b377"
format_stamp: "Formatted at 2026-09-26 23:35:50 on dist-test-slave-gwfx"
server_key: "52a1e608b23f68e81f757b6106fc1c1b"
server_key_iv: "7adaf062a2b42f90e76ee4814131049e"
server_key_version: "encryptionkey@0"
I20260926 23:35:50.412762 20669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-2/wal/instance:
uuid: "39e259ff32bd460ba3082ca3f036b377"
format_stamp: "Formatted at 2026-09-26 23:35:50 on dist-test-slave-gwfx"
server_key: "52a1e608b23f68e81f757b6106fc1c1b"
server_key_iv: "7adaf062a2b42f90e76ee4814131049e"
server_key_version: "encryptionkey@0"
I20260926 23:35:50.420347 20669 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260926 23:35:50.425638 20683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:50.427990 20669 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260926 23:35:50.428274 20669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-2/wal
uuid: "39e259ff32bd460ba3082ca3f036b377"
format_stamp: "Formatted at 2026-09-26 23:35:50 on dist-test-slave-gwfx"
server_key: "52a1e608b23f68e81f757b6106fc1c1b"
server_key_iv: "7adaf062a2b42f90e76ee4814131049e"
server_key_version: "encryptionkey@0"
I20260926 23:35:50.428573 20669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:35:50.445055 20669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:35:50.446182 20669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:35:50.446494 20669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:35:50.447757 20669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 23:35:50.449615 20669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 23:35:50.449756 20669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:50.449899 20669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 23:35:50.449976 20669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:50.489524 20669 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43169
I20260926 23:35:50.489606 20795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43169 every 8 connection(s)
I20260926 23:35:50.491285 20669 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-2/data/info.pb
I20260926 23:35:50.498992 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 20669
I20260926 23:35:50.499192 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-2/wal/instance
I20260926 23:35:50.499742 388 external_mini_cluster.cc:1428] Setting key 788bcc22981542c2355f514b2cd63631
I20260926 23:35:50.509163 20798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:50.495499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46491 (local address 127.0.97.62:35729)
0926 23:35:50.496281 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:50.496287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:50.497893 (+ 1606us) server_negotiation.cc:407] Connection header received
0926 23:35:50.500724 (+ 2831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:50.500732 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:50.500856 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:50.500994 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:50.503155 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:50.504040 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:50.505815 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:50.506262 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:50.506396 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:50.508125 (+ 1729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:50.508130 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:50.508134 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:35:50.508159 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:50.508207 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:35:50.508220 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:50.508311 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:50.508594 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 23:35:50.508727 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":438,"threads_started":1}
I20260926 23:35:50.510437 20799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:50.496160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35729 (local address 127.0.97.3:46491)
0926 23:35:50.497638 (+ 1478us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:50.497717 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:50.500253 (+ 2536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:50.501590 (+ 1337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:50.501607 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:50.501781 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:35:50.502918 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:50.502955 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:50.504222 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:50.504228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:50.505643 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:50.505659 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:50.506530 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:50.506538 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:50.506877 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:50.507720 (+ 843us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 23:35:50.507722 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:35:50.507778 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 23:35:50.507893 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:35:50.508314 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:35:50.508327 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:35:50.508337 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 23:35:50.508556 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 23:35:50.509089 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":406,"threads_started":1}
I20260926 23:35:50.512362 20796 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35729
I20260926 23:35:50.512763 20796 heartbeater.cc:464] Registering TS with master...
I20260926 23:35:50.513605 20796 heartbeater.cc:511] Master 127.0.97.62:35729 requested a full tablet report, sending...
I20260926 23:35:50.515403 20348 ts_manager.cc:194] Registered new tserver with Master: 39e259ff32bd460ba3082ca3f036b377 (127.0.97.3:43169)
I20260926 23:35:50.516531 20348 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:46491
I20260926 23:35:50.517544 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 23:35:50.529134 20798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:50.522402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47862 (local address 127.0.97.62:35729)
0926 23:35:50.522736 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:50.522743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:50.523239 (+ 496us) server_negotiation.cc:407] Connection header received
0926 23:35:50.523449 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:50.523454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:50.523528 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:50.523623 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:50.524163 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:50.524961 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:50.526497 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:50.527026 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:50.527153 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:50.528361 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:50.528367 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:50.528370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:35:50.528395 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:50.528433 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:35:50.528446 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:50.528527 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:50.528664 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 23:35:50.528782 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 23:35:50.535120 20347 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 23:35:50.543434 20347 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 23:35:50.587165 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:50.561578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53116 (local address 127.0.97.3:43169)
0926 23:35:50.576163 (+ 14585us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:50.576178 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:50.576980 (+ 802us) server_negotiation.cc:407] Connection header received
0926 23:35:50.577380 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:50.577388 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:50.577764 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:50.578046 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:35:50.579820 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:50.581461 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:50.583927 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:50.585633 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:50.585777 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:50.586143 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:50.586494 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 23:35:50.586731 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14312,"thread_start_us":14105,"threads_started":1}
I20260926 23:35:50.587255 20809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:50.560380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41209 (local address 127.0.0.1:43438)
0926 23:35:50.561596 (+ 1216us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:50.561666 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:50.561955 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:50.578501 (+ 16546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:50.578506 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:50.578539 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:35:50.579130 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:50.579147 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:50.580831 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:50.580838 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:50.583850 (+ 3012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:50.583868 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:50.586232 (+ 2364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:50.586274 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:50.586490 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:50.586533 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 23:35:50.586623 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 23:35:50.586806 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":42,"thread_start_us":392,"threads_started":1}
I20260926 23:35:50.587786 20813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:50.561939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43419 (local address 127.0.0.1:51558)
0926 23:35:50.575611 (+ 13672us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:50.575642 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:50.575851 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:50.578111 (+ 2260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:50.578123 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:50.578163 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:35:50.578729 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:50.578759 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:50.581164 (+ 2405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:50.581170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:50.583854 (+ 2684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:50.583869 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:50.586543 (+ 2674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:50.586549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:50.586797 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:50.586826 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 23:35:50.586908 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 23:35:50.587058 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13476,"mutex_wait_us":91,"thread_start_us":14524,"threads_started":1}
I20260926 23:35:50.587965 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:50.561008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43169 (local address 127.0.0.1:53116)
0926 23:35:50.576796 (+ 15788us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:50.576820 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:50.577137 (+ 317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:50.578435 (+ 1298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:50.578441 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:50.578465 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:35:50.579595 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:50.579608 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:50.581632 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:50.581638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:50.583640 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:50.583660 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:50.585920 (+ 2260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:50.585929 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:50.586190 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:50.586257 (+ 67us) client_negotiation.cc:769] Sending connection context
0926 23:35:50.586405 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 23:35:50.586680 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15635,"mutex_wait_us":1274,"thread_start_us":12916,"threads_started":1}
I20260926 23:35:50.588718 20814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:50.562411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51558 (local address 127.0.97.2:43419)
0926 23:35:50.577170 (+ 14759us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:50.577190 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:50.577222 (+ 32us) server_negotiation.cc:407] Connection header received
0926 23:35:50.577375 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:50.577385 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:50.577762 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:50.579024 (+ 1262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:35:50.579329 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:50.581002 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:50.584146 (+ 3144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:50.585960 (+ 1814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:50.586137 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:50.586400 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:50.588225 (+ 1825us) server_negotiation.cc:299] Negotiation successful
0926 23:35:50.588417 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14602,"thread_start_us":14462,"threads_started":1}
I20260926 23:35:50.588831 20810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:50.560686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43438 (local address 127.0.97.1:41209)
0926 23:35:50.577809 (+ 17123us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:50.577816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:50.577844 (+ 28us) server_negotiation.cc:407] Connection header received
0926 23:35:50.577973 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:50.577980 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:50.578219 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:50.578854 (+ 635us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:35:50.579333 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:50.580657 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:50.584147 (+ 3490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:50.585961 (+ 1814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:50.587018 (+ 1057us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:50.587333 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:50.588211 (+ 878us) server_negotiation.cc:299] Negotiation successful
0926 23:35:50.588446 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16926,"thread_start_us":16753,"threads_started":1}
I20260926 23:35:50.593343 20600 tablet_service.cc:1467] Processing CreateTablet for tablet 4489c1f1c3a3423c84e5c7c48b078413 (DEFAULT_TABLE table=test-table [id=73eb8f8c277d4bd3ba50dd6509e559f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:35:50.593344 20731 tablet_service.cc:1467] Processing CreateTablet for tablet 4489c1f1c3a3423c84e5c7c48b078413 (DEFAULT_TABLE table=test-table [id=73eb8f8c277d4bd3ba50dd6509e559f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:35:50.595412 20731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4489c1f1c3a3423c84e5c7c48b078413. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:35:50.595414 20600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4489c1f1c3a3423c84e5c7c48b078413. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:35:50.597931 20469 tablet_service.cc:1467] Processing CreateTablet for tablet 4489c1f1c3a3423c84e5c7c48b078413 (DEFAULT_TABLE table=test-table [id=73eb8f8c277d4bd3ba50dd6509e559f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 23:35:50.599530 20469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4489c1f1c3a3423c84e5c7c48b078413. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:35:50.615783 20816 tablet_bootstrap.cc:492] T 4489c1f1c3a3423c84e5c7c48b078413 P 2cd4d13141854299a73eb2016cd01f2b: Bootstrap starting.
I20260926 23:35:50.614746 20815 tablet_bootstrap.cc:492] T 4489c1f1c3a3423c84e5c7c48b078413 P 39e259ff32bd460ba3082ca3f036b377: Bootstrap starting.
I20260926 23:35:50.619354 20817 tablet_bootstrap.cc:492] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf: Bootstrap starting.
I20260926 23:35:50.620802 20816 tablet_bootstrap.cc:654] T 4489c1f1c3a3423c84e5c7c48b078413 P 2cd4d13141854299a73eb2016cd01f2b: Neither blocks nor log segments found. Creating new log.
I20260926 23:35:50.620850 20815 tablet_bootstrap.cc:654] T 4489c1f1c3a3423c84e5c7c48b078413 P 39e259ff32bd460ba3082ca3f036b377: Neither blocks nor log segments found. Creating new log.
I20260926 23:35:50.622418 20816 log.cc:824] T 4489c1f1c3a3423c84e5c7c48b078413 P 2cd4d13141854299a73eb2016cd01f2b: Log is configured to *not* fsync() on all Append() calls
I20260926 23:35:50.622417 20815 log.cc:824] T 4489c1f1c3a3423c84e5c7c48b078413 P 39e259ff32bd460ba3082ca3f036b377: Log is configured to *not* fsync() on all Append() calls
I20260926 23:35:50.622910 20817 tablet_bootstrap.cc:654] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf: Neither blocks nor log segments found. Creating new log.
I20260926 23:35:50.624441 20817 log.cc:824] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf: Log is configured to *not* fsync() on all Append() calls
I20260926 23:35:50.626920 20815 tablet_bootstrap.cc:492] T 4489c1f1c3a3423c84e5c7c48b078413 P 39e259ff32bd460ba3082ca3f036b377: No bootstrap required, opened a new log
I20260926 23:35:50.626927 20816 tablet_bootstrap.cc:492] T 4489c1f1c3a3423c84e5c7c48b078413 P 2cd4d13141854299a73eb2016cd01f2b: No bootstrap required, opened a new log
I20260926 23:35:50.627274 20815 ts_tablet_manager.cc:1397] T 4489c1f1c3a3423c84e5c7c48b078413 P 39e259ff32bd460ba3082ca3f036b377: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260926 23:35:50.627280 20816 ts_tablet_manager.cc:1397] T 4489c1f1c3a3423c84e5c7c48b078413 P 2cd4d13141854299a73eb2016cd01f2b: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260926 23:35:50.629029 20817 tablet_bootstrap.cc:492] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf: No bootstrap required, opened a new log
I20260926 23:35:50.629385 20817 ts_tablet_manager.cc:1397] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260926 23:35:50.636386 20815 raft_consensus.cc:348] T 4489c1f1c3a3423c84e5c7c48b078413 P 39e259ff32bd460ba3082ca3f036b377 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd4d13141854299a73eb2016cd01f2b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41209 } } peers { permanent_uuid: "fcb76f7ec09d4adfaf82530e1af647bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43419 } } peers { permanent_uuid: "39e259ff32bd460ba3082ca3f036b377" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43169 } }
I20260926 23:35:50.636978 20815 raft_consensus.cc:374] T 4489c1f1c3a3423c84e5c7c48b078413 P 39e259ff32bd460ba3082ca3f036b377 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:35:50.637094 20815 raft_consensus.cc:729] T 4489c1f1c3a3423c84e5c7c48b078413 P 39e259ff32bd460ba3082ca3f036b377 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39e259ff32bd460ba3082ca3f036b377, State: Initialized, Role: FOLLOWER
I20260926 23:35:50.637970 20815 consensus_queue.cc:260] T 4489c1f1c3a3423c84e5c7c48b078413 P 39e259ff32bd460ba3082ca3f036b377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd4d13141854299a73eb2016cd01f2b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41209 } } peers { permanent_uuid: "fcb76f7ec09d4adfaf82530e1af647bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43419 } } peers { permanent_uuid: "39e259ff32bd460ba3082ca3f036b377" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43169 } }
I20260926 23:35:50.639197 20816 raft_consensus.cc:348] T 4489c1f1c3a3423c84e5c7c48b078413 P 2cd4d13141854299a73eb2016cd01f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd4d13141854299a73eb2016cd01f2b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41209 } } peers { permanent_uuid: "fcb76f7ec09d4adfaf82530e1af647bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43419 } } peers { permanent_uuid: "39e259ff32bd460ba3082ca3f036b377" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43169 } }
I20260926 23:35:50.639834 20816 raft_consensus.cc:374] T 4489c1f1c3a3423c84e5c7c48b078413 P 2cd4d13141854299a73eb2016cd01f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:35:50.639956 20816 raft_consensus.cc:729] T 4489c1f1c3a3423c84e5c7c48b078413 P 2cd4d13141854299a73eb2016cd01f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd4d13141854299a73eb2016cd01f2b, State: Initialized, Role: FOLLOWER
I20260926 23:35:50.640501 20796 heartbeater.cc:503] Master 127.0.97.62:35729 was elected leader, sending a full tablet report...
I20260926 23:35:50.640830 20816 consensus_queue.cc:260] T 4489c1f1c3a3423c84e5c7c48b078413 P 2cd4d13141854299a73eb2016cd01f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd4d13141854299a73eb2016cd01f2b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41209 } } peers { permanent_uuid: "fcb76f7ec09d4adfaf82530e1af647bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43419 } } peers { permanent_uuid: "39e259ff32bd460ba3082ca3f036b377" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43169 } }
I20260926 23:35:50.641355 20815 ts_tablet_manager.cc:1428] T 4489c1f1c3a3423c84e5c7c48b078413 P 39e259ff32bd460ba3082ca3f036b377: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260926 23:35:50.641904 20817 raft_consensus.cc:348] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd4d13141854299a73eb2016cd01f2b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41209 } } peers { permanent_uuid: "fcb76f7ec09d4adfaf82530e1af647bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43419 } } peers { permanent_uuid: "39e259ff32bd460ba3082ca3f036b377" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43169 } }
I20260926 23:35:50.642390 20817 raft_consensus.cc:374] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:35:50.642503 20817 raft_consensus.cc:729] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcb76f7ec09d4adfaf82530e1af647bf, State: Initialized, Role: FOLLOWER
I20260926 23:35:50.643362 20817 consensus_queue.cc:260] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd4d13141854299a73eb2016cd01f2b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41209 } } peers { permanent_uuid: "fcb76f7ec09d4adfaf82530e1af647bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43419 } } peers { permanent_uuid: "39e259ff32bd460ba3082ca3f036b377" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43169 } }
I20260926 23:35:50.644555 20816 ts_tablet_manager.cc:1428] T 4489c1f1c3a3423c84e5c7c48b078413 P 2cd4d13141854299a73eb2016cd01f2b: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260926 23:35:50.645819 20665 heartbeater.cc:503] Master 127.0.97.62:35729 was elected leader, sending a full tablet report...
I20260926 23:35:50.646824 20817 ts_tablet_manager.cc:1428] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf: Time spent starting tablet: real 0.017s user 0.014s sys 0.005s
I20260926 23:35:50.652534 20823 raft_consensus.cc:482] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 23:35:50.652997 20823 raft_consensus.cc:504] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd4d13141854299a73eb2016cd01f2b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41209 } } peers { permanent_uuid: "fcb76f7ec09d4adfaf82530e1af647bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43419 } } peers { permanent_uuid: "39e259ff32bd460ba3082ca3f036b377" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43169 } }
I20260926 23:35:50.655288 20823 leader_election.cc:288] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cd4d13141854299a73eb2016cd01f2b (127.0.97.1:41209), 39e259ff32bd460ba3082ca3f036b377 (127.0.97.3:43169)
I20260926 23:35:50.663587 20824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:50.655722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43169 (local address 127.0.97.2:51409)
0926 23:35:50.656660 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:50.656684 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:50.656921 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:50.657396 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:50.657400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:50.657430 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:35:50.657868 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:50.657884 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:50.659263 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:50.659270 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:50.661014 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:50.661029 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:50.662568 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:50.662576 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:50.662782 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:50.662867 (+ 85us) client_negotiation.cc:769] Sending connection context
0926 23:35:50.662971 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 23:35:50.663168 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":706,"threads_started":1}
I20260926 23:35:50.663564 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:50.656394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51409 (local address 127.0.97.3:43169)
0926 23:35:50.656585 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:50.656591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:50.656833 (+ 242us) server_negotiation.cc:407] Connection header received
0926 23:35:50.657122 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:50.657127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:50.657224 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:50.657357 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:35:50.658035 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:50.659129 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:50.661244 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:50.662338 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:50.662465 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:50.662568 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:50.663043 (+ 475us) server_negotiation.cc:299] Negotiation successful
0926 23:35:50.663263 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 23:35:50.663727 20810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:50.655919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45159 (local address 127.0.97.1:41209)
0926 23:35:50.656173 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:50.656180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:50.657482 (+ 1302us) server_negotiation.cc:407] Connection header received
0926 23:35:50.657688 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:50.657694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:50.657810 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:50.657924 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 23:35:50.658649 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:50.659655 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:50.661720 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:50.662677 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:50.662848 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:50.662952 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:50.663319 (+ 367us) server_negotiation.cc:299] Negotiation successful
0926 23:35:50.663483 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 23:35:50.664062 20825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:50.656460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41209 (local address 127.0.97.2:45159)
0926 23:35:50.657348 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:50.657373 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 23:35:50.657510 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:35:50.657956 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:35:50.657961 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:35:50.657989 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 23:35:50.658431 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:50.658443 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:50.659833 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:50.659839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:50.661487 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:35:50.661503 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:35:50.662904 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:35:50.662910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:35:50.663093 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:50.663130 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 23:35:50.663236 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 23:35:50.663383 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":511,"threads_started":1}
I20260926 23:35:50.665015 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4489c1f1c3a3423c84e5c7c48b078413" candidate_uuid: "fcb76f7ec09d4adfaf82530e1af647bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39e259ff32bd460ba3082ca3f036b377" is_pre_election: true
I20260926 23:35:50.665301 20489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4489c1f1c3a3423c84e5c7c48b078413" candidate_uuid: "fcb76f7ec09d4adfaf82530e1af647bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd4d13141854299a73eb2016cd01f2b" is_pre_election: true
I20260926 23:35:50.665701 20751 raft_consensus.cc:2454] T 4489c1f1c3a3423c84e5c7c48b078413 P 39e259ff32bd460ba3082ca3f036b377 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcb76f7ec09d4adfaf82530e1af647bf in term 0.
I20260926 23:35:50.665935 20489 raft_consensus.cc:2454] T 4489c1f1c3a3423c84e5c7c48b078413 P 2cd4d13141854299a73eb2016cd01f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcb76f7ec09d4adfaf82530e1af647bf in term 0.
I20260926 23:35:50.666724 20556 leader_election.cc:302] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39e259ff32bd460ba3082ca3f036b377, fcb76f7ec09d4adfaf82530e1af647bf; no voters:
I20260926 23:35:50.667326 20823 raft_consensus.cc:2781] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 23:35:50.667513 20823 raft_consensus.cc:482] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 23:35:50.667637 20823 raft_consensus.cc:3037] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:35:50.670604 20823 raft_consensus.cc:504] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd4d13141854299a73eb2016cd01f2b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41209 } } peers { permanent_uuid: "fcb76f7ec09d4adfaf82530e1af647bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43419 } } peers { permanent_uuid: "39e259ff32bd460ba3082ca3f036b377" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43169 } }
I20260926 23:35:50.671532 20823 leader_election.cc:288] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf [CANDIDATE]: Term 1 election: Requested vote from peers 2cd4d13141854299a73eb2016cd01f2b (127.0.97.1:41209), 39e259ff32bd460ba3082ca3f036b377 (127.0.97.3:43169)
I20260926 23:35:50.672314 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4489c1f1c3a3423c84e5c7c48b078413" candidate_uuid: "fcb76f7ec09d4adfaf82530e1af647bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39e259ff32bd460ba3082ca3f036b377"
I20260926 23:35:50.672399 20489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4489c1f1c3a3423c84e5c7c48b078413" candidate_uuid: "fcb76f7ec09d4adfaf82530e1af647bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd4d13141854299a73eb2016cd01f2b"
I20260926 23:35:50.672574 20751 raft_consensus.cc:3037] T 4489c1f1c3a3423c84e5c7c48b078413 P 39e259ff32bd460ba3082ca3f036b377 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:35:50.672708 20489 raft_consensus.cc:3037] T 4489c1f1c3a3423c84e5c7c48b078413 P 2cd4d13141854299a73eb2016cd01f2b [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:35:50.676079 20751 raft_consensus.cc:2454] T 4489c1f1c3a3423c84e5c7c48b078413 P 39e259ff32bd460ba3082ca3f036b377 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcb76f7ec09d4adfaf82530e1af647bf in term 1.
I20260926 23:35:50.676614 20489 raft_consensus.cc:2454] T 4489c1f1c3a3423c84e5c7c48b078413 P 2cd4d13141854299a73eb2016cd01f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcb76f7ec09d4adfaf82530e1af647bf in term 1.
I20260926 23:35:50.676823 20556 leader_election.cc:302] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39e259ff32bd460ba3082ca3f036b377, fcb76f7ec09d4adfaf82530e1af647bf; no voters:
I20260926 23:35:50.677385 20823 raft_consensus.cc:2781] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf [term 1 FOLLOWER]: Leader election won for term 1
W20260926 23:35:50.677739 20666 tablet.cc:2369] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:35:50.678084 20823 raft_consensus.cc:686] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf [term 1 LEADER]: Becoming Leader. State: Replica: fcb76f7ec09d4adfaf82530e1af647bf, State: Running, Role: LEADER
I20260926 23:35:50.678848 20823 consensus_queue.cc:237] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd4d13141854299a73eb2016cd01f2b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41209 } } peers { permanent_uuid: "fcb76f7ec09d4adfaf82530e1af647bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43419 } } peers { permanent_uuid: "39e259ff32bd460ba3082ca3f036b377" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43169 } }
I20260926 23:35:50.684865 20346 catalog_manager.cc:5495] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf reported cstate change: term changed from 0 to 1, leader changed from <none> to fcb76f7ec09d4adfaf82530e1af647bf (127.0.97.2). New cstate: current_term: 1 leader_uuid: "fcb76f7ec09d4adfaf82530e1af647bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd4d13141854299a73eb2016cd01f2b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fcb76f7ec09d4adfaf82530e1af647bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39e259ff32bd460ba3082ca3f036b377" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43169 } health_report { overall_health: UNKNOWN } } }
I20260926 23:35:50.717826 20814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:50.711465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51562 (local address 127.0.97.2:43419)
0926 23:35:50.711699 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:50.711706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:50.711733 (+ 27us) server_negotiation.cc:407] Connection header received
0926 23:35:50.712007 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:50.712016 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:50.712207 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:50.712396 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:35:50.712819 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:50.713612 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:50.715228 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:50.715706 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:50.715830 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:50.716439 (+ 609us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:35:50.717102 (+ 663us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:35:50.717286 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:50.717374 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 23:35:50.717479 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 23:35:50.728531 20751 raft_consensus.cc:1261] T 4489c1f1c3a3423c84e5c7c48b078413 P 39e259ff32bd460ba3082ca3f036b377 [term 1 FOLLOWER]: Refusing update from remote peer fcb76f7ec09d4adfaf82530e1af647bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:35:50.728531 20489 raft_consensus.cc:1261] T 4489c1f1c3a3423c84e5c7c48b078413 P 2cd4d13141854299a73eb2016cd01f2b [term 1 FOLLOWER]: Refusing update from remote peer fcb76f7ec09d4adfaf82530e1af647bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:35:50.729770 20826 consensus_queue.cc:1035] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "39e259ff32bd460ba3082ca3f036b377" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:35:50.730185 20823 consensus_queue.cc:1035] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd4d13141854299a73eb2016cd01f2b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:35:50.743244 20830 mvcc.cc:205] Tried to move back new op lower bound from 7333747714971840512 to 7333747714785619968. Current Snapshot: MvccSnapshot[applied={T|T < 7333747714971840512}]
I20260926 23:35:50.763887 20347 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:35:50.764369 20347 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 23:35:50.769354 20347 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24c3c01d5ad2465ca41f84e739f974cc: Sending DeleteTablet for 3 replicas of tablet 4489c1f1c3a3423c84e5c7c48b078413
I20260926 23:35:50.771102 20731 tablet_service.cc:1514] Processing DeleteTablet for tablet 4489c1f1c3a3423c84e5c7c48b078413 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:35:50 UTC) from {username='slave'} at 127.0.0.1:53116
I20260926 23:35:50.771102 20469 tablet_service.cc:1514] Processing DeleteTablet for tablet 4489c1f1c3a3423c84e5c7c48b078413 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:35:50 UTC) from {username='slave'} at 127.0.0.1:43438
I20260926 23:35:50.771411 20600 tablet_service.cc:1514] Processing DeleteTablet for tablet 4489c1f1c3a3423c84e5c7c48b078413 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 23:35:50 UTC) from {username='slave'} at 127.0.0.1:51558
I20260926 23:35:50.774250 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 20407
I20260926 23:35:50.789733 20840 tablet_replica.cc:330] T 4489c1f1c3a3423c84e5c7c48b078413 P 39e259ff32bd460ba3082ca3f036b377: stopping tablet replica
I20260926 23:35:50.790619 20840 raft_consensus.cc:2229] T 4489c1f1c3a3423c84e5c7c48b078413 P 39e259ff32bd460ba3082ca3f036b377 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:35:50.791497 20840 raft_consensus.cc:2258] T 4489c1f1c3a3423c84e5c7c48b078413 P 39e259ff32bd460ba3082ca3f036b377 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:35:50.794780 20840 ts_tablet_manager.cc:1905] T 4489c1f1c3a3423c84e5c7c48b078413 P 39e259ff32bd460ba3082ca3f036b377: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:35:50.795727 20839 tablet_replica.cc:330] T 4489c1f1c3a3423c84e5c7c48b078413 P 2cd4d13141854299a73eb2016cd01f2b: stopping tablet replica
I20260926 23:35:50.796365 20839 raft_consensus.cc:2229] T 4489c1f1c3a3423c84e5c7c48b078413 P 2cd4d13141854299a73eb2016cd01f2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 23:35:50.797070 20839 raft_consensus.cc:2258] T 4489c1f1c3a3423c84e5c7c48b078413 P 2cd4d13141854299a73eb2016cd01f2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:35:50.798153 20841 tablet_replica.cc:330] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf: stopping tablet replica
I20260926 23:35:50.798730 20841 raft_consensus.cc:2229] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf [term 1 LEADER]: Raft consensus shutting down.
I20260926 23:35:50.799575 20841 raft_consensus.cc:2258] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 23:35:50.800542 20810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:50.775644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43442 (local address 127.0.97.1:41209)
0926 23:35:50.775870 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:50.775877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:50.780460 (+ 4583us) server_negotiation.cc:407] Connection header received
0926 23:35:50.781509 (+ 1049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:50.781518 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:50.781711 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:50.781809 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:50.783806 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:50.785279 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:50.789897 (+ 4618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:50.790626 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:50.790782 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:50.797229 (+ 6447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:50.797244 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:50.797281 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:35:50.798278 (+ 997us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:50.798385 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:35:50.798425 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:50.798587 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:50.799943 (+ 1356us) server_negotiation.cc:299] Negotiation successful
0926 23:35:50.800115 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 23:35:50.801285 20839 ts_tablet_manager.cc:1905] T 4489c1f1c3a3423c84e5c7c48b078413 P 2cd4d13141854299a73eb2016cd01f2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:35:50.802037 20529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:35:50.803300 20841 ts_tablet_manager.cc:1905] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 23:35:50.810526 20840 ts_tablet_manager.cc:1918] T 4489c1f1c3a3423c84e5c7c48b078413 P 39e259ff32bd460ba3082ca3f036b377: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:35:50.811038 20840 log.cc:1196] T 4489c1f1c3a3423c84e5c7c48b078413 P 39e259ff32bd460ba3082ca3f036b377: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-2/wal/wals/4489c1f1c3a3423c84e5c7c48b078413
I20260926 23:35:50.811753 20840 ts_tablet_manager.cc:1939] T 4489c1f1c3a3423c84e5c7c48b078413 P 39e259ff32bd460ba3082ca3f036b377: Deleting consensus metadata
I20260926 23:35:50.814370 20335 catalog_manager.cc:4841] TS 39e259ff32bd460ba3082ca3f036b377 (127.0.97.3:43169): tablet 4489c1f1c3a3423c84e5c7c48b078413 (table test-table [id=73eb8f8c277d4bd3ba50dd6509e559f4]) successfully deleted
I20260926 23:35:50.814762 20841 ts_tablet_manager.cc:1918] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 23:35:50.815052 20841 log.cc:1196] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf: Deleting WAL directory at /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790465400694755-388-0/minicluster-data/ts-1/wal/wals/4489c1f1c3a3423c84e5c7c48b078413
I20260926 23:35:50.815590 20841 ts_tablet_manager.cc:1939] T 4489c1f1c3a3423c84e5c7c48b078413 P fcb76f7ec09d4adfaf82530e1af647bf: Deleting consensus metadata
I20260926 23:35:50.817572 20333 catalog_manager.cc:4841] TS fcb76f7ec09d4adfaf82530e1af647bf (127.0.97.2:43419): tablet 4489c1f1c3a3423c84e5c7c48b078413 (table test-table [id=73eb8f8c277d4bd3ba50dd6509e559f4]) successfully deleted
I20260926 23:35:50.919341 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 20407
W20260926 23:35:50.946302 20333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:41209 (error 108)
I20260926 23:35:50.946492 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 20538
I20260926 23:35:50.952450 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:50.951912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41209 (local address 127.0.0.1:43448)
0926 23:35:50.952142 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:50.952266 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 23:35:50.952795 20333 catalog_manager.cc:4584] TS 2cd4d13141854299a73eb2016cd01f2b (127.0.97.1:41209): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4489c1f1c3a3423c84e5c7c48b078413: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41209: connect: Connection refused (error 111)
I20260926 23:35:50.957108 20814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:50.947552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51568 (local address 127.0.97.2:43419)
0926 23:35:50.948248 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:50.948255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:50.948276 (+ 21us) server_negotiation.cc:407] Connection header received
0926 23:35:50.948367 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:50.948373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:50.948521 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:50.948636 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:50.949359 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:50.950717 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:50.952844 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:50.953594 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:50.953731 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:50.954993 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:50.955006 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:50.955031 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:35:50.956084 (+ 1053us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:50.956186 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:35:50.956238 (+ 52us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:50.956353 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:50.956519 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 23:35:50.956644 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553}
I20260926 23:35:50.957988 20660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:35:51.006986 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:51.006325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41209 (local address 127.0.0.1:43460)
0926 23:35:51.006574 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:51.006737 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260926 23:35:51.008533 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:51.008046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41209 (local address 127.0.0.1:43470)
0926 23:35:51.008244 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:51.008346 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 23:35:51.069685 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:51.069020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41209 (local address 127.0.0.1:43478)
0926 23:35:51.069314 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:51.069450 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260926 23:35:51.070962 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:51.070534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41209 (local address 127.0.0.1:43482)
0926 23:35:51.070662 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:51.070833 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 23:35:51.085474 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 20538
I20260926 23:35:51.105633 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 20669
I20260926 23:35:51.113253 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:51.106566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53120 (local address 127.0.97.3:43169)
0926 23:35:51.106817 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:51.106868 (+ 51us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:51.106897 (+ 29us) server_negotiation.cc:407] Connection header received
0926 23:35:51.107041 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:51.107046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:51.107145 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:51.107251 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:51.107683 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:51.108524 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:51.109873 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:51.110292 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:51.110407 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:51.111668 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:51.111678 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:51.111699 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:35:51.112415 (+ 716us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:51.112475 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:35:51.112498 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:51.112616 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:51.112748 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 23:35:51.112892 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 23:35:51.114092 20791 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:35:51.151769 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:51.151213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41209 (local address 127.0.0.1:43492)
0926 23:35:51.151453 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:51.151558 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 23:35:51.152959 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:51.152492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41209 (local address 127.0.0.1:43504)
0926 23:35:51.152728 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:51.152821 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260926 23:35:51.210021 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 20669
I20260926 23:35:51.226783 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 20317
I20260926 23:35:51.228268 20378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:35:51.358191 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:51.357491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41209 (local address 127.0.0.1:43514)
0926 23:35:51.357785 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:51.357925 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 23:35:51.358151 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 20317
I20260926 23:35:51.359546 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:51.358952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41209 (local address 127.0.0.1:43524)
0926 23:35:51.359191 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 23:35:51.359378 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
2026-09-26T23:35:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3321 ms)
[----------] 6 tests from EncryptionPolicyTest (25056 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:35:51 dist-test-slave-gwfx krb5kdc[20850](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:35:51 dist-test-slave-gwfx krb5kdc[20850](info): set up 2 sockets
Sep 26 23:35:51 dist-test-slave-gwfx krb5kdc[20850](info): commencing operation
krb5kdc: starting...
W20260926 23:35:52.427996 388 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.006s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:35:52 dist-test-slave-gwfx krb5kdc[20850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:35:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:35:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:35:52.760426 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42067
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:35:53.201088 20863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:35:53.201553 20863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:35:53.201700 20863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:35:53.210280 20863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:35:53.210479 20863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:35:53.210561 20863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:35:53.210650 20863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:35:53.210747 20863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:35:53.223341 20863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42067
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:35:53.225266 20863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:35:53.227205 20863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:35:53.242390 20868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:53.242360 20871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:53.242412 20869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:35:53.242877 20863 server_base.cc:1034] running on GCE node
I20260926 23:35:53.244185 20863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:35:53.246028 20863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:35:53.247308 20863 hybrid_clock.cc:648] HybridClock initialized: now 1790465753247247 us; error 52 us; skew 500 ppm
I20260926 23:35:53.247828 20863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:35:53 dist-test-slave-gwfx krb5kdc[20850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:35:53.251787 20863 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:35:53.253759 20863 webserver.cc:460] Webserver started at http://127.0.0.1:34097/ using document root <none> and password file <none>
I20260926 23:35:53.254614 20863 fs_manager.cc:359] Metadata directory not provided
I20260926 23:35:53.254794 20863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:35:53.255134 20863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:35:53.258847 20863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "a1ac9af8e8f74b1d847a5ca22126173e"
format_stamp: "Formatted at 2026-09-26 23:35:53 on dist-test-slave-gwfx"
server_key: "d5afedcd31553eb9ecce8bab1aae44b0"
server_key_iv: "3b2b67f7cf0c719dd9293ecfd67d7cc9"
server_key_version: "encryptionkey@0"
I20260926 23:35:53.259742 20863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "a1ac9af8e8f74b1d847a5ca22126173e"
format_stamp: "Formatted at 2026-09-26 23:35:53 on dist-test-slave-gwfx"
server_key: "d5afedcd31553eb9ecce8bab1aae44b0"
server_key_iv: "3b2b67f7cf0c719dd9293ecfd67d7cc9"
server_key_version: "encryptionkey@0"
I20260926 23:35:53.267300 20863 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260926 23:35:53.273442 20878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:53.275638 20863 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260926 23:35:53.275949 20863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "a1ac9af8e8f74b1d847a5ca22126173e"
format_stamp: "Formatted at 2026-09-26 23:35:53 on dist-test-slave-gwfx"
server_key: "d5afedcd31553eb9ecce8bab1aae44b0"
server_key_iv: "3b2b67f7cf0c719dd9293ecfd67d7cc9"
server_key_version: "encryptionkey@0"
I20260926 23:35:53.276202 20863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:35:53.292804 20863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:35:53.300171 20863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:35:53.300499 20863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:35:53.322938 20863 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42067
I20260926 23:35:53.322942 20929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42067 every 8 connection(s)
I20260926 23:35:53.325009 20863 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:35:53.326124 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 20863
I20260926 23:35:53.326370 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:35:53.327248 388 external_mini_cluster.cc:1428] Setting key ff85c7e71b7f1493c6e4a18130846e9a
I20260926 23:35:53.331136 20930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 23:35:53 dist-test-slave-gwfx krb5kdc[20850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 23:35:53.345233 20930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1ac9af8e8f74b1d847a5ca22126173e: Bootstrap starting.
I20260926 23:35:53.349943 20930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1ac9af8e8f74b1d847a5ca22126173e: Neither blocks nor log segments found. Creating new log.
I20260926 23:35:53.351440 20930 log.cc:824] T 00000000000000000000000000000000 P a1ac9af8e8f74b1d847a5ca22126173e: Log is configured to *not* fsync() on all Append() calls
I20260926 23:35:53.354058 20933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:53.329684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52183 (local address 127.0.0.1:42067)
0926 23:35:53.330764 (+ 1080us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:53.330782 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:53.330835 (+ 53us) server_negotiation.cc:407] Connection header received
0926 23:35:53.332776 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:53.332806 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:53.333280 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:53.333740 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:53.335467 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:53.337268 (+ 1801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:53.338902 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:53.339703 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:53.339915 (+ 212us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:53.344730 (+ 4815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:53.344749 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:53.344770 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:35:53.344821 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:53.348440 (+ 3619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:53.349886 (+ 1446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:53.349894 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:53.349902 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:53.350147 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:53.350645 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:53.350649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:53.350651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:53.351372 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:53.351733 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:53.352373 (+ 640us) server_negotiation.cc:299] Negotiation successful
0926 23:35:53.352785 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":443,"threads_started":1}
I20260926 23:35:53.355559 20930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1ac9af8e8f74b1d847a5ca22126173e: No bootstrap required, opened a new log
I20260926 23:35:53.359750 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:35:53.368194 20930 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1ac9af8e8f74b1d847a5ca22126173e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1ac9af8e8f74b1d847a5ca22126173e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42067 } }
I20260926 23:35:53.368976 20930 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1ac9af8e8f74b1d847a5ca22126173e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:35:53.369125 20930 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1ac9af8e8f74b1d847a5ca22126173e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1ac9af8e8f74b1d847a5ca22126173e, State: Initialized, Role: FOLLOWER
I20260926 23:35:53.370174 20930 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1ac9af8e8f74b1d847a5ca22126173e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1ac9af8e8f74b1d847a5ca22126173e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42067 } }
I20260926 23:35:53.370564 20930 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1ac9af8e8f74b1d847a5ca22126173e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:35:53.370790 20930 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1ac9af8e8f74b1d847a5ca22126173e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:35:53.371006 20930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1ac9af8e8f74b1d847a5ca22126173e [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:35:53.374773 20930 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1ac9af8e8f74b1d847a5ca22126173e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1ac9af8e8f74b1d847a5ca22126173e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42067 } }
I20260926 23:35:53.375545 20930 leader_election.cc:302] T 00000000000000000000000000000000 P a1ac9af8e8f74b1d847a5ca22126173e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1ac9af8e8f74b1d847a5ca22126173e; no voters:
I20260926 23:35:53.376330 20930 leader_election.cc:288] T 00000000000000000000000000000000 P a1ac9af8e8f74b1d847a5ca22126173e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:35:53.376485 20940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1ac9af8e8f74b1d847a5ca22126173e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:35:53.377784 20940 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1ac9af8e8f74b1d847a5ca22126173e [term 1 LEADER]: Becoming Leader. State: Replica: a1ac9af8e8f74b1d847a5ca22126173e, State: Running, Role: LEADER
I20260926 23:35:53.378643 20940 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1ac9af8e8f74b1d847a5ca22126173e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1ac9af8e8f74b1d847a5ca22126173e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42067 } }
I20260926 23:35:53.379616 20930 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1ac9af8e8f74b1d847a5ca22126173e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:35:53.382308 20941 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1ac9af8e8f74b1d847a5ca22126173e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1ac9af8e8f74b1d847a5ca22126173e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1ac9af8e8f74b1d847a5ca22126173e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42067 } } }
I20260926 23:35:53.382869 20941 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1ac9af8e8f74b1d847a5ca22126173e [sys.catalog]: This master's current role is: LEADER
I20260926 23:35:53.383589 20942 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1ac9af8e8f74b1d847a5ca22126173e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1ac9af8e8f74b1d847a5ca22126173e. Latest consensus state: current_term: 1 leader_uuid: "a1ac9af8e8f74b1d847a5ca22126173e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1ac9af8e8f74b1d847a5ca22126173e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42067 } } }
I20260926 23:35:53.383978 20942 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1ac9af8e8f74b1d847a5ca22126173e [sys.catalog]: This master's current role is: LEADER
I20260926 23:35:53.384447 20946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:35:53.386816 20933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:53.366953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54181 (local address 127.0.0.1:42067)
0926 23:35:53.367162 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:53.367168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:53.367725 (+ 557us) server_negotiation.cc:407] Connection header received
0926 23:35:53.367928 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:53.367933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:53.368030 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:53.368140 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:53.370478 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:53.371527 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:53.373545 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:53.374060 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:53.374168 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:53.377439 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:53.377449 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:53.377454 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:35:53.377522 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:53.381315 (+ 3793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:53.382748 (+ 1433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:53.382754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:53.382757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:53.382970 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:53.383637 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:53.383641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:53.383645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:53.384119 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:53.385810 (+ 1691us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:53.386043 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 23:35:53.386301 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 23:35:53.396463 20946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:35:53.409704 20946 catalog_manager.cc:1347] Generated new cluster ID: 3e1a5922c1964de1b644928641dc5569
I20260926 23:35:53.409945 20946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:35:53.461395 20946 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:35:53.463433 20946 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:35:53.471105 20946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1ac9af8e8f74b1d847a5ca22126173e: Generated new TSK 0
I20260926 23:35:53.472361 20946 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:35:53.480978 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 20863
I20260926 23:35:53.482774 20925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:35:53.590029 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 20863
W20260926 23:35:53.608575 20848 connection.cc:535] client connection to 127.0.0.1:42067 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42067): Connection reset by peer (error 104)
2026-09-26T23:35:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2220 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:35:53 dist-test-slave-gwfx krb5kdc[20962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:35:53 dist-test-slave-gwfx krb5kdc[20962](info): set up 2 sockets
Sep 26 23:35:53 dist-test-slave-gwfx krb5kdc[20962](info): commencing operation
krb5kdc: starting...
W20260926 23:35:54.647346 388 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:35:54 dist-test-slave-gwfx krb5kdc[20962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:35:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:35:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:35:54.959532 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36971
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:35:55.370301 20975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:35:55.370757 20975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:35:55.370916 20975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:35:55.379407 20975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:35:55.379590 20975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:35:55.379676 20975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:35:55.379773 20975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:35:55.379837 20975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:35:55.391419 20975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36971
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:35:55.393214 20975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:35:55.394961 20975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:35:55.409340 20981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:55.409322 20980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:55.409390 20983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:35:55.410158 20975 server_base.cc:1034] running on GCE node
I20260926 23:35:55.411289 20975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:35:55.413270 20975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:35:55.414557 20975 hybrid_clock.cc:648] HybridClock initialized: now 1790465755414514 us; error 59 us; skew 500 ppm
I20260926 23:35:55.415112 20975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:35:55 dist-test-slave-gwfx krb5kdc[20962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:35:55.419080 20975 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:35:55.421195 20975 webserver.cc:460] Webserver started at http://127.0.0.1:39041/ using document root <none> and password file <none>
I20260926 23:35:55.422160 20975 fs_manager.cc:359] Metadata directory not provided
I20260926 23:35:55.422427 20975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:35:55.422923 20975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:35:55.427112 20975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "43f993ef16c543efa853e421a576b4e2"
format_stamp: "Formatted at 2026-09-26 23:35:55 on dist-test-slave-gwfx"
server_key: "cd86a9938a8d6ced4b475b92ac8c652e"
server_key_iv: "4d021c711160a99f7f97903dbfd6a7eb"
server_key_version: "encryptionkey@0"
I20260926 23:35:55.428155 20975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "43f993ef16c543efa853e421a576b4e2"
format_stamp: "Formatted at 2026-09-26 23:35:55 on dist-test-slave-gwfx"
server_key: "cd86a9938a8d6ced4b475b92ac8c652e"
server_key_iv: "4d021c711160a99f7f97903dbfd6a7eb"
server_key_version: "encryptionkey@0"
I20260926 23:35:55.435374 20975 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260926 23:35:55.440475 20990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:55.442432 20975 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 23:35:55.442756 20975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "43f993ef16c543efa853e421a576b4e2"
format_stamp: "Formatted at 2026-09-26 23:35:55 on dist-test-slave-gwfx"
server_key: "cd86a9938a8d6ced4b475b92ac8c652e"
server_key_iv: "4d021c711160a99f7f97903dbfd6a7eb"
server_key_version: "encryptionkey@0"
I20260926 23:35:55.443027 20975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:35:55.456409 20975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:35:55.462239 20975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:35:55.462627 20975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:35:55.484203 20975 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36971
I20260926 23:35:55.484218 21041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36971 every 8 connection(s)
I20260926 23:35:55.485961 20975 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:35:55.490921 21042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:35:55.494059 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 20975
I20260926 23:35:55.494298 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:35:55.494930 388 external_mini_cluster.cc:1428] Setting key e7ac83b9a0a746c7616d71b886a64f04
I20260926 23:35:55.506373 21042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43f993ef16c543efa853e421a576b4e2: Bootstrap starting.
I20260926 23:35:55.510625 21042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43f993ef16c543efa853e421a576b4e2: Neither blocks nor log segments found. Creating new log.
I20260926 23:35:55.511718 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:55.497539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53825 (local address 127.0.0.1:36971)
0926 23:35:55.498634 (+ 1095us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:55.498655 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:55.498758 (+ 103us) server_negotiation.cc:407] Connection header received
0926 23:35:55.500855 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:55.500890 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:55.501598 (+ 708us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:55.502136 (+ 538us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:55.502928 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:55.504529 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:55.506060 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:55.506979 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:55.507195 (+ 216us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:55.508777 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:55.508787 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:55.508806 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:35:55.509490 (+ 684us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:55.509601 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:35:55.509631 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:55.509786 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:55.510094 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 23:35:55.510300 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":812,"thread_start_us":479,"threads_started":1}
I20260926 23:35:55.512075 21042 log.cc:824] T 00000000000000000000000000000000 P 43f993ef16c543efa853e421a576b4e2: Log is configured to *not* fsync() on all Append() calls
I20260926 23:35:55.515705 21042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43f993ef16c543efa853e421a576b4e2: No bootstrap required, opened a new log
I20260926 23:35:55.516510 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:35:55.525614 21042 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43f993ef16c543efa853e421a576b4e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43f993ef16c543efa853e421a576b4e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36971 } }
I20260926 23:35:55.526356 21042 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43f993ef16c543efa853e421a576b4e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:35:55.526486 21042 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43f993ef16c543efa853e421a576b4e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43f993ef16c543efa853e421a576b4e2, State: Initialized, Role: FOLLOWER
I20260926 23:35:55.527593 21042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43f993ef16c543efa853e421a576b4e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43f993ef16c543efa853e421a576b4e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36971 } }
I20260926 23:35:55.527951 21042 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43f993ef16c543efa853e421a576b4e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:35:55.528141 21042 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43f993ef16c543efa853e421a576b4e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:35:55.528378 21042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43f993ef16c543efa853e421a576b4e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:35:55.530398 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:55.521933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58563 (local address 127.0.0.1:36971)
0926 23:35:55.522201 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:55.522210 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:55.522502 (+ 292us) server_negotiation.cc:407] Connection header received
0926 23:35:55.522860 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:55.522866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:55.522964 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:55.523064 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:55.523881 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:55.524837 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:55.526974 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:55.527699 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:55.527834 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:55.529372 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:55.529378 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:55.529382 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:35:55.529412 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:55.529489 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:35:55.529512 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:55.529709 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:55.529930 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 23:35:55.530015 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260926 23:35:55.531807 21042 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43f993ef16c543efa853e421a576b4e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43f993ef16c543efa853e421a576b4e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36971 } }
I20260926 23:35:55.532508 21042 leader_election.cc:302] T 00000000000000000000000000000000 P 43f993ef16c543efa853e421a576b4e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43f993ef16c543efa853e421a576b4e2; no voters:
I20260926 23:35:55.533156 21042 leader_election.cc:288] T 00000000000000000000000000000000 P 43f993ef16c543efa853e421a576b4e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:35:55.533306 21052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43f993ef16c543efa853e421a576b4e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:35:55.534021 21052 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43f993ef16c543efa853e421a576b4e2 [term 1 LEADER]: Becoming Leader. State: Replica: 43f993ef16c543efa853e421a576b4e2, State: Running, Role: LEADER
I20260926 23:35:55.534731 21052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43f993ef16c543efa853e421a576b4e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43f993ef16c543efa853e421a576b4e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36971 } }
I20260926 23:35:55.535619 21042 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43f993ef16c543efa853e421a576b4e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:35:55.537858 21054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43f993ef16c543efa853e421a576b4e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43f993ef16c543efa853e421a576b4e2. Latest consensus state: current_term: 1 leader_uuid: "43f993ef16c543efa853e421a576b4e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43f993ef16c543efa853e421a576b4e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36971 } } }
I20260926 23:35:55.538331 21054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43f993ef16c543efa853e421a576b4e2 [sys.catalog]: This master's current role is: LEADER
I20260926 23:35:55.539443 21057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:35:55.539222 21053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43f993ef16c543efa853e421a576b4e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43f993ef16c543efa853e421a576b4e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43f993ef16c543efa853e421a576b4e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36971 } } }
I20260926 23:35:55.539678 21053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43f993ef16c543efa853e421a576b4e2 [sys.catalog]: This master's current role is: LEADER
I20260926 23:35:55.548554 21057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:35:55.561317 21057 catalog_manager.cc:1347] Generated new cluster ID: 1b5cff8ef3614d9f95796dfdf337ebe4
I20260926 23:35:55.561493 21057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:35:55.573089 21057 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:35:55.575591 21057 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:35:55.587514 21057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43f993ef16c543efa853e421a576b4e2: Generated new TSK 0
I20260926 23:35:55.589063 21057 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:35:55.608171 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 20975
W20260926 23:35:55.609791 21037 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53825
W20260926 23:35:55.610791 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 20975: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:35:55.611032 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 20975
W20260926 23:35:55.629262 20960 connection.cc:535] client connection to 127.0.0.1:36971 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36971): Connection reset by peer (error 104)
2026-09-26T23:35:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2020 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:35:55 dist-test-slave-gwfx krb5kdc[21073](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:35:55 dist-test-slave-gwfx krb5kdc[21073](info): set up 2 sockets
Sep 26 23:35:55 dist-test-slave-gwfx krb5kdc[21073](info): commencing operation
krb5kdc: starting...
W20260926 23:35:56.683272 388 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.003s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:35:56 dist-test-slave-gwfx krb5kdc[21073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:35:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:35:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:35:56.999104 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40551
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:35:57.413836 21086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:35:57.414283 21086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:35:57.414435 21086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:35:57.423032 21086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:35:57.423206 21086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:35:57.423295 21086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:35:57.423382 21086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:35:57.423444 21086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:35:57.435266 21086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40551
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:35:57.437070 21086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:35:57.438791 21086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:35:57.453806 21094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:57.453823 21091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:57.453792 21092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:35:57.454931 21086 server_base.cc:1034] running on GCE node
I20260926 23:35:57.455991 21086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:35:57.457964 21086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:35:57.459231 21086 hybrid_clock.cc:648] HybridClock initialized: now 1790465757459178 us; error 49 us; skew 500 ppm
I20260926 23:35:57.459681 21086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:35:57 dist-test-slave-gwfx krb5kdc[21073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:35:57.469635 21086 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:35:57.472083 21086 webserver.cc:460] Webserver started at http://127.0.0.1:44707/ using document root <none> and password file <none>
I20260926 23:35:57.473078 21086 fs_manager.cc:359] Metadata directory not provided
I20260926 23:35:57.473238 21086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:35:57.473634 21086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:35:57.477648 21086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "e8edf4c0b4b74831a8ac9fea94de60eb"
format_stamp: "Formatted at 2026-09-26 23:35:57 on dist-test-slave-gwfx"
server_key: "47319490f23e36a07b05b56d87d17d85"
server_key_iv: "6c2abd5db5dcf2c53141e55ef0652826"
server_key_version: "encryptionkey@0"
I20260926 23:35:57.478675 21086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "e8edf4c0b4b74831a8ac9fea94de60eb"
format_stamp: "Formatted at 2026-09-26 23:35:57 on dist-test-slave-gwfx"
server_key: "47319490f23e36a07b05b56d87d17d85"
server_key_iv: "6c2abd5db5dcf2c53141e55ef0652826"
server_key_version: "encryptionkey@0"
I20260926 23:35:57.485607 21086 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 23:35:57.490206 21101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:57.492296 21086 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:35:57.492568 21086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "e8edf4c0b4b74831a8ac9fea94de60eb"
format_stamp: "Formatted at 2026-09-26 23:35:57 on dist-test-slave-gwfx"
server_key: "47319490f23e36a07b05b56d87d17d85"
server_key_iv: "6c2abd5db5dcf2c53141e55ef0652826"
server_key_version: "encryptionkey@0"
I20260926 23:35:57.492856 21086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:35:57.500644 21086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:35:57.506765 21086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:35:57.507141 21086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:35:57.527313 21086 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40551
I20260926 23:35:57.527351 21152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40551 every 8 connection(s)
I20260926 23:35:57.529102 21086 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:35:57.533268 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 21086
I20260926 23:35:57.533486 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:35:57.533859 21153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:35:57.534063 388 external_mini_cluster.cc:1428] Setting key 6d1bbebad8141c8a512f9f47adfb57af
I20260926 23:35:57.543361 21156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:57.536249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36731 (local address 127.0.0.1:40551)
0926 23:35:57.537136 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:57.537152 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:57.537203 (+ 51us) server_negotiation.cc:407] Connection header received
0926 23:35:57.539323 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:57.539352 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:57.539819 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:57.540303 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:57.541416 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:57.541423 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:57.541442 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:35:57.541464 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:57.541530 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:35:57.541567 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:57.541815 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:57.541956 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 23:35:57.542169 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":378,"threads_started":1}
I20260926 23:35:57.548276 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:35:57.549606 21153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8edf4c0b4b74831a8ac9fea94de60eb: Bootstrap starting.
I20260926 23:35:57.553741 21153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8edf4c0b4b74831a8ac9fea94de60eb: Neither blocks nor log segments found. Creating new log.
I20260926 23:35:57.555171 21153 log.cc:824] T 00000000000000000000000000000000 P e8edf4c0b4b74831a8ac9fea94de60eb: Log is configured to *not* fsync() on all Append() calls
I20260926 23:35:57.556833 21156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:57.553737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49513 (local address 127.0.0.1:40551)
0926 23:35:57.553990 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:57.553999 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:57.554534 (+ 535us) server_negotiation.cc:407] Connection header received
0926 23:35:57.554779 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:57.554785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:57.554860 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:57.554966 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:57.556069 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:57.556076 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:57.556081 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:35:57.556104 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:57.556169 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:35:57.556198 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:57.556335 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:57.556409 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 23:35:57.556488 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 23:35:57.559325 21153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8edf4c0b4b74831a8ac9fea94de60eb: No bootstrap required, opened a new log
I20260926 23:35:57.569180 21153 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8edf4c0b4b74831a8ac9fea94de60eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8edf4c0b4b74831a8ac9fea94de60eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40551 } }
I20260926 23:35:57.569824 21153 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8edf4c0b4b74831a8ac9fea94de60eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:35:57.569944 21153 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8edf4c0b4b74831a8ac9fea94de60eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8edf4c0b4b74831a8ac9fea94de60eb, State: Initialized, Role: FOLLOWER
I20260926 23:35:57.570802 21153 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8edf4c0b4b74831a8ac9fea94de60eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8edf4c0b4b74831a8ac9fea94de60eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40551 } }
I20260926 23:35:57.571127 21153 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8edf4c0b4b74831a8ac9fea94de60eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:35:57.571291 21153 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8edf4c0b4b74831a8ac9fea94de60eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:35:57.571487 21153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8edf4c0b4b74831a8ac9fea94de60eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:35:57.574632 21153 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8edf4c0b4b74831a8ac9fea94de60eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8edf4c0b4b74831a8ac9fea94de60eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40551 } }
I20260926 23:35:57.575397 21153 leader_election.cc:302] T 00000000000000000000000000000000 P e8edf4c0b4b74831a8ac9fea94de60eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8edf4c0b4b74831a8ac9fea94de60eb; no voters:
I20260926 23:35:57.576108 21153 leader_election.cc:288] T 00000000000000000000000000000000 P e8edf4c0b4b74831a8ac9fea94de60eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:35:57.576303 21163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8edf4c0b4b74831a8ac9fea94de60eb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:35:57.577031 21163 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8edf4c0b4b74831a8ac9fea94de60eb [term 1 LEADER]: Becoming Leader. State: Replica: e8edf4c0b4b74831a8ac9fea94de60eb, State: Running, Role: LEADER
I20260926 23:35:57.577668 21163 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8edf4c0b4b74831a8ac9fea94de60eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8edf4c0b4b74831a8ac9fea94de60eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40551 } }
I20260926 23:35:57.578732 21153 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8edf4c0b4b74831a8ac9fea94de60eb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:35:57.580991 21165 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8edf4c0b4b74831a8ac9fea94de60eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8edf4c0b4b74831a8ac9fea94de60eb. Latest consensus state: current_term: 1 leader_uuid: "e8edf4c0b4b74831a8ac9fea94de60eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8edf4c0b4b74831a8ac9fea94de60eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40551 } } }
I20260926 23:35:57.581517 21165 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8edf4c0b4b74831a8ac9fea94de60eb [sys.catalog]: This master's current role is: LEADER
I20260926 23:35:57.584234 21164 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8edf4c0b4b74831a8ac9fea94de60eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8edf4c0b4b74831a8ac9fea94de60eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8edf4c0b4b74831a8ac9fea94de60eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40551 } } }
I20260926 23:35:57.584872 21164 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8edf4c0b4b74831a8ac9fea94de60eb [sys.catalog]: This master's current role is: LEADER
I20260926 23:35:57.585126 21171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:35:57.590207 21171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:35:57.603334 21171 catalog_manager.cc:1347] Generated new cluster ID: 739a7865366c4488b6d4add18afb1577
I20260926 23:35:57.603493 21171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:35:57.620296 21171 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:35:57.621017 21171 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:35:57.634732 21171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8edf4c0b4b74831a8ac9fea94de60eb: Generated new TSK 0
I20260926 23:35:57.636204 21171 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:35:57.652791 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 21086
W20260926 23:35:57.654805 21148 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36731
W20260926 23:35:57.655658 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 21086: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:35:57.655856 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 21086
2026-09-26T23:35:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2044 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:35:57 dist-test-slave-gwfx krb5kdc[21184](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:35:57 dist-test-slave-gwfx krb5kdc[21184](info): set up 2 sockets
Sep 26 23:35:57 dist-test-slave-gwfx krb5kdc[21184](info): commencing operation
krb5kdc: starting...
W20260926 23:35:58.727322 388 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.002s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:35:58 dist-test-slave-gwfx krb5kdc[21184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:35:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:35:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:35:59.035596 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36509
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:35:59.480312 21197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:35:59.480737 21197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:35:59.480845 21197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:35:59.489374 21197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:35:59.489534 21197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:35:59.489593 21197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:35:59.489665 21197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:35:59.489714 21197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:35:59.501833 21197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:35:59.503661 21197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:35:59.505686 21197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:35:59.522257 21202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:59.522411 21205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:35:59.522400 21203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:35:59.522816 21197 server_base.cc:1034] running on GCE node
I20260926 23:35:59.524235 21197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:35:59.526548 21197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:35:59.528012 21197 hybrid_clock.cc:648] HybridClock initialized: now 1790465759527975 us; error 130 us; skew 500 ppm
I20260926 23:35:59.528514 21197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:35:59 dist-test-slave-gwfx krb5kdc[21184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:35:59.532734 21197 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:35:59.534901 21197 webserver.cc:460] Webserver started at http://127.0.0.1:41807/ using document root <none> and password file <none>
I20260926 23:35:59.535732 21197 fs_manager.cc:359] Metadata directory not provided
I20260926 23:35:59.535854 21197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:35:59.536198 21197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:35:59.539960 21197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "da0138fa2bd2431e93bc8f94bdbb9361"
format_stamp: "Formatted at 2026-09-26 23:35:59 on dist-test-slave-gwfx"
server_key: "2b75412de9c80d812c7e05f5c89bda41"
server_key_iv: "37717ee8fa72ef2cecdacc80ab74896e"
server_key_version: "encryptionkey@0"
I20260926 23:35:59.540970 21197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "da0138fa2bd2431e93bc8f94bdbb9361"
format_stamp: "Formatted at 2026-09-26 23:35:59 on dist-test-slave-gwfx"
server_key: "2b75412de9c80d812c7e05f5c89bda41"
server_key_iv: "37717ee8fa72ef2cecdacc80ab74896e"
server_key_version: "encryptionkey@0"
I20260926 23:35:59.548210 21197 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260926 23:35:59.553596 21212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:35:59.556041 21197 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260926 23:35:59.556542 21197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "da0138fa2bd2431e93bc8f94bdbb9361"
format_stamp: "Formatted at 2026-09-26 23:35:59 on dist-test-slave-gwfx"
server_key: "2b75412de9c80d812c7e05f5c89bda41"
server_key_iv: "37717ee8fa72ef2cecdacc80ab74896e"
server_key_version: "encryptionkey@0"
I20260926 23:35:59.556856 21197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:35:59.572495 21197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:35:59.577875 21197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:35:59.578207 21197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:35:59.599021 21197 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36509
I20260926 23:35:59.599074 21263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36509 every 8 connection(s)
I20260926 23:35:59.600632 21197 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:35:59.602321 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 21197
I20260926 23:35:59.602485 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:35:59.603024 388 external_mini_cluster.cc:1428] Setting key 015f6b07c3e227ab06542fdfe2b1f06b
I20260926 23:35:59.606472 21264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 23:35:59 dist-test-slave-gwfx krb5kdc[21184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 23:35:59.622054 21264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da0138fa2bd2431e93bc8f94bdbb9361: Bootstrap starting.
I20260926 23:35:59.626554 21264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da0138fa2bd2431e93bc8f94bdbb9361: Neither blocks nor log segments found. Creating new log.
I20260926 23:35:59.628197 21264 log.cc:824] T 00000000000000000000000000000000 P da0138fa2bd2431e93bc8f94bdbb9361: Log is configured to *not* fsync() on all Append() calls
I20260926 23:35:59.632407 21264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da0138fa2bd2431e93bc8f94bdbb9361: No bootstrap required, opened a new log
I20260926 23:35:59.632916 21267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:59.605373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52947 (local address 127.0.0.1:36509)
0926 23:35:59.606378 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:59.606399 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:59.606473 (+ 74us) server_negotiation.cc:407] Connection header received
0926 23:35:59.609080 (+ 2607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:59.609117 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:59.609778 (+ 661us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:59.610365 (+ 587us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:59.612430 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:59.614650 (+ 2220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:59.616563 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:59.617715 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:59.617991 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:59.623564 (+ 5573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:59.623576 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:59.623598 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:35:59.623664 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:59.627216 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:59.628592 (+ 1376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:59.628601 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:59.628610 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:59.628851 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:59.629445 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:59.629449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:59.629451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:59.630159 (+ 708us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:59.630529 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:59.631112 (+ 583us) server_negotiation.cc:299] Negotiation successful
0926 23:35:59.631635 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":413,"threads_started":1}
I20260926 23:35:59.638422 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:35:59.642403 21264 raft_consensus.cc:348] T 00000000000000000000000000000000 P da0138fa2bd2431e93bc8f94bdbb9361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0138fa2bd2431e93bc8f94bdbb9361" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36509 } }
I20260926 23:35:59.643021 21264 raft_consensus.cc:374] T 00000000000000000000000000000000 P da0138fa2bd2431e93bc8f94bdbb9361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:35:59.643146 21264 raft_consensus.cc:729] T 00000000000000000000000000000000 P da0138fa2bd2431e93bc8f94bdbb9361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da0138fa2bd2431e93bc8f94bdbb9361, State: Initialized, Role: FOLLOWER
I20260926 23:35:59.644057 21264 consensus_queue.cc:260] T 00000000000000000000000000000000 P da0138fa2bd2431e93bc8f94bdbb9361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0138fa2bd2431e93bc8f94bdbb9361" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36509 } }
I20260926 23:35:59.644408 21264 raft_consensus.cc:388] T 00000000000000000000000000000000 P da0138fa2bd2431e93bc8f94bdbb9361 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:35:59.644591 21264 raft_consensus.cc:482] T 00000000000000000000000000000000 P da0138fa2bd2431e93bc8f94bdbb9361 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:35:59.644805 21264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da0138fa2bd2431e93bc8f94bdbb9361 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:35:59.647993 21264 raft_consensus.cc:504] T 00000000000000000000000000000000 P da0138fa2bd2431e93bc8f94bdbb9361 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0138fa2bd2431e93bc8f94bdbb9361" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36509 } }
I20260926 23:35:59.648671 21264 leader_election.cc:302] T 00000000000000000000000000000000 P da0138fa2bd2431e93bc8f94bdbb9361 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da0138fa2bd2431e93bc8f94bdbb9361; no voters:
I20260926 23:35:59.649514 21264 leader_election.cc:288] T 00000000000000000000000000000000 P da0138fa2bd2431e93bc8f94bdbb9361 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:35:59.649736 21274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da0138fa2bd2431e93bc8f94bdbb9361 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:35:59.651028 21274 raft_consensus.cc:686] T 00000000000000000000000000000000 P da0138fa2bd2431e93bc8f94bdbb9361 [term 1 LEADER]: Becoming Leader. State: Replica: da0138fa2bd2431e93bc8f94bdbb9361, State: Running, Role: LEADER
I20260926 23:35:59.651916 21274 consensus_queue.cc:237] T 00000000000000000000000000000000 P da0138fa2bd2431e93bc8f94bdbb9361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0138fa2bd2431e93bc8f94bdbb9361" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36509 } }
I20260926 23:35:59.652693 21264 sys_catalog.cc:564] T 00000000000000000000000000000000 P da0138fa2bd2431e93bc8f94bdbb9361 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:35:59.655539 21275 sys_catalog.cc:455] T 00000000000000000000000000000000 P da0138fa2bd2431e93bc8f94bdbb9361 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da0138fa2bd2431e93bc8f94bdbb9361" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0138fa2bd2431e93bc8f94bdbb9361" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36509 } } }
I20260926 23:35:59.655786 21276 sys_catalog.cc:455] T 00000000000000000000000000000000 P da0138fa2bd2431e93bc8f94bdbb9361 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da0138fa2bd2431e93bc8f94bdbb9361. Latest consensus state: current_term: 1 leader_uuid: "da0138fa2bd2431e93bc8f94bdbb9361" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0138fa2bd2431e93bc8f94bdbb9361" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36509 } } }
I20260926 23:35:59.656126 21276 sys_catalog.cc:458] T 00000000000000000000000000000000 P da0138fa2bd2431e93bc8f94bdbb9361 [sys.catalog]: This master's current role is: LEADER
I20260926 23:35:59.656126 21275 sys_catalog.cc:458] T 00000000000000000000000000000000 P da0138fa2bd2431e93bc8f94bdbb9361 [sys.catalog]: This master's current role is: LEADER
I20260926 23:35:59.659294 21280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:35:59.665455 21267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:35:59.643632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51997 (local address 127.0.0.1:36509)
0926 23:35:59.643833 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 23:35:59.643839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:35:59.644535 (+ 696us) server_negotiation.cc:407] Connection header received
0926 23:35:59.644643 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:35:59.644650 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:35:59.644766 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:35:59.644909 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:35:59.646474 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:59.647664 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:59.649218 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:35:59.650144 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:35:59.651237 (+ 1093us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:35:59.654141 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:35:59.654148 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:35:59.654154 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:35:59.654211 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:35:59.659655 (+ 5444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:59.660835 (+ 1180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:59.660840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:59.660844 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:59.661063 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:35:59.661593 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:35:59.661597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:35:59.661600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:35:59.662059 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:35:59.662400 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0926 23:35:59.664439 (+ 2039us) server_negotiation.cc:299] Negotiation successful
0926 23:35:59.664838 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 23:35:59.667090 21280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:35:59.680186 21280 catalog_manager.cc:1347] Generated new cluster ID: f7c85ca7a8614aa9bb495972ebb02214
I20260926 23:35:59.680333 21280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:35:59.697893 21280 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:35:59.700040 21280 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:35:59.710441 21280 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da0138fa2bd2431e93bc8f94bdbb9361: Generated new TSK 0
I20260926 23:35:59.711807 21280 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:35:59.719949 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 21197
I20260926 23:35:59.721683 21259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:35:59.833271 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 21197
2026-09-26T23:35:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2179 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:35:59 dist-test-slave-gwfx krb5kdc[21296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:35:59 dist-test-slave-gwfx krb5kdc[21296](info): set up 2 sockets
Sep 26 23:35:59 dist-test-slave-gwfx krb5kdc[21296](info): commencing operation
krb5kdc: starting...
W20260926 23:36:00.927331 388 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:36:01 dist-test-slave-gwfx krb5kdc[21296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:36:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:36:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:36:01.247236 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46481
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:36:01.690025 21309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:36:01.690493 21309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:36:01.690657 21309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:36:01.699833 21309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:36:01.700053 21309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:36:01.700145 21309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:36:01.700246 21309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:36:01.700316 21309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:36:01.712301 21309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:36:01.714208 21309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:36:01.716086 21309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:36:01.732157 21314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:01.732110 21315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:01.732084 21317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:36:01.733992 21309 server_base.cc:1034] running on GCE node
I20260926 23:36:01.735188 21309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:36:01.737185 21309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:36:01.738464 21309 hybrid_clock.cc:648] HybridClock initialized: now 1790465761738364 us; error 96 us; skew 500 ppm
I20260926 23:36:01.739037 21309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:36:01 dist-test-slave-gwfx krb5kdc[21296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:36:01.743381 21309 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:36:01.745349 21309 webserver.cc:460] Webserver started at http://127.0.0.1:38377/ using document root <none> and password file <none>
I20260926 23:36:01.746240 21309 fs_manager.cc:359] Metadata directory not provided
I20260926 23:36:01.746378 21309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:36:01.746790 21309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:36:01.750617 21309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "201b23c1377f4ed7bd3cb2996da5c3d5"
format_stamp: "Formatted at 2026-09-26 23:36:01 on dist-test-slave-gwfx"
server_key: "330fb8186b9b4f96fbdf578d9b91fc01"
server_key_iv: "14ad7b1727a29fabbd94d6e960514a22"
server_key_version: "encryptionkey@0"
I20260926 23:36:01.751693 21309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "201b23c1377f4ed7bd3cb2996da5c3d5"
format_stamp: "Formatted at 2026-09-26 23:36:01 on dist-test-slave-gwfx"
server_key: "330fb8186b9b4f96fbdf578d9b91fc01"
server_key_iv: "14ad7b1727a29fabbd94d6e960514a22"
server_key_version: "encryptionkey@0"
I20260926 23:36:01.758750 21309 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.006s
I20260926 23:36:01.763635 21324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:36:01.765448 21309 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260926 23:36:01.765681 21309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "201b23c1377f4ed7bd3cb2996da5c3d5"
format_stamp: "Formatted at 2026-09-26 23:36:01 on dist-test-slave-gwfx"
server_key: "330fb8186b9b4f96fbdf578d9b91fc01"
server_key_iv: "14ad7b1727a29fabbd94d6e960514a22"
server_key_version: "encryptionkey@0"
I20260926 23:36:01.765936 21309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:36:01.787895 21309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:36:01.793182 21309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:36:01.793562 21309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:36:01.815837 21309 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46481
I20260926 23:36:01.815860 21375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46481 every 8 connection(s)
I20260926 23:36:01.817703 21309 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:36:01.824189 21376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:36:01.824611 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 21309
I20260926 23:36:01.824846 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:36:01.825554 388 external_mini_cluster.cc:1428] Setting key 1925923241b165bcd1f57da7b1bbd62b
I20260926 23:36:01.843386 21376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 201b23c1377f4ed7bd3cb2996da5c3d5: Bootstrap starting.
I20260926 23:36:01.845695 21379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:01.828097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33683 (local address 127.0.0.1:46481)
0926 23:36:01.829392 (+ 1295us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:01.829410 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:01.829472 (+ 62us) server_negotiation.cc:407] Connection header received
0926 23:36:01.832648 (+ 3176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:01.832693 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:01.833496 (+ 803us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:01.834044 (+ 548us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:01.834915 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:01.836640 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:01.839249 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:01.840409 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:01.840638 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:01.842389 (+ 1751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:01.842401 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:01.842426 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:36:01.843328 (+ 902us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:01.843448 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:36:01.843485 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:01.843650 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:01.843893 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 23:36:01.844257 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":987,"thread_start_us":665,"threads_started":1}
I20260926 23:36:01.847764 21376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 201b23c1377f4ed7bd3cb2996da5c3d5: Neither blocks nor log segments found. Creating new log.
I20260926 23:36:01.849273 21376 log.cc:824] T 00000000000000000000000000000000 P 201b23c1377f4ed7bd3cb2996da5c3d5: Log is configured to *not* fsync() on all Append() calls
I20260926 23:36:01.851349 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:36:01.853703 21376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 201b23c1377f4ed7bd3cb2996da5c3d5: No bootstrap required, opened a new log
I20260926 23:36:01.864722 21379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:01.856807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42511 (local address 127.0.0.1:46481)
0926 23:36:01.857060 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:01.857067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:01.857484 (+ 417us) server_negotiation.cc:407] Connection header received
0926 23:36:01.857765 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:01.857770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:01.857857 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:01.857950 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:01.858781 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:01.859758 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:01.861952 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:01.862501 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:01.862626 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:01.863955 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:01.863960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:01.863964 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:36:01.863989 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:01.864036 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:36:01.864054 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:01.864129 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:01.864273 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 23:36:01.864368 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 23:36:01.864925 21376 raft_consensus.cc:348] T 00000000000000000000000000000000 P 201b23c1377f4ed7bd3cb2996da5c3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201b23c1377f4ed7bd3cb2996da5c3d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } }
I20260926 23:36:01.865538 21376 raft_consensus.cc:374] T 00000000000000000000000000000000 P 201b23c1377f4ed7bd3cb2996da5c3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:36:01.865657 21376 raft_consensus.cc:729] T 00000000000000000000000000000000 P 201b23c1377f4ed7bd3cb2996da5c3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201b23c1377f4ed7bd3cb2996da5c3d5, State: Initialized, Role: FOLLOWER
I20260926 23:36:01.866405 21376 consensus_queue.cc:260] T 00000000000000000000000000000000 P 201b23c1377f4ed7bd3cb2996da5c3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201b23c1377f4ed7bd3cb2996da5c3d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } }
I20260926 23:36:01.866768 21376 raft_consensus.cc:388] T 00000000000000000000000000000000 P 201b23c1377f4ed7bd3cb2996da5c3d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:36:01.866925 21376 raft_consensus.cc:482] T 00000000000000000000000000000000 P 201b23c1377f4ed7bd3cb2996da5c3d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:36:01.867116 21376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 201b23c1377f4ed7bd3cb2996da5c3d5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:36:01.869639 21376 raft_consensus.cc:504] T 00000000000000000000000000000000 P 201b23c1377f4ed7bd3cb2996da5c3d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201b23c1377f4ed7bd3cb2996da5c3d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } }
I20260926 23:36:01.870335 21376 leader_election.cc:302] T 00000000000000000000000000000000 P 201b23c1377f4ed7bd3cb2996da5c3d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 201b23c1377f4ed7bd3cb2996da5c3d5; no voters:
I20260926 23:36:01.871204 21376 leader_election.cc:288] T 00000000000000000000000000000000 P 201b23c1377f4ed7bd3cb2996da5c3d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:36:01.871441 21386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 201b23c1377f4ed7bd3cb2996da5c3d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:36:01.872134 21386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 201b23c1377f4ed7bd3cb2996da5c3d5 [term 1 LEADER]: Becoming Leader. State: Replica: 201b23c1377f4ed7bd3cb2996da5c3d5, State: Running, Role: LEADER
I20260926 23:36:01.872859 21386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 201b23c1377f4ed7bd3cb2996da5c3d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201b23c1377f4ed7bd3cb2996da5c3d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } }
I20260926 23:36:01.874001 21376 sys_catalog.cc:564] T 00000000000000000000000000000000 P 201b23c1377f4ed7bd3cb2996da5c3d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:36:01.876813 21388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 201b23c1377f4ed7bd3cb2996da5c3d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 201b23c1377f4ed7bd3cb2996da5c3d5. Latest consensus state: current_term: 1 leader_uuid: "201b23c1377f4ed7bd3cb2996da5c3d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201b23c1377f4ed7bd3cb2996da5c3d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } } }
I20260926 23:36:01.877331 21388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 201b23c1377f4ed7bd3cb2996da5c3d5 [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:01.878072 21387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 201b23c1377f4ed7bd3cb2996da5c3d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "201b23c1377f4ed7bd3cb2996da5c3d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201b23c1377f4ed7bd3cb2996da5c3d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } } }
I20260926 23:36:01.878485 21387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 201b23c1377f4ed7bd3cb2996da5c3d5 [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:01.882315 21392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:36:01.888867 21392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:36:01.901515 21392 catalog_manager.cc:1347] Generated new cluster ID: d1f41aa6140c4b09aee31a63948a3e39
I20260926 23:36:01.901666 21392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:36:01.914806 21392 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:36:01.916644 21392 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:36:01.925109 21392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 201b23c1377f4ed7bd3cb2996da5c3d5: Generated new TSK 0
I20260926 23:36:01.926565 21392 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:36:01.938651 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 21309
W20260926 23:36:01.940230 21371 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33683
W20260926 23:36:01.941133 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 21309: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:36:01.941326 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 21309
2026-09-26T23:36:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2107 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:36:02 dist-test-slave-gwfx krb5kdc[21407](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:36:02 dist-test-slave-gwfx krb5kdc[21407](info): set up 2 sockets
Sep 26 23:36:02 dist-test-slave-gwfx krb5kdc[21407](info): commencing operation
krb5kdc: starting...
W20260926 23:36:03.011832 388 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.003s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:36:03 dist-test-slave-gwfx krb5kdc[21407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:36:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:36:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:36:03.347596 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37569
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:36:03.774811 21420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:36:03.775257 21420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:36:03.775413 21420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:36:03.783934 21420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:36:03.784127 21420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:36:03.784214 21420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:36:03.784301 21420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:36:03.784368 21420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:36:03.795995 21420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:36:03.797768 21420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:36:03.799510 21420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:36:03.813848 21428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:03.813814 21426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:03.814383 21425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:36:03.815009 21420 server_base.cc:1034] running on GCE node
I20260926 23:36:03.816048 21420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:36:03.817914 21420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:36:03.819175 21420 hybrid_clock.cc:648] HybridClock initialized: now 1790465763819111 us; error 58 us; skew 500 ppm
I20260926 23:36:03.819706 21420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:36:03 dist-test-slave-gwfx krb5kdc[21407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:36:03.824138 21420 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:36:03.826233 21420 webserver.cc:460] Webserver started at http://127.0.0.1:40573/ using document root <none> and password file <none>
I20260926 23:36:03.827271 21420 fs_manager.cc:359] Metadata directory not provided
I20260926 23:36:03.827440 21420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:36:03.827848 21420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:36:03.832201 21420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "26e961c3320a4f6ebb1e1f057b41a7e3"
format_stamp: "Formatted at 2026-09-26 23:36:03 on dist-test-slave-gwfx"
server_key: "3724b2ac9e92385e80fe3ca4105110a4"
server_key_iv: "2b6a454ae7311fee44ff6db6a926a162"
server_key_version: "encryptionkey@0"
I20260926 23:36:03.833228 21420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "26e961c3320a4f6ebb1e1f057b41a7e3"
format_stamp: "Formatted at 2026-09-26 23:36:03 on dist-test-slave-gwfx"
server_key: "3724b2ac9e92385e80fe3ca4105110a4"
server_key_iv: "2b6a454ae7311fee44ff6db6a926a162"
server_key_version: "encryptionkey@0"
I20260926 23:36:03.840564 21420 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260926 23:36:03.845748 21435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:36:03.847759 21420 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:36:03.847982 21420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "26e961c3320a4f6ebb1e1f057b41a7e3"
format_stamp: "Formatted at 2026-09-26 23:36:03 on dist-test-slave-gwfx"
server_key: "3724b2ac9e92385e80fe3ca4105110a4"
server_key_iv: "2b6a454ae7311fee44ff6db6a926a162"
server_key_version: "encryptionkey@0"
I20260926 23:36:03.848243 21420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:36:03.855684 21420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:36:03.861065 21420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:36:03.861413 21420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:36:03.883136 21420 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37569
I20260926 23:36:03.883160 21486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37569 every 8 connection(s)
I20260926 23:36:03.885109 21420 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:36:03.890376 21487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:36:03.891431 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 21420
I20260926 23:36:03.891611 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:36:03.892143 388 external_mini_cluster.cc:1428] Setting key 1d0e9886b4b81274aad4168e3a7b3a8e
I20260926 23:36:03.901758 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:03.894405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40437 (local address 127.0.0.1:37569)
0926 23:36:03.895406 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:03.895423 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:03.895482 (+ 59us) server_negotiation.cc:407] Connection header received
0926 23:36:03.897638 (+ 2156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:03.897677 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:03.898210 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:03.898764 (+ 554us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:03.899935 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:03.899942 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:03.899957 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:36:03.899973 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:03.900045 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:36:03.900072 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:03.900302 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:03.900441 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 23:36:03.900670 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":467,"threads_started":1}
I20260926 23:36:03.906280 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:36:03.906261 21487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26e961c3320a4f6ebb1e1f057b41a7e3: Bootstrap starting.
I20260926 23:36:03.910297 21487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26e961c3320a4f6ebb1e1f057b41a7e3: Neither blocks nor log segments found. Creating new log.
I20260926 23:36:03.911988 21487 log.cc:824] T 00000000000000000000000000000000 P 26e961c3320a4f6ebb1e1f057b41a7e3: Log is configured to *not* fsync() on all Append() calls
I20260926 23:36:03.914000 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:03.911174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55477 (local address 127.0.0.1:37569)
0926 23:36:03.911396 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:03.911403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:03.911734 (+ 331us) server_negotiation.cc:407] Connection header received
0926 23:36:03.911929 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:03.911934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:03.912000 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:03.912091 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:03.913151 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:03.913156 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:03.913159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:36:03.913181 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:03.913233 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:36:03.913251 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:03.913372 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:03.913596 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 23:36:03.913684 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 23:36:03.915859 21487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26e961c3320a4f6ebb1e1f057b41a7e3: No bootstrap required, opened a new log
I20260926 23:36:03.925019 21487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26e961c3320a4f6ebb1e1f057b41a7e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26e961c3320a4f6ebb1e1f057b41a7e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37569 } }
I20260926 23:36:03.925596 21487 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26e961c3320a4f6ebb1e1f057b41a7e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:36:03.925711 21487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26e961c3320a4f6ebb1e1f057b41a7e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26e961c3320a4f6ebb1e1f057b41a7e3, State: Initialized, Role: FOLLOWER
I20260926 23:36:03.926447 21487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26e961c3320a4f6ebb1e1f057b41a7e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26e961c3320a4f6ebb1e1f057b41a7e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37569 } }
I20260926 23:36:03.926746 21487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26e961c3320a4f6ebb1e1f057b41a7e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:36:03.926903 21487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26e961c3320a4f6ebb1e1f057b41a7e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:36:03.927093 21487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26e961c3320a4f6ebb1e1f057b41a7e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:36:03.930025 21487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26e961c3320a4f6ebb1e1f057b41a7e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26e961c3320a4f6ebb1e1f057b41a7e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37569 } }
I20260926 23:36:03.930740 21487 leader_election.cc:302] T 00000000000000000000000000000000 P 26e961c3320a4f6ebb1e1f057b41a7e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26e961c3320a4f6ebb1e1f057b41a7e3; no voters:
I20260926 23:36:03.931468 21487 leader_election.cc:288] T 00000000000000000000000000000000 P 26e961c3320a4f6ebb1e1f057b41a7e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:36:03.931619 21497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26e961c3320a4f6ebb1e1f057b41a7e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:36:03.932873 21497 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26e961c3320a4f6ebb1e1f057b41a7e3 [term 1 LEADER]: Becoming Leader. State: Replica: 26e961c3320a4f6ebb1e1f057b41a7e3, State: Running, Role: LEADER
I20260926 23:36:03.933609 21497 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26e961c3320a4f6ebb1e1f057b41a7e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26e961c3320a4f6ebb1e1f057b41a7e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37569 } }
I20260926 23:36:03.934155 21487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26e961c3320a4f6ebb1e1f057b41a7e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:36:03.937057 21499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26e961c3320a4f6ebb1e1f057b41a7e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26e961c3320a4f6ebb1e1f057b41a7e3. Latest consensus state: current_term: 1 leader_uuid: "26e961c3320a4f6ebb1e1f057b41a7e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26e961c3320a4f6ebb1e1f057b41a7e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37569 } } }
I20260926 23:36:03.937649 21499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26e961c3320a4f6ebb1e1f057b41a7e3 [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:03.937119 21498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26e961c3320a4f6ebb1e1f057b41a7e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26e961c3320a4f6ebb1e1f057b41a7e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26e961c3320a4f6ebb1e1f057b41a7e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37569 } } }
I20260926 23:36:03.938318 21498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26e961c3320a4f6ebb1e1f057b41a7e3 [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:03.941301 21506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:36:03.946439 21506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:36:03.957007 21506 catalog_manager.cc:1347] Generated new cluster ID: 7e604c865fc44945b8f22f7fc12b5791
I20260926 23:36:03.957170 21506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:36:03.975937 21506 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:36:03.976516 21506 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:36:03.987185 21506 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26e961c3320a4f6ebb1e1f057b41a7e3: Generated new TSK 0
I20260926 23:36:03.988708 21506 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:36:04.006132 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 21420
W20260926 23:36:04.007947 21482 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40437
W20260926 23:36:04.008955 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 21420: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:36:04.009120 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 21420
2026-09-26T23:36:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2067 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:36:04 dist-test-slave-gwfx krb5kdc[21518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:36:04 dist-test-slave-gwfx krb5kdc[21518](info): set up 2 sockets
Sep 26 23:36:04 dist-test-slave-gwfx krb5kdc[21518](info): commencing operation
krb5kdc: starting...
W20260926 23:36:05.079531 388 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.003s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:36:05 dist-test-slave-gwfx krb5kdc[21518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:36:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:36:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:36:05.394737 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33447
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:36:05.802850 21531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:36:05.803325 21531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:36:05.803491 21531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:36:05.811760 21531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:36:05.811914 21531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:36:05.811987 21531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:36:05.812083 21531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:36:05.812145 21531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:36:05.823800 21531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:36:05.825759 21531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:36:05.827608 21531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:36:05.841840 21536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:05.841740 21537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:05.841799 21539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:36:05.842871 21531 server_base.cc:1034] running on GCE node
I20260926 23:36:05.843845 21531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:36:05.845824 21531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:36:05.847061 21531 hybrid_clock.cc:648] HybridClock initialized: now 1790465765847003 us; error 59 us; skew 500 ppm
I20260926 23:36:05.847486 21531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:36:05 dist-test-slave-gwfx krb5kdc[21518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:36:05.851220 21531 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:36:05.853300 21531 webserver.cc:460] Webserver started at http://127.0.0.1:38517/ using document root <none> and password file <none>
I20260926 23:36:05.854178 21531 fs_manager.cc:359] Metadata directory not provided
I20260926 23:36:05.854315 21531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:36:05.854678 21531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:36:05.858449 21531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "04d6f5f5586b4d7ea51c7c2d3f5c155d"
format_stamp: "Formatted at 2026-09-26 23:36:05 on dist-test-slave-gwfx"
server_key: "4a286345e5e22aa75c7228c73d8c686d"
server_key_iv: "793d2d8f2bfccce8cebfc2369f64ecd5"
server_key_version: "encryptionkey@0"
I20260926 23:36:05.859412 21531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "04d6f5f5586b4d7ea51c7c2d3f5c155d"
format_stamp: "Formatted at 2026-09-26 23:36:05 on dist-test-slave-gwfx"
server_key: "4a286345e5e22aa75c7228c73d8c686d"
server_key_iv: "793d2d8f2bfccce8cebfc2369f64ecd5"
server_key_version: "encryptionkey@0"
I20260926 23:36:05.866112 21531 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 23:36:05.870793 21546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:36:05.872593 21531 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:36:05.872804 21531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "04d6f5f5586b4d7ea51c7c2d3f5c155d"
format_stamp: "Formatted at 2026-09-26 23:36:05 on dist-test-slave-gwfx"
server_key: "4a286345e5e22aa75c7228c73d8c686d"
server_key_iv: "793d2d8f2bfccce8cebfc2369f64ecd5"
server_key_version: "encryptionkey@0"
I20260926 23:36:05.873041 21531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:36:05.899086 21531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:36:05.904155 21531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:36:05.904479 21531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:36:05.925297 21531 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33447
I20260926 23:36:05.925323 21597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33447 every 8 connection(s)
I20260926 23:36:05.927100 21531 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:36:05.928311 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 21531
I20260926 23:36:05.928510 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:36:05.928992 388 external_mini_cluster.cc:1428] Setting key 6002496fcfc8008d765802ed17a64247
I20260926 23:36:05.932556 21598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 23:36:05 dist-test-slave-gwfx krb5kdc[21518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 23:36:05.948177 21598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04d6f5f5586b4d7ea51c7c2d3f5c155d: Bootstrap starting.
I20260926 23:36:05.953258 21598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04d6f5f5586b4d7ea51c7c2d3f5c155d: Neither blocks nor log segments found. Creating new log.
I20260926 23:36:05.954674 21598 log.cc:824] T 00000000000000000000000000000000 P 04d6f5f5586b4d7ea51c7c2d3f5c155d: Log is configured to *not* fsync() on all Append() calls
I20260926 23:36:05.957904 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:05.931287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:43989 (local address 127.0.0.1:33447)
0926 23:36:05.932193 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:05.932211 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:05.932267 (+ 56us) server_negotiation.cc:407] Connection header received
0926 23:36:05.934834 (+ 2567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:05.934868 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:05.935501 (+ 633us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:05.936052 (+ 551us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:05.937640 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:05.939915 (+ 2275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:05.941679 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:05.942851 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:05.943079 (+ 228us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:05.947914 (+ 4835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:05.947925 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:05.947947 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:05.948009 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:05.952340 (+ 4331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:05.953678 (+ 1338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:05.953693 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:05.953703 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:05.953941 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:05.954493 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:05.954498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:05.954500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:05.955295 (+ 795us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:05.955721 (+ 426us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:05.956277 (+ 556us) server_negotiation.cc:299] Negotiation successful
0926 23:36:05.956668 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":351,"threads_started":1}
I20260926 23:36:05.958602 21598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04d6f5f5586b4d7ea51c7c2d3f5c155d: No bootstrap required, opened a new log
I20260926 23:36:05.963222 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:36:05.968212 21598 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04d6f5f5586b4d7ea51c7c2d3f5c155d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d6f5f5586b4d7ea51c7c2d3f5c155d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33447 } }
I20260926 23:36:05.968955 21598 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04d6f5f5586b4d7ea51c7c2d3f5c155d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:36:05.969113 21598 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04d6f5f5586b4d7ea51c7c2d3f5c155d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04d6f5f5586b4d7ea51c7c2d3f5c155d, State: Initialized, Role: FOLLOWER
I20260926 23:36:05.969903 21598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04d6f5f5586b4d7ea51c7c2d3f5c155d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d6f5f5586b4d7ea51c7c2d3f5c155d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33447 } }
I20260926 23:36:05.970191 21598 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04d6f5f5586b4d7ea51c7c2d3f5c155d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:36:05.970327 21598 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04d6f5f5586b4d7ea51c7c2d3f5c155d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:36:05.970499 21598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04d6f5f5586b4d7ea51c7c2d3f5c155d [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:36:05.973196 21598 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04d6f5f5586b4d7ea51c7c2d3f5c155d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d6f5f5586b4d7ea51c7c2d3f5c155d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33447 } }
I20260926 23:36:05.973856 21598 leader_election.cc:302] T 00000000000000000000000000000000 P 04d6f5f5586b4d7ea51c7c2d3f5c155d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04d6f5f5586b4d7ea51c7c2d3f5c155d; no voters:
I20260926 23:36:05.974597 21598 leader_election.cc:288] T 00000000000000000000000000000000 P 04d6f5f5586b4d7ea51c7c2d3f5c155d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:36:05.974757 21608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04d6f5f5586b4d7ea51c7c2d3f5c155d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:36:05.977533 21608 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04d6f5f5586b4d7ea51c7c2d3f5c155d [term 1 LEADER]: Becoming Leader. State: Replica: 04d6f5f5586b4d7ea51c7c2d3f5c155d, State: Running, Role: LEADER
I20260926 23:36:05.977901 21598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04d6f5f5586b4d7ea51c7c2d3f5c155d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:36:05.978415 21608 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04d6f5f5586b4d7ea51c7c2d3f5c155d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d6f5f5586b4d7ea51c7c2d3f5c155d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33447 } }
I20260926 23:36:05.985468 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:05.968520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:48271 (local address 127.0.0.1:33447)
0926 23:36:05.968773 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:05.968780 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:05.969141 (+ 361us) server_negotiation.cc:407] Connection header received
0926 23:36:05.969377 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:05.969382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:05.969476 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:05.969590 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:05.971361 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:05.972266 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:05.973603 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:05.974096 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:05.974219 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:05.977344 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:05.977352 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:05.977357 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:05.977416 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:05.981520 (+ 4104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:05.982738 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:05.982749 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:05.982753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:05.983029 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:05.983566 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:05.983572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:05.983576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:05.983972 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:05.984315 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:05.984718 (+ 403us) server_negotiation.cc:299] Negotiation successful
0926 23:36:05.984940 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260926 23:36:05.991776 21610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04d6f5f5586b4d7ea51c7c2d3f5c155d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04d6f5f5586b4d7ea51c7c2d3f5c155d. Latest consensus state: current_term: 1 leader_uuid: "04d6f5f5586b4d7ea51c7c2d3f5c155d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d6f5f5586b4d7ea51c7c2d3f5c155d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33447 } } }
I20260926 23:36:05.992327 21610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04d6f5f5586b4d7ea51c7c2d3f5c155d [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:05.992481 21609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04d6f5f5586b4d7ea51c7c2d3f5c155d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04d6f5f5586b4d7ea51c7c2d3f5c155d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d6f5f5586b4d7ea51c7c2d3f5c155d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33447 } } }
I20260926 23:36:05.992913 21609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04d6f5f5586b4d7ea51c7c2d3f5c155d [sys.catalog]: This master's current role is: LEADER
W20260926 23:36:06.000520 21624 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04d6f5f5586b4d7ea51c7c2d3f5c155d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 23:36:06.000736 21624 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 23:36:06.000912 21625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:36:06.003136 21625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:36:06.014978 21625 catalog_manager.cc:1347] Generated new cluster ID: 572961ffda7a4fd6aef70c9713079025
I20260926 23:36:06.015146 21625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:36:06.027292 21625 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:36:06.029615 21625 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:36:06.038318 21625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04d6f5f5586b4d7ea51c7c2d3f5c155d: Generated new TSK 0
I20260926 23:36:06.039836 21625 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:36:06.057062 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 21531
I20260926 23:36:06.059311 21593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:36:06.162232 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 21531
W20260926 23:36:06.180745 21516 connection.cc:535] client connection to 127.0.0.1:33447 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33447): Connection reset by peer (error 104)
2026-09-26T23:36:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2152 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:36:06 dist-test-slave-gwfx krb5kdc[21631](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:36:06 dist-test-slave-gwfx krb5kdc[21631](info): set up 2 sockets
Sep 26 23:36:06 dist-test-slave-gwfx krb5kdc[21631](info): commencing operation
krb5kdc: starting...
W20260926 23:36:07.215534 388 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.004s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:36:07 dist-test-slave-gwfx krb5kdc[21631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:36:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:36:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:36:07.520771 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45361
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:36:07.927986 21644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:36:07.928498 21644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:36:07.928761 21644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:36:07.937186 21644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:36:07.937357 21644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:36:07.937443 21644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:36:07.937527 21644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:36:07.937592 21644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:36:07.949256 21644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:36:07.950794 21644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:36:07.952194 21644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:36:07.966814 21649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:07.966773 21650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:07.966913 21652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:36:07.968041 21644 server_base.cc:1034] running on GCE node
I20260926 23:36:07.969027 21644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:36:07.970805 21644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:36:07.972259 21644 hybrid_clock.cc:648] HybridClock initialized: now 1790465767972162 us; error 97 us; skew 500 ppm
I20260926 23:36:07.972738 21644 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:36:07 dist-test-slave-gwfx krb5kdc[21631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:36:07.976490 21644 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:36:07.978955 21644 webserver.cc:460] Webserver started at http://127.0.0.1:45343/ using document root <none> and password file <none>
I20260926 23:36:07.979859 21644 fs_manager.cc:359] Metadata directory not provided
I20260926 23:36:07.980137 21644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:36:07.980530 21644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:36:07.984951 21644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "fd4761b7e59f4509b60eb98c51d4a8a8"
format_stamp: "Formatted at 2026-09-26 23:36:07 on dist-test-slave-gwfx"
server_key: "3f801d43f19e8c0d22823e0ae98ff5ff"
server_key_iv: "212ff52eb285d01a52a10eb5e98aad3e"
server_key_version: "encryptionkey@0"
I20260926 23:36:07.985971 21644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "fd4761b7e59f4509b60eb98c51d4a8a8"
format_stamp: "Formatted at 2026-09-26 23:36:07 on dist-test-slave-gwfx"
server_key: "3f801d43f19e8c0d22823e0ae98ff5ff"
server_key_iv: "212ff52eb285d01a52a10eb5e98aad3e"
server_key_version: "encryptionkey@0"
I20260926 23:36:07.993191 21644 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20260926 23:36:07.998445 21659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:36:08.000774 21644 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260926 23:36:08.001082 21644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "fd4761b7e59f4509b60eb98c51d4a8a8"
format_stamp: "Formatted at 2026-09-26 23:36:07 on dist-test-slave-gwfx"
server_key: "3f801d43f19e8c0d22823e0ae98ff5ff"
server_key_iv: "212ff52eb285d01a52a10eb5e98aad3e"
server_key_version: "encryptionkey@0"
I20260926 23:36:08.001364 21644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:36:08.032495 21644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:36:08.038806 21644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:36:08.039176 21644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:36:08.063668 21644 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45361
I20260926 23:36:08.063684 21710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45361 every 8 connection(s)
I20260926 23:36:08.065531 21644 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:36:08.070108 21711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:36:08.074738 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 21644
I20260926 23:36:08.074988 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:36:08.075986 388 external_mini_cluster.cc:1428] Setting key 15aa3769dbb4a62708a81420c3a5dfd5
I20260926 23:36:08.087246 21711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd4761b7e59f4509b60eb98c51d4a8a8: Bootstrap starting.
I20260926 23:36:08.091030 21711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd4761b7e59f4509b60eb98c51d4a8a8: Neither blocks nor log segments found. Creating new log.
I20260926 23:36:08.092324 21711 log.cc:824] T 00000000000000000000000000000000 P fd4761b7e59f4509b60eb98c51d4a8a8: Log is configured to *not* fsync() on all Append() calls
I20260926 23:36:08.094525 21714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:08.079013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:53501 (local address 127.0.0.1:45361)
0926 23:36:08.080163 (+ 1150us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:08.080181 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:08.080249 (+ 68us) server_negotiation.cc:407] Connection header received
0926 23:36:08.082832 (+ 2583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:08.082875 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:08.083611 (+ 736us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:08.084242 (+ 631us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:08.085026 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:08.086655 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:08.088433 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:08.089373 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:08.089623 (+ 250us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:08.091148 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:08.091161 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:08.091191 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:36:08.091987 (+ 796us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:08.092094 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:36:08.092129 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:08.092299 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:08.092621 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 23:36:08.092861 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":806,"thread_start_us":545,"threads_started":1}
I20260926 23:36:08.097028 21711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd4761b7e59f4509b60eb98c51d4a8a8: No bootstrap required, opened a new log
I20260926 23:36:08.100162 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:36:08.105840 21711 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd4761b7e59f4509b60eb98c51d4a8a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd4761b7e59f4509b60eb98c51d4a8a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45361 } }
I20260926 23:36:08.106400 21711 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd4761b7e59f4509b60eb98c51d4a8a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:36:08.106509 21711 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd4761b7e59f4509b60eb98c51d4a8a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd4761b7e59f4509b60eb98c51d4a8a8, State: Initialized, Role: FOLLOWER
I20260926 23:36:08.108064 21711 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd4761b7e59f4509b60eb98c51d4a8a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd4761b7e59f4509b60eb98c51d4a8a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45361 } }
I20260926 23:36:08.108893 21711 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd4761b7e59f4509b60eb98c51d4a8a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:36:08.109328 21711 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd4761b7e59f4509b60eb98c51d4a8a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:36:08.109627 21711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd4761b7e59f4509b60eb98c51d4a8a8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:36:08.112974 21711 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd4761b7e59f4509b60eb98c51d4a8a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd4761b7e59f4509b60eb98c51d4a8a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45361 } }
I20260926 23:36:08.113668 21711 leader_election.cc:302] T 00000000000000000000000000000000 P fd4761b7e59f4509b60eb98c51d4a8a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd4761b7e59f4509b60eb98c51d4a8a8; no voters:
I20260926 23:36:08.114446 21711 leader_election.cc:288] T 00000000000000000000000000000000 P fd4761b7e59f4509b60eb98c51d4a8a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:36:08.114741 21721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd4761b7e59f4509b60eb98c51d4a8a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:36:08.116008 21714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:08.105475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:44215 (local address 127.0.0.1:45361)
0926 23:36:08.105807 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:08.105820 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:08.106089 (+ 269us) server_negotiation.cc:407] Connection header received
0926 23:36:08.106405 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:08.106411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:08.106508 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:08.106618 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:08.107459 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:08.109607 (+ 2148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:08.111640 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:08.112429 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:08.112629 (+ 200us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:08.114211 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:08.114220 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:08.114225 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:36:08.114262 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:08.114385 (+ 123us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:36:08.114420 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:08.114571 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:08.115527 (+ 956us) server_negotiation.cc:299] Negotiation successful
0926 23:36:08.115605 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260926 23:36:08.116143 21721 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd4761b7e59f4509b60eb98c51d4a8a8 [term 1 LEADER]: Becoming Leader. State: Replica: fd4761b7e59f4509b60eb98c51d4a8a8, State: Running, Role: LEADER
I20260926 23:36:08.117149 21721 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd4761b7e59f4509b60eb98c51d4a8a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd4761b7e59f4509b60eb98c51d4a8a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45361 } }
I20260926 23:36:08.117849 21711 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd4761b7e59f4509b60eb98c51d4a8a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:36:08.121099 21722 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd4761b7e59f4509b60eb98c51d4a8a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd4761b7e59f4509b60eb98c51d4a8a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd4761b7e59f4509b60eb98c51d4a8a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45361 } } }
I20260926 23:36:08.121649 21722 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd4761b7e59f4509b60eb98c51d4a8a8 [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:08.121279 21723 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd4761b7e59f4509b60eb98c51d4a8a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd4761b7e59f4509b60eb98c51d4a8a8. Latest consensus state: current_term: 1 leader_uuid: "fd4761b7e59f4509b60eb98c51d4a8a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd4761b7e59f4509b60eb98c51d4a8a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45361 } } }
I20260926 23:36:08.121868 21723 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd4761b7e59f4509b60eb98c51d4a8a8 [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:08.124333 21727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:36:08.130656 21727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:36:08.144330 21727 catalog_manager.cc:1347] Generated new cluster ID: dc90960b740b4ce0990e224ed1533906
I20260926 23:36:08.144673 21727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:36:08.157702 21727 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:36:08.160477 21727 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:36:08.179876 21727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd4761b7e59f4509b60eb98c51d4a8a8: Generated new TSK 0
I20260926 23:36:08.181047 21727 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:36:08.196544 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 21644
W20260926 23:36:08.198179 21706 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.110:53501
W20260926 23:36:08.199306 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 21644: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:36:08.199488 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 21644
W20260926 23:36:08.214934 21629 connection.cc:535] client connection to 127.0.0.1:45361 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45361): Connection reset by peer (error 104)
2026-09-26T23:36:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2033 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:36:08 dist-test-slave-gwfx krb5kdc[21742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:36:08 dist-test-slave-gwfx krb5kdc[21742](info): set up 2 sockets
Sep 26 23:36:08 dist-test-slave-gwfx krb5kdc[21742](info): commencing operation
krb5kdc: starting...
W20260926 23:36:09.259452 388 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:36:09 dist-test-slave-gwfx krb5kdc[21742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:36:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:36:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:36:09.570626 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46827
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:36:09.979797 21755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:36:09.980242 21755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:36:09.980401 21755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:36:09.989283 21755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:36:09.989508 21755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:36:09.989609 21755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:36:09.989718 21755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:36:09.989782 21755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:36:10.001595 21755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:36:10.003487 21755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:36:10.005267 21755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:36:10.020543 21760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:10.020581 21761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:10.021420 21763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:36:10.021553 21755 server_base.cc:1034] running on GCE node
I20260926 23:36:10.022576 21755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:36:10.024552 21755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:36:10.025802 21755 hybrid_clock.cc:648] HybridClock initialized: now 1790465770025746 us; error 52 us; skew 500 ppm
I20260926 23:36:10.026273 21755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:36:10 dist-test-slave-gwfx krb5kdc[21742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:36:10.034999 21755 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:36:10.037202 21755 webserver.cc:460] Webserver started at http://127.0.0.1:37201/ using document root <none> and password file <none>
I20260926 23:36:10.038170 21755 fs_manager.cc:359] Metadata directory not provided
I20260926 23:36:10.038329 21755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:36:10.038770 21755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:36:10.042872 21755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "e2639a81cefd46df8846c69b180e67aa"
format_stamp: "Formatted at 2026-09-26 23:36:10 on dist-test-slave-gwfx"
server_key: "6efd9ae3e19a71fec24b988cc233e75c"
server_key_iv: "f875bc1df1b0eea48df792c6937ca9a4"
server_key_version: "encryptionkey@0"
I20260926 23:36:10.043944 21755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "e2639a81cefd46df8846c69b180e67aa"
format_stamp: "Formatted at 2026-09-26 23:36:10 on dist-test-slave-gwfx"
server_key: "6efd9ae3e19a71fec24b988cc233e75c"
server_key_iv: "f875bc1df1b0eea48df792c6937ca9a4"
server_key_version: "encryptionkey@0"
I20260926 23:36:10.051029 21755 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260926 23:36:10.056082 21770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:36:10.058388 21755 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:36:10.058740 21755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "e2639a81cefd46df8846c69b180e67aa"
format_stamp: "Formatted at 2026-09-26 23:36:10 on dist-test-slave-gwfx"
server_key: "6efd9ae3e19a71fec24b988cc233e75c"
server_key_iv: "f875bc1df1b0eea48df792c6937ca9a4"
server_key_version: "encryptionkey@0"
I20260926 23:36:10.059054 21755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:36:10.066619 21755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:36:10.073050 21755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:36:10.073451 21755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:36:10.097223 21755 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46827
I20260926 23:36:10.097316 21821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46827 every 8 connection(s)
I20260926 23:36:10.099215 21755 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:36:10.104466 21822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:36:10.105099 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 21755
I20260926 23:36:10.105314 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:36:10.105909 388 external_mini_cluster.cc:1428] Setting key 44d7b0c9cbb05bd4e861b2a6e819cd76
I20260926 23:36:10.118372 21825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:10.108678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:38687 (local address 127.0.0.1:46827)
0926 23:36:10.109935 (+ 1257us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:10.109957 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:10.110020 (+ 63us) server_negotiation.cc:407] Connection header received
0926 23:36:10.112274 (+ 2254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:10.112318 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:10.112942 (+ 624us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:10.113479 (+ 537us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:10.114908 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:10.114917 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:10.114936 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:36:10.114958 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:10.115126 (+ 168us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:36:10.115161 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:10.115436 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:10.115583 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 23:36:10.115891 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":913,"thread_start_us":554,"threads_started":1}
I20260926 23:36:10.122004 21822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2639a81cefd46df8846c69b180e67aa: Bootstrap starting.
I20260926 23:36:10.124784 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:36:10.125893 21822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2639a81cefd46df8846c69b180e67aa: Neither blocks nor log segments found. Creating new log.
I20260926 23:36:10.127068 21822 log.cc:824] T 00000000000000000000000000000000 P e2639a81cefd46df8846c69b180e67aa: Log is configured to *not* fsync() on all Append() calls
I20260926 23:36:10.131600 21822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2639a81cefd46df8846c69b180e67aa: No bootstrap required, opened a new log
I20260926 23:36:10.133602 21825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:10.130735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:60315 (local address 127.0.0.1:46827)
0926 23:36:10.130967 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:10.130974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:10.131274 (+ 300us) server_negotiation.cc:407] Connection header received
0926 23:36:10.131450 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:10.131456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:10.131535 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:10.131699 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:10.132703 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:10.132710 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:10.132714 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:36:10.132739 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:10.132825 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:36:10.132862 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:10.133086 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:10.133144 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 23:36:10.133226 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 23:36:10.141695 21822 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2639a81cefd46df8846c69b180e67aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2639a81cefd46df8846c69b180e67aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46827 } }
I20260926 23:36:10.142287 21822 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2639a81cefd46df8846c69b180e67aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:36:10.142380 21822 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2639a81cefd46df8846c69b180e67aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2639a81cefd46df8846c69b180e67aa, State: Initialized, Role: FOLLOWER
I20260926 23:36:10.143116 21822 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2639a81cefd46df8846c69b180e67aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2639a81cefd46df8846c69b180e67aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46827 } }
I20260926 23:36:10.143378 21822 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2639a81cefd46df8846c69b180e67aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:36:10.143505 21822 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2639a81cefd46df8846c69b180e67aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:36:10.143651 21822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2639a81cefd46df8846c69b180e67aa [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:36:10.146582 21822 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2639a81cefd46df8846c69b180e67aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2639a81cefd46df8846c69b180e67aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46827 } }
I20260926 23:36:10.147298 21822 leader_election.cc:302] T 00000000000000000000000000000000 P e2639a81cefd46df8846c69b180e67aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2639a81cefd46df8846c69b180e67aa; no voters:
I20260926 23:36:10.148016 21822 leader_election.cc:288] T 00000000000000000000000000000000 P e2639a81cefd46df8846c69b180e67aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:36:10.148288 21832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2639a81cefd46df8846c69b180e67aa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:36:10.149290 21832 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2639a81cefd46df8846c69b180e67aa [term 1 LEADER]: Becoming Leader. State: Replica: e2639a81cefd46df8846c69b180e67aa, State: Running, Role: LEADER
I20260926 23:36:10.149988 21832 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2639a81cefd46df8846c69b180e67aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2639a81cefd46df8846c69b180e67aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46827 } }
I20260926 23:36:10.150624 21822 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2639a81cefd46df8846c69b180e67aa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:36:10.153704 21834 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2639a81cefd46df8846c69b180e67aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2639a81cefd46df8846c69b180e67aa. Latest consensus state: current_term: 1 leader_uuid: "e2639a81cefd46df8846c69b180e67aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2639a81cefd46df8846c69b180e67aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46827 } } }
I20260926 23:36:10.154315 21834 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2639a81cefd46df8846c69b180e67aa [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:10.155156 21833 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2639a81cefd46df8846c69b180e67aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2639a81cefd46df8846c69b180e67aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2639a81cefd46df8846c69b180e67aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46827 } } }
I20260926 23:36:10.155510 21833 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2639a81cefd46df8846c69b180e67aa [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:10.160789 21839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:36:10.168149 21839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:36:10.180866 21839 catalog_manager.cc:1347] Generated new cluster ID: 608b941ab89b488cbd3826e1a683c432
I20260926 23:36:10.181041 21839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:36:10.194156 21839 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:36:10.194909 21839 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:36:10.204633 21839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2639a81cefd46df8846c69b180e67aa: Generated new TSK 0
I20260926 23:36:10.205793 21839 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:36:10.225035 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 21755
W20260926 23:36:10.226557 21817 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.110:38687
W20260926 23:36:10.227622 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 21755: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:36:10.227821 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 21755
2026-09-26T23:36:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2031 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:36:10 dist-test-slave-gwfx krb5kdc[21853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:36:10 dist-test-slave-gwfx krb5kdc[21853](info): set up 2 sockets
Sep 26 23:36:10 dist-test-slave-gwfx krb5kdc[21853](info): commencing operation
krb5kdc: starting...
W20260926 23:36:11.283746 388 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:36:11 dist-test-slave-gwfx krb5kdc[21853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:36:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:36:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:36:11.592505 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45785
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:36:11.988984 21866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:36:11.989341 21866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:36:11.989459 21866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:36:11.998118 21866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:36:11.998363 21866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:36:11.998420 21866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:36:11.998538 21866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:36:11.998605 21866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:36:12.009356 21866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45785
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:36:12.010841 21866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:36:12.012214 21866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:36:12.025652 21871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:12.025905 21872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:12.025964 21874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:36:12.027714 21866 server_base.cc:1034] running on GCE node
I20260926 23:36:12.028744 21866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:36:12.030524 21866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:36:12.031872 21866 hybrid_clock.cc:648] HybridClock initialized: now 1790465772031800 us; error 68 us; skew 500 ppm
I20260926 23:36:12.032356 21866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:36:12 dist-test-slave-gwfx krb5kdc[21853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465772, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:36:12.036312 21866 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:36:12.038519 21866 webserver.cc:460] Webserver started at http://127.0.0.1:33873/ using document root <none> and password file <none>
I20260926 23:36:12.039438 21866 fs_manager.cc:359] Metadata directory not provided
I20260926 23:36:12.039592 21866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:36:12.039925 21866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:36:12.043381 21866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "70ea0b62e490483588cd51defdfea7d0"
format_stamp: "Formatted at 2026-09-26 23:36:12 on dist-test-slave-gwfx"
server_key: "933079aa2d75cf72dd59b9bc0df45abb"
server_key_iv: "683ec8b325b4821af896ea654560d64c"
server_key_version: "encryptionkey@0"
I20260926 23:36:12.044288 21866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "70ea0b62e490483588cd51defdfea7d0"
format_stamp: "Formatted at 2026-09-26 23:36:12 on dist-test-slave-gwfx"
server_key: "933079aa2d75cf72dd59b9bc0df45abb"
server_key_iv: "683ec8b325b4821af896ea654560d64c"
server_key_version: "encryptionkey@0"
I20260926 23:36:12.051155 21866 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 23:36:12.055528 21881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:36:12.057420 21866 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 23:36:12.057665 21866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "70ea0b62e490483588cd51defdfea7d0"
format_stamp: "Formatted at 2026-09-26 23:36:12 on dist-test-slave-gwfx"
server_key: "933079aa2d75cf72dd59b9bc0df45abb"
server_key_iv: "683ec8b325b4821af896ea654560d64c"
server_key_version: "encryptionkey@0"
I20260926 23:36:12.057895 21866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:36:12.080443 21866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:36:12.086277 21866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:36:12.086639 21866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:36:12.107574 21866 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45785
I20260926 23:36:12.107607 21932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45785 every 8 connection(s)
I20260926 23:36:12.109349 21866 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:36:12.114176 21933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:36:12.115624 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 21866
I20260926 23:36:12.115816 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:36:12.116408 388 external_mini_cluster.cc:1428] Setting key b91a5380075fe558f773939627de7091
I20260926 23:36:12.130386 21933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70ea0b62e490483588cd51defdfea7d0: Bootstrap starting.
Sep 26 23:36:12 dist-test-slave-gwfx krb5kdc[21853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 23:36:12.134918 21933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70ea0b62e490483588cd51defdfea7d0: Neither blocks nor log segments found. Creating new log.
I20260926 23:36:12.136226 21933 log.cc:824] T 00000000000000000000000000000000 P 70ea0b62e490483588cd51defdfea7d0: Log is configured to *not* fsync() on all Append() calls
I20260926 23:36:12.140503 21933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70ea0b62e490483588cd51defdfea7d0: No bootstrap required, opened a new log
I20260926 23:36:12.143735 21936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:12.118571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:54073 (local address 127.0.0.1:45785)
0926 23:36:12.119462 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:12.119482 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:12.119534 (+ 52us) server_negotiation.cc:407] Connection header received
0926 23:36:12.121487 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:12.121523 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:12.122096 (+ 573us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:12.122523 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:12.124494 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:12.125933 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:12.127882 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:12.128691 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:12.128839 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:12.134182 (+ 5343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:12.134192 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:12.134220 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:12.134272 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:12.138160 (+ 3888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:12.139500 (+ 1340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:12.139508 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:12.139516 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:12.139724 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:12.140277 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:12.140281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:12.140284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:12.141034 (+ 750us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:12.141331 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:12.141916 (+ 585us) server_negotiation.cc:299] Negotiation successful
0926 23:36:12.142307 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":362,"threads_started":1}
I20260926 23:36:12.148692 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:36:12.150556 21933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70ea0b62e490483588cd51defdfea7d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ea0b62e490483588cd51defdfea7d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45785 } }
I20260926 23:36:12.151180 21933 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70ea0b62e490483588cd51defdfea7d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:36:12.151291 21933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70ea0b62e490483588cd51defdfea7d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70ea0b62e490483588cd51defdfea7d0, State: Initialized, Role: FOLLOWER
I20260926 23:36:12.152146 21933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70ea0b62e490483588cd51defdfea7d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ea0b62e490483588cd51defdfea7d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45785 } }
I20260926 23:36:12.152520 21933 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70ea0b62e490483588cd51defdfea7d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:36:12.152733 21933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70ea0b62e490483588cd51defdfea7d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:36:12.152949 21933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70ea0b62e490483588cd51defdfea7d0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:36:12.156324 21933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70ea0b62e490483588cd51defdfea7d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ea0b62e490483588cd51defdfea7d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45785 } }
I20260926 23:36:12.157035 21933 leader_election.cc:302] T 00000000000000000000000000000000 P 70ea0b62e490483588cd51defdfea7d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70ea0b62e490483588cd51defdfea7d0; no voters:
I20260926 23:36:12.157773 21933 leader_election.cc:288] T 00000000000000000000000000000000 P 70ea0b62e490483588cd51defdfea7d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:36:12.157886 21943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70ea0b62e490483588cd51defdfea7d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:36:12.159214 21943 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70ea0b62e490483588cd51defdfea7d0 [term 1 LEADER]: Becoming Leader. State: Replica: 70ea0b62e490483588cd51defdfea7d0, State: Running, Role: LEADER
I20260926 23:36:12.159993 21943 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70ea0b62e490483588cd51defdfea7d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ea0b62e490483588cd51defdfea7d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45785 } }
I20260926 23:36:12.160686 21933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70ea0b62e490483588cd51defdfea7d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:36:12.164220 21945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70ea0b62e490483588cd51defdfea7d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70ea0b62e490483588cd51defdfea7d0. Latest consensus state: current_term: 1 leader_uuid: "70ea0b62e490483588cd51defdfea7d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ea0b62e490483588cd51defdfea7d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45785 } } }
I20260926 23:36:12.164934 21945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70ea0b62e490483588cd51defdfea7d0 [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:12.164763 21944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70ea0b62e490483588cd51defdfea7d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70ea0b62e490483588cd51defdfea7d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ea0b62e490483588cd51defdfea7d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45785 } } }
I20260926 23:36:12.165271 21944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70ea0b62e490483588cd51defdfea7d0 [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:12.170444 21949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:36:12.172745 21936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:12.154779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:43225 (local address 127.0.0.1:45785)
0926 23:36:12.155020 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:12.155028 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:12.155466 (+ 438us) server_negotiation.cc:407] Connection header received
0926 23:36:12.155633 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:12.155639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:12.155738 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:12.155893 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:12.157601 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:12.158828 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:12.160289 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:12.160807 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:12.160945 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:12.164292 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:12.164303 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:12.164308 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:12.164351 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:12.168418 (+ 4067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:12.169596 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:12.169603 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:12.169607 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:12.169809 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:12.170506 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:12.170512 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:12.170515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:12.171120 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:12.171422 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:12.172095 (+ 673us) server_negotiation.cc:299] Negotiation successful
0926 23:36:12.172334 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 23:36:12.177300 21949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:36:12.189051 21949 catalog_manager.cc:1347] Generated new cluster ID: 3f71ae2bc79847f1a1ab7a138d7b5433
I20260926 23:36:12.189265 21949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:36:12.219913 21949 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:36:12.222082 21949 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:36:12.240159 21949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70ea0b62e490483588cd51defdfea7d0: Generated new TSK 0
I20260926 23:36:12.241619 21949 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:36:12.260550 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 21866
I20260926 23:36:12.261897 21928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:36:12.368122 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 21866
W20260926 23:36:12.386965 21851 connection.cc:535] client connection to 127.0.0.1:45785 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45785): Connection reset by peer (error 104)
2026-09-26T23:36:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2145 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:36:12 dist-test-slave-gwfx krb5kdc[21965](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:36:12 dist-test-slave-gwfx krb5kdc[21965](info): set up 2 sockets
Sep 26 23:36:12 dist-test-slave-gwfx krb5kdc[21965](info): commencing operation
krb5kdc: starting...
W20260926 23:36:13.443539 388 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:36:13 dist-test-slave-gwfx krb5kdc[21965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:36:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:36:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:36:13.758226 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33893
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:36:14.161732 21978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:36:14.162147 21978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:36:14.162278 21978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:36:14.170960 21978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:36:14.171139 21978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:36:14.171198 21978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:36:14.171293 21978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:36:14.171352 21978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:36:14.183131 21978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33893
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:36:14.184967 21978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:36:14.186718 21978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:36:14.201367 21984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:14.201643 21986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:14.201385 21983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:36:14.203152 21978 server_base.cc:1034] running on GCE node
I20260926 23:36:14.204300 21978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:36:14.206600 21978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:36:14.207991 21978 hybrid_clock.cc:648] HybridClock initialized: now 1790465774207892 us; error 83 us; skew 500 ppm
I20260926 23:36:14.208665 21978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:36:14 dist-test-slave-gwfx krb5kdc[21965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:36:14.213469 21978 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:36:14.215739 21978 webserver.cc:460] Webserver started at http://127.0.0.1:34531/ using document root <none> and password file <none>
I20260926 23:36:14.216706 21978 fs_manager.cc:359] Metadata directory not provided
I20260926 23:36:14.216851 21978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:36:14.217228 21978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:36:14.221378 21978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "010a3f9432f74315bc6833c42d58af84"
format_stamp: "Formatted at 2026-09-26 23:36:14 on dist-test-slave-gwfx"
server_key: "d2f22921e0228c3a7c5e86c99cfa6c1e"
server_key_iv: "b9b1fcae890bac4547bbd9f5670f505f"
server_key_version: "encryptionkey@0"
I20260926 23:36:14.222507 21978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "010a3f9432f74315bc6833c42d58af84"
format_stamp: "Formatted at 2026-09-26 23:36:14 on dist-test-slave-gwfx"
server_key: "d2f22921e0228c3a7c5e86c99cfa6c1e"
server_key_iv: "b9b1fcae890bac4547bbd9f5670f505f"
server_key_version: "encryptionkey@0"
I20260926 23:36:14.232811 21978 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.007s sys 0.000s
I20260926 23:36:14.239080 21993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:36:14.244163 21978 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.001s
I20260926 23:36:14.244493 21978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "010a3f9432f74315bc6833c42d58af84"
format_stamp: "Formatted at 2026-09-26 23:36:14 on dist-test-slave-gwfx"
server_key: "d2f22921e0228c3a7c5e86c99cfa6c1e"
server_key_iv: "b9b1fcae890bac4547bbd9f5670f505f"
server_key_version: "encryptionkey@0"
I20260926 23:36:14.244764 21978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:36:14.282249 21978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:36:14.288277 21978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:36:14.288763 21978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:36:14.312945 21978 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33893
I20260926 23:36:14.312995 22044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33893 every 8 connection(s)
I20260926 23:36:14.314782 21978 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:36:14.319507 22045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:36:14.322428 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 21978
I20260926 23:36:14.322640 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:36:14.323249 388 external_mini_cluster.cc:1428] Setting key f8d8030bca08a6105674ace3b6d04634
I20260926 23:36:14.333464 22045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 010a3f9432f74315bc6833c42d58af84: Bootstrap starting.
I20260926 23:36:14.337271 22045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 010a3f9432f74315bc6833c42d58af84: Neither blocks nor log segments found. Creating new log.
I20260926 23:36:14.338543 22045 log.cc:824] T 00000000000000000000000000000000 P 010a3f9432f74315bc6833c42d58af84: Log is configured to *not* fsync() on all Append() calls
I20260926 23:36:14.340418 22048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:14.325704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:34907 (local address 127.0.0.1:33893)
0926 23:36:14.326896 (+ 1192us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:14.326916 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:14.326989 (+ 73us) server_negotiation.cc:407] Connection header received
0926 23:36:14.329174 (+ 2185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:14.329211 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:14.329881 (+ 670us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:14.330434 (+ 553us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:14.331250 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:14.332779 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:14.334343 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:14.335276 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:14.335487 (+ 211us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:14.336938 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:14.336947 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:14.336968 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:36:14.337746 (+ 778us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:14.337870 (+ 124us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:36:14.337903 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:14.338061 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:14.338444 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 23:36:14.338714 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":815,"thread_start_us":432,"threads_started":1}
I20260926 23:36:14.343047 22045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 010a3f9432f74315bc6833c42d58af84: No bootstrap required, opened a new log
I20260926 23:36:14.346418 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:36:14.353001 22045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 010a3f9432f74315bc6833c42d58af84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "010a3f9432f74315bc6833c42d58af84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33893 } }
I20260926 23:36:14.353525 22045 raft_consensus.cc:374] T 00000000000000000000000000000000 P 010a3f9432f74315bc6833c42d58af84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:36:14.353639 22045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 010a3f9432f74315bc6833c42d58af84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 010a3f9432f74315bc6833c42d58af84, State: Initialized, Role: FOLLOWER
I20260926 23:36:14.354447 22045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 010a3f9432f74315bc6833c42d58af84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "010a3f9432f74315bc6833c42d58af84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33893 } }
I20260926 23:36:14.354772 22045 raft_consensus.cc:388] T 00000000000000000000000000000000 P 010a3f9432f74315bc6833c42d58af84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:36:14.354933 22045 raft_consensus.cc:482] T 00000000000000000000000000000000 P 010a3f9432f74315bc6833c42d58af84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:36:14.355108 22045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 010a3f9432f74315bc6833c42d58af84 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:36:14.358373 22045 raft_consensus.cc:504] T 00000000000000000000000000000000 P 010a3f9432f74315bc6833c42d58af84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "010a3f9432f74315bc6833c42d58af84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33893 } }
I20260926 23:36:14.359207 22045 leader_election.cc:302] T 00000000000000000000000000000000 P 010a3f9432f74315bc6833c42d58af84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 010a3f9432f74315bc6833c42d58af84; no voters:
I20260926 23:36:14.360072 22045 leader_election.cc:288] T 00000000000000000000000000000000 P 010a3f9432f74315bc6833c42d58af84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:36:14.360338 22055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 010a3f9432f74315bc6833c42d58af84 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:36:14.361598 22048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:14.352091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:53523 (local address 127.0.0.1:33893)
0926 23:36:14.352411 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:14.352420 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:14.353141 (+ 721us) server_negotiation.cc:407] Connection header received
0926 23:36:14.353244 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:14.353248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:14.353330 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:14.353430 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:14.354265 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:14.355344 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:14.357494 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:14.358116 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:14.358260 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:14.360050 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:14.360062 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:14.360066 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:36:14.360101 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:14.360203 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:36:14.360230 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:14.360400 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:14.361050 (+ 650us) server_negotiation.cc:299] Negotiation successful
0926 23:36:14.361141 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 23:36:14.361606 22055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 010a3f9432f74315bc6833c42d58af84 [term 1 LEADER]: Becoming Leader. State: Replica: 010a3f9432f74315bc6833c42d58af84, State: Running, Role: LEADER
I20260926 23:36:14.362761 22045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 010a3f9432f74315bc6833c42d58af84 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:36:14.363349 22055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 010a3f9432f74315bc6833c42d58af84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "010a3f9432f74315bc6833c42d58af84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33893 } }
I20260926 23:36:14.367532 22056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 010a3f9432f74315bc6833c42d58af84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "010a3f9432f74315bc6833c42d58af84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "010a3f9432f74315bc6833c42d58af84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33893 } } }
I20260926 23:36:14.368137 22056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 010a3f9432f74315bc6833c42d58af84 [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:14.368749 22057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 010a3f9432f74315bc6833c42d58af84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 010a3f9432f74315bc6833c42d58af84. Latest consensus state: current_term: 1 leader_uuid: "010a3f9432f74315bc6833c42d58af84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "010a3f9432f74315bc6833c42d58af84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33893 } } }
I20260926 23:36:14.369082 22057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 010a3f9432f74315bc6833c42d58af84 [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:14.371325 22065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:36:14.377503 22065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:36:14.390066 22065 catalog_manager.cc:1347] Generated new cluster ID: 9ffe4f7917254584b429779fd31a4616
I20260926 23:36:14.390242 22065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:36:14.418766 22065 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:36:14.420899 22065 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:36:14.431226 22065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 010a3f9432f74315bc6833c42d58af84: Generated new TSK 0
I20260926 23:36:14.432701 22065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:36:14.449867 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 21978
W20260926 23:36:14.451858 22040 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.110:34907
W20260926 23:36:14.452863 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 21978: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:36:14.453073 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 21978
W20260926 23:36:14.469883 21963 connection.cc:535] client connection to 127.0.0.1:33893 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33893): Connection reset by peer (error 104)
2026-09-26T23:36:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2078 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:36:14 dist-test-slave-gwfx krb5kdc[22076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:36:14 dist-test-slave-gwfx krb5kdc[22076](info): set up 2 sockets
Sep 26 23:36:14 dist-test-slave-gwfx krb5kdc[22076](info): commencing operation
krb5kdc: starting...
W20260926 23:36:15.507273 388 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.007s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:36:15 dist-test-slave-gwfx krb5kdc[22076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:36:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:36:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 23:36:15.812081 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44879
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:36:16.196333 22089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:36:16.196779 22089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:36:16.196893 22089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:36:16.205265 22089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:36:16.205443 22089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:36:16.205497 22089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:36:16.205566 22089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:36:16.205616 22089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:36:16.217633 22089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44879
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790465400694755-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:36:16.219419 22089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:36:16.221094 22089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:36:16.235502 22097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:16.235556 22095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:16.235651 22094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:36:16.237040 22089 server_base.cc:1034] running on GCE node
I20260926 23:36:16.238188 22089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:36:16.240340 22089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:36:16.241636 22089 hybrid_clock.cc:648] HybridClock initialized: now 1790465776241547 us; error 75 us; skew 500 ppm
I20260926 23:36:16.242138 22089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:36:16 dist-test-slave-gwfx krb5kdc[22076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:36:16.250242 22089 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 23:36:16.252516 22089 webserver.cc:460] Webserver started at http://127.0.0.1:37725/ using document root <none> and password file <none>
I20260926 23:36:16.253461 22089 fs_manager.cc:359] Metadata directory not provided
I20260926 23:36:16.253621 22089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:36:16.253998 22089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:36:16.258163 22089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "3bb343163dfe4be09de6114a194c8e64"
format_stamp: "Formatted at 2026-09-26 23:36:16 on dist-test-slave-gwfx"
server_key: "981ccad5fb1381d9e69a69dded68318b"
server_key_iv: "8819e9e5d30aa413df87637822580a4d"
server_key_version: "encryptionkey@0"
I20260926 23:36:16.259317 22089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "3bb343163dfe4be09de6114a194c8e64"
format_stamp: "Formatted at 2026-09-26 23:36:16 on dist-test-slave-gwfx"
server_key: "981ccad5fb1381d9e69a69dded68318b"
server_key_iv: "8819e9e5d30aa413df87637822580a4d"
server_key_version: "encryptionkey@0"
I20260926 23:36:16.267144 22089 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260926 23:36:16.272347 22104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:36:16.274384 22089 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20260926 23:36:16.274636 22089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "3bb343163dfe4be09de6114a194c8e64"
format_stamp: "Formatted at 2026-09-26 23:36:16 on dist-test-slave-gwfx"
server_key: "981ccad5fb1381d9e69a69dded68318b"
server_key_iv: "8819e9e5d30aa413df87637822580a4d"
server_key_version: "encryptionkey@0"
I20260926 23:36:16.274966 22089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:36:16.282255 22089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:36:16.287380 22089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:36:16.287701 22089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:36:16.309314 22089 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44879
I20260926 23:36:16.309379 22155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44879 every 8 connection(s)
I20260926 23:36:16.311112 22089 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:36:16.314273 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 22089
I20260926 23:36:16.314459 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:36:16.315001 388 external_mini_cluster.cc:1428] Setting key b236e0ffd139abf3ccb043f7c7421ba1
I20260926 23:36:16.315634 22156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:36:16.325558 22159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:16.317160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:43485 (local address 127.0.0.1:44879)
0926 23:36:16.318138 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:16.318155 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:16.318212 (+ 57us) server_negotiation.cc:407] Connection header received
0926 23:36:16.320781 (+ 2569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:16.320817 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:16.321446 (+ 629us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:16.322017 (+ 571us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:16.323313 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:16.323322 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:16.323342 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:36:16.323370 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:16.323460 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:36:16.323487 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:16.323759 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:16.323926 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 23:36:16.324183 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":383,"threads_started":1}
I20260926 23:36:16.331539 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:36:16.332468 22156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bb343163dfe4be09de6114a194c8e64: Bootstrap starting.
I20260926 23:36:16.336009 22156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bb343163dfe4be09de6114a194c8e64: Neither blocks nor log segments found. Creating new log.
I20260926 23:36:16.337385 22156 log.cc:824] T 00000000000000000000000000000000 P 3bb343163dfe4be09de6114a194c8e64: Log is configured to *not* fsync() on all Append() calls
I20260926 23:36:16.339910 22159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:16.336651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:36889 (local address 127.0.0.1:44879)
0926 23:36:16.336890 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:16.336898 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:16.337570 (+ 672us) server_negotiation.cc:407] Connection header received
0926 23:36:16.337829 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:16.337834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:16.337917 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:16.338084 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:16.339179 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:16.339184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:16.339188 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:36:16.339206 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:16.339252 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:36:16.339276 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:16.339464 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:16.339530 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 23:36:16.339619 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 23:36:16.340700 22156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bb343163dfe4be09de6114a194c8e64: No bootstrap required, opened a new log
I20260926 23:36:16.348512 22156 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bb343163dfe4be09de6114a194c8e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb343163dfe4be09de6114a194c8e64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44879 } }
I20260926 23:36:16.348961 22156 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bb343163dfe4be09de6114a194c8e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:36:16.349031 22156 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bb343163dfe4be09de6114a194c8e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bb343163dfe4be09de6114a194c8e64, State: Initialized, Role: FOLLOWER
I20260926 23:36:16.349638 22156 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bb343163dfe4be09de6114a194c8e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb343163dfe4be09de6114a194c8e64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44879 } }
I20260926 23:36:16.349872 22156 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bb343163dfe4be09de6114a194c8e64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:36:16.350010 22156 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bb343163dfe4be09de6114a194c8e64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:36:16.350168 22156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bb343163dfe4be09de6114a194c8e64 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:36:16.352486 22156 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bb343163dfe4be09de6114a194c8e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb343163dfe4be09de6114a194c8e64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44879 } }
I20260926 23:36:16.353026 22156 leader_election.cc:302] T 00000000000000000000000000000000 P 3bb343163dfe4be09de6114a194c8e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bb343163dfe4be09de6114a194c8e64; no voters:
I20260926 23:36:16.353706 22156 leader_election.cc:288] T 00000000000000000000000000000000 P 3bb343163dfe4be09de6114a194c8e64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:36:16.354012 22166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bb343163dfe4be09de6114a194c8e64 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:36:16.355556 22166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bb343163dfe4be09de6114a194c8e64 [term 1 LEADER]: Becoming Leader. State: Replica: 3bb343163dfe4be09de6114a194c8e64, State: Running, Role: LEADER
I20260926 23:36:16.356115 22156 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bb343163dfe4be09de6114a194c8e64 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:36:16.356360 22166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bb343163dfe4be09de6114a194c8e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb343163dfe4be09de6114a194c8e64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44879 } }
I20260926 23:36:16.360029 22168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bb343163dfe4be09de6114a194c8e64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bb343163dfe4be09de6114a194c8e64. Latest consensus state: current_term: 1 leader_uuid: "3bb343163dfe4be09de6114a194c8e64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb343163dfe4be09de6114a194c8e64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44879 } } }
I20260926 23:36:16.360548 22168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bb343163dfe4be09de6114a194c8e64 [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:16.361203 22167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bb343163dfe4be09de6114a194c8e64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bb343163dfe4be09de6114a194c8e64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb343163dfe4be09de6114a194c8e64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44879 } } }
I20260926 23:36:16.361614 22167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bb343163dfe4be09de6114a194c8e64 [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:16.364081 22175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:36:16.369704 22175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:36:16.380494 22175 catalog_manager.cc:1347] Generated new cluster ID: 8d9f27e496f746c4a28e65963b6d1787
I20260926 23:36:16.380685 22175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:36:16.406122 22175 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:36:16.406843 22175 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:36:16.422403 22175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bb343163dfe4be09de6114a194c8e64: Generated new TSK 0
I20260926 23:36:16.423704 22175 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:36:16.439975 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 22089
W20260926 23:36:16.441476 22151 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.110:43485
W20260926 23:36:16.442366 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 22089: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:36:16.442526 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 22089
2026-09-26T23:36:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1991 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:36:16 dist-test-slave-gwfx krb5kdc[22187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:36:16 dist-test-slave-gwfx krb5kdc[22187](info): set up 2 sockets
Sep 26 23:36:16 dist-test-slave-gwfx krb5kdc[22187](info): commencing operation
krb5kdc: starting...
W20260926 23:36:17.499553 388 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:36:17 dist-test-slave-gwfx krb5kdc[22187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:36:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:36:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:36:17.794653 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33519
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:36:18.210280 22200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:36:18.210727 22200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:36:18.210872 22200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:36:18.219776 22200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:36:18.220130 22200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:36:18.220252 22200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:36:18.220348 22200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:36:18.220434 22200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:36:18.232493 22200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33519
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:36:18.234282 22200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:36:18.236068 22200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:36:18.251041 22206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:18.251119 22205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:18.251111 22208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:36:18.253037 22200 server_base.cc:1034] running on GCE node
I20260926 23:36:18.254177 22200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:36:18.256100 22200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:36:18.257362 22200 hybrid_clock.cc:648] HybridClock initialized: now 1790465778257304 us; error 60 us; skew 500 ppm
I20260926 23:36:18.257817 22200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:36:18 dist-test-slave-gwfx krb5kdc[22187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:36:18.261492 22200 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:36:18.263471 22200 webserver.cc:460] Webserver started at http://127.0.97.62:38325/ using document root <none> and password file <none>
I20260926 23:36:18.264287 22200 fs_manager.cc:359] Metadata directory not provided
I20260926 23:36:18.264400 22200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:36:18.264740 22200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:36:18.268138 22200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "d9211145b47e4e69a11ff516a540efb2"
format_stamp: "Formatted at 2026-09-26 23:36:18 on dist-test-slave-gwfx"
server_key: "5f0becf82147dbabea4d2564648158d1"
server_key_iv: "dada41d25e1498e296192e2a86a006d8"
server_key_version: "encryptionkey@0"
I20260926 23:36:18.268985 22200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "d9211145b47e4e69a11ff516a540efb2"
format_stamp: "Formatted at 2026-09-26 23:36:18 on dist-test-slave-gwfx"
server_key: "5f0becf82147dbabea4d2564648158d1"
server_key_iv: "dada41d25e1498e296192e2a86a006d8"
server_key_version: "encryptionkey@0"
I20260926 23:36:18.275522 22200 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 23:36:18.280953 22215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:36:18.283143 22200 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.004s
I20260926 23:36:18.283409 22200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "d9211145b47e4e69a11ff516a540efb2"
format_stamp: "Formatted at 2026-09-26 23:36:18 on dist-test-slave-gwfx"
server_key: "5f0becf82147dbabea4d2564648158d1"
server_key_iv: "dada41d25e1498e296192e2a86a006d8"
server_key_version: "encryptionkey@0"
I20260926 23:36:18.283638 22200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:36:18.305788 22200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:36:18.311352 22200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:36:18.311681 22200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:36:18.333659 22200 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33519
I20260926 23:36:18.333725 22266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33519 every 8 connection(s)
I20260926 23:36:18.335716 22200 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:36:18.336481 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 22200
I20260926 23:36:18.336689 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:36:18.337424 388 external_mini_cluster.cc:1428] Setting key 7521c6d20b6df181c0670f4e4eab72fb
I20260926 23:36:18.341671 22267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 23:36:18 dist-test-slave-gwfx krb5kdc[22187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:36:18.355651 22267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9211145b47e4e69a11ff516a540efb2: Bootstrap starting.
I20260926 23:36:18.360500 22267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9211145b47e4e69a11ff516a540efb2: Neither blocks nor log segments found. Creating new log.
I20260926 23:36:18.362114 22267 log.cc:824] T 00000000000000000000000000000000 P d9211145b47e4e69a11ff516a540efb2: Log is configured to *not* fsync() on all Append() calls
I20260926 23:36:18.364064 22270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:18.339651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39073 (local address 127.0.97.62:33519)
0926 23:36:18.340695 (+ 1044us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:18.340711 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:18.340775 (+ 64us) server_negotiation.cc:407] Connection header received
0926 23:36:18.342766 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:18.342798 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:18.343343 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:18.343790 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:18.345367 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:18.346928 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:18.348332 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:18.349097 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:18.349296 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:18.354008 (+ 4712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:18.354017 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:18.354037 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:18.354096 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:18.358037 (+ 3941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:18.359389 (+ 1352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:18.359401 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:18.359411 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:18.359679 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:18.360223 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:18.360228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:18.360231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:18.361108 (+ 877us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:18.361475 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:18.361990 (+ 515us) server_negotiation.cc:299] Negotiation successful
0926 23:36:18.362421 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":752,"thread_start_us":454,"threads_started":1}
I20260926 23:36:18.366247 22267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9211145b47e4e69a11ff516a540efb2: No bootstrap required, opened a new log
I20260926 23:36:18.369256 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:36:18.375507 22267 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9211145b47e4e69a11ff516a540efb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9211145b47e4e69a11ff516a540efb2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33519 } }
I20260926 23:36:18.375984 22267 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9211145b47e4e69a11ff516a540efb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:36:18.376087 22267 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9211145b47e4e69a11ff516a540efb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9211145b47e4e69a11ff516a540efb2, State: Initialized, Role: FOLLOWER
I20260926 23:36:18.376963 22267 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9211145b47e4e69a11ff516a540efb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9211145b47e4e69a11ff516a540efb2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33519 } }
I20260926 23:36:18.377282 22267 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9211145b47e4e69a11ff516a540efb2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:36:18.377465 22267 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9211145b47e4e69a11ff516a540efb2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:36:18.377672 22267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9211145b47e4e69a11ff516a540efb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:36:18.380870 22267 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9211145b47e4e69a11ff516a540efb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9211145b47e4e69a11ff516a540efb2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33519 } }
I20260926 23:36:18.381546 22267 leader_election.cc:302] T 00000000000000000000000000000000 P d9211145b47e4e69a11ff516a540efb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9211145b47e4e69a11ff516a540efb2; no voters:
I20260926 23:36:18.382323 22267 leader_election.cc:288] T 00000000000000000000000000000000 P d9211145b47e4e69a11ff516a540efb2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:36:18.382457 22277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9211145b47e4e69a11ff516a540efb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:36:18.385524 22267 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9211145b47e4e69a11ff516a540efb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:36:18.389181 22270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:18.373701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55211 (local address 127.0.97.62:33519)
0926 23:36:18.373959 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:18.373966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:18.374305 (+ 339us) server_negotiation.cc:407] Connection header received
0926 23:36:18.374460 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:18.374466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:18.374566 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:18.374724 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:18.376202 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:18.377366 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:18.378910 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:18.379616 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:18.379775 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:18.382432 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:18.382440 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:18.382449 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:18.382508 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:18.385841 (+ 3333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:18.386679 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:18.386734 (+ 55us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:18.386738 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:18.386886 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:18.387396 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:18.387402 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:18.387405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:18.387908 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:18.388223 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:18.388553 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 23:36:18.388746 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 23:36:18.389477 22277 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9211145b47e4e69a11ff516a540efb2 [term 1 LEADER]: Becoming Leader. State: Replica: d9211145b47e4e69a11ff516a540efb2, State: Running, Role: LEADER
I20260926 23:36:18.390570 22277 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9211145b47e4e69a11ff516a540efb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9211145b47e4e69a11ff516a540efb2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33519 } }
I20260926 23:36:18.394516 22278 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9211145b47e4e69a11ff516a540efb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9211145b47e4e69a11ff516a540efb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9211145b47e4e69a11ff516a540efb2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33519 } } }
I20260926 23:36:18.396241 22278 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9211145b47e4e69a11ff516a540efb2 [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:18.394522 22279 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9211145b47e4e69a11ff516a540efb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9211145b47e4e69a11ff516a540efb2. Latest consensus state: current_term: 1 leader_uuid: "d9211145b47e4e69a11ff516a540efb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9211145b47e4e69a11ff516a540efb2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33519 } } }
I20260926 23:36:18.397892 22279 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9211145b47e4e69a11ff516a540efb2 [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:18.398940 22288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:36:18.405458 22288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:36:18.417743 22288 catalog_manager.cc:1347] Generated new cluster ID: 9d3d0f674f0542cd9cdeaf2f890c09e0
I20260926 23:36:18.417882 22288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:36:18.436803 22288 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:36:18.438292 22288 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:36:18.448421 22288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9211145b47e4e69a11ff516a540efb2: Generated new TSK 0
I20260926 23:36:18.449496 22288 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:36:18.459981 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 22200
I20260926 23:36:18.461700 22262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:36:18.561285 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 22200
W20260926 23:36:18.576857 22185 connection.cc:535] client connection to 127.0.97.62:33519 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:33519): Connection reset by peer (error 104)
2026-09-26T23:36:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2115 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:36:18 dist-test-slave-gwfx krb5kdc[22299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:36:18 dist-test-slave-gwfx krb5kdc[22299](info): set up 2 sockets
Sep 26 23:36:18 dist-test-slave-gwfx krb5kdc[22299](info): commencing operation
krb5kdc: starting...
W20260926 23:36:19.603421 388 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:36:19 dist-test-slave-gwfx krb5kdc[22299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:36:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:36:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:36:19.903152 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43383
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:36:20.300058 22312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:36:20.300523 22312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:36:20.300686 22312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:36:20.309283 22312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:36:20.309504 22312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:36:20.309600 22312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:36:20.309708 22312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:36:20.309780 22312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:36:20.321964 22312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43383
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:36:20.324245 22312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:36:20.326146 22312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:36:20.341449 22317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:20.341826 22318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:20.342324 22320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:36:20.343348 22312 server_base.cc:1034] running on GCE node
I20260926 23:36:20.344548 22312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:36:20.346946 22312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:36:20.348608 22312 hybrid_clock.cc:648] HybridClock initialized: now 1790465780348514 us; error 93 us; skew 500 ppm
I20260926 23:36:20.349125 22312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:36:20 dist-test-slave-gwfx krb5kdc[22299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:36:20.353807 22312 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:36:20.356021 22312 webserver.cc:460] Webserver started at http://127.0.97.62:34891/ using document root <none> and password file <none>
I20260926 23:36:20.356972 22312 fs_manager.cc:359] Metadata directory not provided
I20260926 23:36:20.357108 22312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:36:20.357461 22312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:36:20.361272 22312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "4e330a06806d496e8e4feb5d98708b21"
format_stamp: "Formatted at 2026-09-26 23:36:20 on dist-test-slave-gwfx"
server_key: "21c43c2d1d1b90bad2777828e3380e06"
server_key_iv: "6c5b75acf4a96bd16eb1f6f2cc99e886"
server_key_version: "encryptionkey@0"
I20260926 23:36:20.362162 22312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "4e330a06806d496e8e4feb5d98708b21"
format_stamp: "Formatted at 2026-09-26 23:36:20 on dist-test-slave-gwfx"
server_key: "21c43c2d1d1b90bad2777828e3380e06"
server_key_iv: "6c5b75acf4a96bd16eb1f6f2cc99e886"
server_key_version: "encryptionkey@0"
I20260926 23:36:20.369252 22312 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260926 23:36:20.374294 22327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:36:20.376364 22312 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 23:36:20.376642 22312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "4e330a06806d496e8e4feb5d98708b21"
format_stamp: "Formatted at 2026-09-26 23:36:20 on dist-test-slave-gwfx"
server_key: "21c43c2d1d1b90bad2777828e3380e06"
server_key_iv: "6c5b75acf4a96bd16eb1f6f2cc99e886"
server_key_version: "encryptionkey@0"
I20260926 23:36:20.376875 22312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:36:20.403468 22312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:36:20.408645 22312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:36:20.408932 22312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:36:20.430415 22312 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43383
I20260926 23:36:20.430446 22378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43383 every 8 connection(s)
I20260926 23:36:20.432184 22312 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:36:20.435984 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 22312
I20260926 23:36:20.436152 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:36:20.436687 388 external_mini_cluster.cc:1428] Setting key 0bee16073731ba90f85d5202c912242c
I20260926 23:36:20.436681 22379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:36:20.452814 22379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e330a06806d496e8e4feb5d98708b21: Bootstrap starting.
I20260926 23:36:20.453984 22382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:20.439155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58815 (local address 127.0.97.62:43383)
0926 23:36:20.440324 (+ 1169us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:20.440342 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:20.440411 (+ 69us) server_negotiation.cc:407] Connection header received
0926 23:36:20.442341 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:20.442374 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:20.443009 (+ 635us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:20.443503 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:20.444204 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:20.446200 (+ 1996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:20.448058 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:20.449178 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:20.449411 (+ 233us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:20.450972 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:20.450983 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:20.451004 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:36:20.451793 (+ 789us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:20.451908 (+ 115us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:36:20.451943 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:20.452135 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:20.452437 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 23:36:20.452636 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":850,"thread_start_us":479,"threads_started":1}
I20260926 23:36:20.456632 22379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e330a06806d496e8e4feb5d98708b21: Neither blocks nor log segments found. Creating new log.
I20260926 23:36:20.457844 22379 log.cc:824] T 00000000000000000000000000000000 P 4e330a06806d496e8e4feb5d98708b21: Log is configured to *not* fsync() on all Append() calls
I20260926 23:36:20.459386 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:36:20.461681 22379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e330a06806d496e8e4feb5d98708b21: No bootstrap required, opened a new log
I20260926 23:36:20.471526 22379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e330a06806d496e8e4feb5d98708b21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e330a06806d496e8e4feb5d98708b21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43383 } }
I20260926 23:36:20.472061 22379 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e330a06806d496e8e4feb5d98708b21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:36:20.472194 22379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e330a06806d496e8e4feb5d98708b21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e330a06806d496e8e4feb5d98708b21, State: Initialized, Role: FOLLOWER
I20260926 23:36:20.472914 22379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e330a06806d496e8e4feb5d98708b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e330a06806d496e8e4feb5d98708b21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43383 } }
I20260926 23:36:20.473232 22379 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e330a06806d496e8e4feb5d98708b21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:36:20.473446 22379 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e330a06806d496e8e4feb5d98708b21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:36:20.473482 22382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:20.464872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60749 (local address 127.0.97.62:43383)
0926 23:36:20.465186 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:20.465194 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:20.465730 (+ 536us) server_negotiation.cc:407] Connection header received
0926 23:36:20.466020 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:20.466027 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:20.466122 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:20.466277 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:20.467052 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:20.467985 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:20.470164 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:20.470843 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:20.470974 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:20.472404 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:20.472410 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:20.472414 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:36:20.472441 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:20.472499 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:36:20.472519 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:20.472648 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:20.472920 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 23:36:20.473039 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260926 23:36:20.473640 22379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e330a06806d496e8e4feb5d98708b21 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:36:20.476336 22379 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e330a06806d496e8e4feb5d98708b21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e330a06806d496e8e4feb5d98708b21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43383 } }
I20260926 23:36:20.476900 22379 leader_election.cc:302] T 00000000000000000000000000000000 P 4e330a06806d496e8e4feb5d98708b21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e330a06806d496e8e4feb5d98708b21; no voters:
I20260926 23:36:20.477552 22379 leader_election.cc:288] T 00000000000000000000000000000000 P 4e330a06806d496e8e4feb5d98708b21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:36:20.477820 22389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e330a06806d496e8e4feb5d98708b21 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:36:20.478639 22389 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e330a06806d496e8e4feb5d98708b21 [term 1 LEADER]: Becoming Leader. State: Replica: 4e330a06806d496e8e4feb5d98708b21, State: Running, Role: LEADER
I20260926 23:36:20.479378 22389 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e330a06806d496e8e4feb5d98708b21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e330a06806d496e8e4feb5d98708b21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43383 } }
I20260926 23:36:20.479910 22379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e330a06806d496e8e4feb5d98708b21 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:36:20.484848 22391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e330a06806d496e8e4feb5d98708b21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e330a06806d496e8e4feb5d98708b21. Latest consensus state: current_term: 1 leader_uuid: "4e330a06806d496e8e4feb5d98708b21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e330a06806d496e8e4feb5d98708b21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43383 } } }
I20260926 23:36:20.485386 22391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e330a06806d496e8e4feb5d98708b21 [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:20.484822 22390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e330a06806d496e8e4feb5d98708b21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e330a06806d496e8e4feb5d98708b21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e330a06806d496e8e4feb5d98708b21" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43383 } } }
I20260926 23:36:20.485896 22390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e330a06806d496e8e4feb5d98708b21 [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:20.487691 22400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:36:20.494212 22400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:36:20.507669 22400 catalog_manager.cc:1347] Generated new cluster ID: d13c912f734848c9863c265d52f8b14a
I20260926 23:36:20.507818 22400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:36:20.520385 22400 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:36:20.522776 22400 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:36:20.533257 22400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e330a06806d496e8e4feb5d98708b21: Generated new TSK 0
I20260926 23:36:20.534504 22400 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:36:20.551393 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 22312
W20260926 23:36:20.553335 22374 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58815
W20260926 23:36:20.554519 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 22312: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:36:20.554772 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 22312
W20260926 23:36:20.570151 22297 connection.cc:535] client connection to 127.0.97.62:43383 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:43383): Connection reset by peer (error 104)
2026-09-26T23:36:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1993 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:36:20 dist-test-slave-gwfx krb5kdc[22410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:36:20 dist-test-slave-gwfx krb5kdc[22410](info): set up 2 sockets
Sep 26 23:36:20 dist-test-slave-gwfx krb5kdc[22410](info): commencing operation
krb5kdc: starting...
W20260926 23:36:21.611327 388 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:36:21 dist-test-slave-gwfx krb5kdc[22410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:36:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:36:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:36:21.911137 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43085
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:36:22.309672 22423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:36:22.310105 22423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:36:22.310238 22423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:36:22.318619 22423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:36:22.318846 22423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:36:22.318928 22423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:36:22.319024 22423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:36:22.319087 22423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:36:22.330946 22423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43085
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:36:22.332752 22423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:36:22.334492 22423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:36:22.348958 22428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:22.349092 22431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:22.349138 22429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:36:22.350652 22423 server_base.cc:1034] running on GCE node
I20260926 23:36:22.351763 22423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:36:22.353591 22423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:36:22.354913 22423 hybrid_clock.cc:648] HybridClock initialized: now 1790465782354827 us; error 58 us; skew 500 ppm
I20260926 23:36:22.355448 22423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:36:22 dist-test-slave-gwfx krb5kdc[22410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:36:22.363111 22423 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:36:22.365053 22423 webserver.cc:460] Webserver started at http://127.0.97.62:37593/ using document root <none> and password file <none>
I20260926 23:36:22.366158 22423 fs_manager.cc:359] Metadata directory not provided
I20260926 23:36:22.366381 22423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:36:22.366827 22423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:36:22.371232 22423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "8d4c8693f9e04709a3072d56a9aae532"
format_stamp: "Formatted at 2026-09-26 23:36:22 on dist-test-slave-gwfx"
server_key: "cce8ee76ec18b5f7ead7c83a40b1c4b5"
server_key_iv: "a5b48b91b0e64c7dcfe1b498d48f0dce"
server_key_version: "encryptionkey@0"
I20260926 23:36:22.372228 22423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "8d4c8693f9e04709a3072d56a9aae532"
format_stamp: "Formatted at 2026-09-26 23:36:22 on dist-test-slave-gwfx"
server_key: "cce8ee76ec18b5f7ead7c83a40b1c4b5"
server_key_iv: "a5b48b91b0e64c7dcfe1b498d48f0dce"
server_key_version: "encryptionkey@0"
I20260926 23:36:22.379492 22423 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260926 23:36:22.384936 22438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:36:22.386933 22423 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.003s
I20260926 23:36:22.387166 22423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "8d4c8693f9e04709a3072d56a9aae532"
format_stamp: "Formatted at 2026-09-26 23:36:22 on dist-test-slave-gwfx"
server_key: "cce8ee76ec18b5f7ead7c83a40b1c4b5"
server_key_iv: "a5b48b91b0e64c7dcfe1b498d48f0dce"
server_key_version: "encryptionkey@0"
I20260926 23:36:22.387424 22423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:36:22.394869 22423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:36:22.400342 22423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:36:22.400683 22423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:36:22.423084 22423 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43085
I20260926 23:36:22.423094 22489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43085 every 8 connection(s)
I20260926 23:36:22.424885 22423 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:36:22.429297 22490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:36:22.433583 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 22423
I20260926 23:36:22.433818 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:36:22.434398 388 external_mini_cluster.cc:1428] Setting key e6c2c45cc6329fddc0fde2106a9bee9f
I20260926 23:36:22.442165 22490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d4c8693f9e04709a3072d56a9aae532: Bootstrap starting.
I20260926 23:36:22.445577 22493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:22.436999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44563 (local address 127.0.97.62:43085)
0926 23:36:22.438025 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:22.438045 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:22.438099 (+ 54us) server_negotiation.cc:407] Connection header received
0926 23:36:22.440538 (+ 2439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:22.440572 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:22.441085 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:22.441557 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:22.443019 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:22.443027 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:22.443045 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:36:22.443068 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:22.443156 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:36:22.443188 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:22.443424 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:22.443564 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 23:36:22.443847 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":749,"thread_start_us":447,"threads_started":1}
I20260926 23:36:22.446720 22490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d4c8693f9e04709a3072d56a9aae532: Neither blocks nor log segments found. Creating new log.
I20260926 23:36:22.447971 22490 log.cc:824] T 00000000000000000000000000000000 P 8d4c8693f9e04709a3072d56a9aae532: Log is configured to *not* fsync() on all Append() calls
I20260926 23:36:22.450625 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:36:22.451471 22490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d4c8693f9e04709a3072d56a9aae532: No bootstrap required, opened a new log
I20260926 23:36:22.459573 22493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:22.456569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42731 (local address 127.0.97.62:43085)
0926 23:36:22.456818 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:22.456825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:22.457171 (+ 346us) server_negotiation.cc:407] Connection header received
0926 23:36:22.457348 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:22.457354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:22.457434 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:22.457610 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:22.458772 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:22.458779 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:22.458784 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:36:22.458806 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:22.458864 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:36:22.458884 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:22.459033 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:22.459168 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 23:36:22.459244 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260926 23:36:22.461875 22490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d4c8693f9e04709a3072d56a9aae532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d4c8693f9e04709a3072d56a9aae532" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43085 } }
I20260926 23:36:22.462317 22490 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d4c8693f9e04709a3072d56a9aae532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:36:22.462399 22490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d4c8693f9e04709a3072d56a9aae532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d4c8693f9e04709a3072d56a9aae532, State: Initialized, Role: FOLLOWER
I20260926 23:36:22.463176 22490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d4c8693f9e04709a3072d56a9aae532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d4c8693f9e04709a3072d56a9aae532" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43085 } }
I20260926 23:36:22.463434 22490 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d4c8693f9e04709a3072d56a9aae532 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:36:22.463577 22490 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d4c8693f9e04709a3072d56a9aae532 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:36:22.463778 22490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d4c8693f9e04709a3072d56a9aae532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:36:22.466236 22490 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d4c8693f9e04709a3072d56a9aae532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d4c8693f9e04709a3072d56a9aae532" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43085 } }
I20260926 23:36:22.466871 22490 leader_election.cc:302] T 00000000000000000000000000000000 P 8d4c8693f9e04709a3072d56a9aae532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d4c8693f9e04709a3072d56a9aae532; no voters:
I20260926 23:36:22.467602 22490 leader_election.cc:288] T 00000000000000000000000000000000 P 8d4c8693f9e04709a3072d56a9aae532 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:36:22.467891 22500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d4c8693f9e04709a3072d56a9aae532 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:36:22.469131 22500 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d4c8693f9e04709a3072d56a9aae532 [term 1 LEADER]: Becoming Leader. State: Replica: 8d4c8693f9e04709a3072d56a9aae532, State: Running, Role: LEADER
I20260926 23:36:22.469866 22500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d4c8693f9e04709a3072d56a9aae532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d4c8693f9e04709a3072d56a9aae532" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43085 } }
I20260926 23:36:22.470268 22490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d4c8693f9e04709a3072d56a9aae532 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:36:22.473527 22502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d4c8693f9e04709a3072d56a9aae532 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d4c8693f9e04709a3072d56a9aae532. Latest consensus state: current_term: 1 leader_uuid: "8d4c8693f9e04709a3072d56a9aae532" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d4c8693f9e04709a3072d56a9aae532" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43085 } } }
I20260926 23:36:22.474130 22502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d4c8693f9e04709a3072d56a9aae532 [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:22.474396 22501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d4c8693f9e04709a3072d56a9aae532 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d4c8693f9e04709a3072d56a9aae532" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d4c8693f9e04709a3072d56a9aae532" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43085 } } }
I20260926 23:36:22.474807 22501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d4c8693f9e04709a3072d56a9aae532 [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:22.476887 22509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:36:22.482937 22509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:36:22.495121 22509 catalog_manager.cc:1347] Generated new cluster ID: fa2894439e734e55ad4fa66591f3f350
I20260926 23:36:22.495309 22509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:36:22.524152 22509 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:36:22.524809 22509 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:36:22.540160 22509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d4c8693f9e04709a3072d56a9aae532: Generated new TSK 0
I20260926 23:36:22.541654 22509 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:36:22.556172 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 22423
W20260926 23:36:22.558269 22485 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44563
W20260926 23:36:22.559269 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 22423: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:36:22.559486 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 22423
2026-09-26T23:36:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2005 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:36:22 dist-test-slave-gwfx krb5kdc[22521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:36:22 dist-test-slave-gwfx krb5kdc[22521](info): set up 2 sockets
Sep 26 23:36:22 dist-test-slave-gwfx krb5kdc[22521](info): commencing operation
krb5kdc: starting...
W20260926 23:36:23.623507 388 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.003s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:36:23 dist-test-slave-gwfx krb5kdc[22521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:36:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:36:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:36:23.935331 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35091
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:36:24.331030 22534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:36:24.331475 22534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:36:24.331632 22534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:36:24.340024 22534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:36:24.340415 22534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:36:24.340678 22534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:36:24.340864 22534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:36:24.340940 22534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:36:24.352912 22534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:36:24.354372 22534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:36:24.355767 22534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:36:24.369928 22539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:24.369831 22542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:24.369851 22540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:36:24.370954 22534 server_base.cc:1034] running on GCE node
I20260926 23:36:24.372004 22534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:36:24.374035 22534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:36:24.375274 22534 hybrid_clock.cc:648] HybridClock initialized: now 1790465784375233 us; error 43 us; skew 500 ppm
I20260926 23:36:24.375702 22534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:36:24 dist-test-slave-gwfx krb5kdc[22521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:36:24.379608 22534 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:36:24.381613 22534 webserver.cc:460] Webserver started at http://127.0.97.62:35667/ using document root <none> and password file <none>
I20260926 23:36:24.382493 22534 fs_manager.cc:359] Metadata directory not provided
I20260926 23:36:24.382609 22534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:36:24.382972 22534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:36:24.386828 22534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "6923a98951014d0998e62cda6d17d98a"
format_stamp: "Formatted at 2026-09-26 23:36:24 on dist-test-slave-gwfx"
server_key: "504314a1932ae2f628271b8c9e8cfdbe"
server_key_iv: "524219db53fdddf6a76181f42beacd10"
server_key_version: "encryptionkey@0"
I20260926 23:36:24.387804 22534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "6923a98951014d0998e62cda6d17d98a"
format_stamp: "Formatted at 2026-09-26 23:36:24 on dist-test-slave-gwfx"
server_key: "504314a1932ae2f628271b8c9e8cfdbe"
server_key_iv: "524219db53fdddf6a76181f42beacd10"
server_key_version: "encryptionkey@0"
I20260926 23:36:24.394495 22534 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 23:36:24.399459 22549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:36:24.401468 22534 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 23:36:24.401693 22534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "6923a98951014d0998e62cda6d17d98a"
format_stamp: "Formatted at 2026-09-26 23:36:24 on dist-test-slave-gwfx"
server_key: "504314a1932ae2f628271b8c9e8cfdbe"
server_key_iv: "524219db53fdddf6a76181f42beacd10"
server_key_version: "encryptionkey@0"
I20260926 23:36:24.401938 22534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:36:24.448016 22534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:36:24.453207 22534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:36:24.453508 22534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:36:24.474799 22534 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35091
I20260926 23:36:24.474864 22600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35091 every 8 connection(s)
I20260926 23:36:24.476557 22534 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:36:24.478643 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 22534
I20260926 23:36:24.478892 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:36:24.479525 388 external_mini_cluster.cc:1428] Setting key 7a693e8bb900c8dc020d31a6b4a6d794
I20260926 23:36:24.482649 22601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 23:36:24 dist-test-slave-gwfx krb5kdc[22521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:36:24.498739 22601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6923a98951014d0998e62cda6d17d98a: Bootstrap starting.
I20260926 23:36:24.503785 22601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6923a98951014d0998e62cda6d17d98a: Neither blocks nor log segments found. Creating new log.
I20260926 23:36:24.505124 22601 log.cc:824] T 00000000000000000000000000000000 P 6923a98951014d0998e62cda6d17d98a: Log is configured to *not* fsync() on all Append() calls
I20260926 23:36:24.508917 22601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6923a98951014d0998e62cda6d17d98a: No bootstrap required, opened a new log
I20260926 23:36:24.508947 22604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:24.481849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39163 (local address 127.0.97.62:35091)
0926 23:36:24.482860 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:24.482878 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:24.482938 (+ 60us) server_negotiation.cc:407] Connection header received
0926 23:36:24.484850 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:24.484878 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:24.485405 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:24.485851 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:24.487847 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:24.489452 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:24.491387 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:24.492743 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:24.492989 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:24.498975 (+ 5986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:24.498991 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:24.499021 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:24.499084 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:24.503891 (+ 4807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:24.504792 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:24.504805 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:24.504815 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:24.505031 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:24.505492 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:24.505498 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:24.505501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:24.506277 (+ 776us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:24.506716 (+ 439us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:24.507174 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 23:36:24.507736 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":435,"threads_started":1}
I20260926 23:36:24.513468 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:36:24.518237 22601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6923a98951014d0998e62cda6d17d98a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6923a98951014d0998e62cda6d17d98a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35091 } }
I20260926 23:36:24.518797 22601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6923a98951014d0998e62cda6d17d98a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:36:24.518911 22601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6923a98951014d0998e62cda6d17d98a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6923a98951014d0998e62cda6d17d98a, State: Initialized, Role: FOLLOWER
I20260926 23:36:24.519625 22601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6923a98951014d0998e62cda6d17d98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6923a98951014d0998e62cda6d17d98a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35091 } }
I20260926 23:36:24.519873 22601 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6923a98951014d0998e62cda6d17d98a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:36:24.520007 22601 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6923a98951014d0998e62cda6d17d98a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:36:24.520175 22601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6923a98951014d0998e62cda6d17d98a [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:36:24.522717 22601 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6923a98951014d0998e62cda6d17d98a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6923a98951014d0998e62cda6d17d98a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35091 } }
I20260926 23:36:24.523366 22601 leader_election.cc:302] T 00000000000000000000000000000000 P 6923a98951014d0998e62cda6d17d98a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6923a98951014d0998e62cda6d17d98a; no voters:
I20260926 23:36:24.524115 22601 leader_election.cc:288] T 00000000000000000000000000000000 P 6923a98951014d0998e62cda6d17d98a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:36:24.524319 22611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6923a98951014d0998e62cda6d17d98a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:36:24.526916 22611 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6923a98951014d0998e62cda6d17d98a [term 1 LEADER]: Becoming Leader. State: Replica: 6923a98951014d0998e62cda6d17d98a, State: Running, Role: LEADER
I20260926 23:36:24.526985 22601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6923a98951014d0998e62cda6d17d98a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:36:24.527644 22611 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6923a98951014d0998e62cda6d17d98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6923a98951014d0998e62cda6d17d98a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35091 } }
I20260926 23:36:24.531207 22613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6923a98951014d0998e62cda6d17d98a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6923a98951014d0998e62cda6d17d98a. Latest consensus state: current_term: 1 leader_uuid: "6923a98951014d0998e62cda6d17d98a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6923a98951014d0998e62cda6d17d98a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35091 } } }
I20260926 23:36:24.531823 22613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6923a98951014d0998e62cda6d17d98a [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:24.531847 22612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6923a98951014d0998e62cda6d17d98a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6923a98951014d0998e62cda6d17d98a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6923a98951014d0998e62cda6d17d98a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35091 } } }
I20260926 23:36:24.532245 22612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6923a98951014d0998e62cda6d17d98a [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:24.535538 22604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:24.518395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60687 (local address 127.0.97.62:35091)
0926 23:36:24.518603 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:24.518611 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:24.519024 (+ 413us) server_negotiation.cc:407] Connection header received
0926 23:36:24.519219 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:24.519224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:24.519303 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:24.519391 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:24.521061 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:24.521858 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:24.523258 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:24.523822 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:24.523987 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:24.526737 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:24.526745 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:24.526750 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:24.526810 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:24.530492 (+ 3682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:24.531655 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:24.531662 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:24.531669 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:24.531887 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:24.532418 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:24.532423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:24.532427 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:24.532954 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:24.534599 (+ 1645us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:24.534799 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 23:36:24.535084 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 23:36:24.543517 22620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:36:24.550037 22620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:36:24.561539 22620 catalog_manager.cc:1347] Generated new cluster ID: f1bd78a43f3e4561b891f918b22694af
I20260926 23:36:24.561698 22620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:36:24.586586 22620 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:36:24.588224 22620 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:36:24.601720 22620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6923a98951014d0998e62cda6d17d98a: Generated new TSK 0
I20260926 23:36:24.603042 22620 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:36:24.625089 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 22534
I20260926 23:36:24.626809 22596 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:36:24.731776 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 22534
2026-09-26T23:36:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2173 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:36:24 dist-test-slave-gwfx krb5kdc[22633](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:36:24 dist-test-slave-gwfx krb5kdc[22633](info): set up 2 sockets
Sep 26 23:36:24 dist-test-slave-gwfx krb5kdc[22633](info): commencing operation
krb5kdc: starting...
W20260926 23:36:25.795360 388 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:36:25 dist-test-slave-gwfx krb5kdc[22633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:36:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:36:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:36:26.110504 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40291
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:36:26.530884 22646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:36:26.531343 22646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:36:26.531482 22646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:36:26.539589 22646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:36:26.539762 22646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:36:26.539810 22646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:36:26.539878 22646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:36:26.539927 22646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:36:26.551291 22646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:36:26.553085 22646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:36:26.555096 22646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:36:26.570115 22651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:26.570097 22654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:26.570115 22652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:36:26.571128 22646 server_base.cc:1034] running on GCE node
I20260926 23:36:26.572207 22646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:36:26.574158 22646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:36:26.575429 22646 hybrid_clock.cc:648] HybridClock initialized: now 1790465786575363 us; error 54 us; skew 500 ppm
I20260926 23:36:26.575930 22646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:36:26 dist-test-slave-gwfx krb5kdc[22633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:36:26.580016 22646 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:36:26.582057 22646 webserver.cc:460] Webserver started at http://127.0.97.62:46373/ using document root <none> and password file <none>
I20260926 23:36:26.583057 22646 fs_manager.cc:359] Metadata directory not provided
I20260926 23:36:26.583209 22646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:36:26.583564 22646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:36:26.587478 22646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "b05f156d450c4b4898800079476aae5d"
format_stamp: "Formatted at 2026-09-26 23:36:26 on dist-test-slave-gwfx"
server_key: "40b40010ebdb3e0a4c3edfc0e6a17ce8"
server_key_iv: "0a5334165c1b18f81407d19a31ee1044"
server_key_version: "encryptionkey@0"
I20260926 23:36:26.588511 22646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "b05f156d450c4b4898800079476aae5d"
format_stamp: "Formatted at 2026-09-26 23:36:26 on dist-test-slave-gwfx"
server_key: "40b40010ebdb3e0a4c3edfc0e6a17ce8"
server_key_iv: "0a5334165c1b18f81407d19a31ee1044"
server_key_version: "encryptionkey@0"
I20260926 23:36:26.596011 22646 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260926 23:36:26.600935 22661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:36:26.603173 22646 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 23:36:26.603461 22646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "b05f156d450c4b4898800079476aae5d"
format_stamp: "Formatted at 2026-09-26 23:36:26 on dist-test-slave-gwfx"
server_key: "40b40010ebdb3e0a4c3edfc0e6a17ce8"
server_key_iv: "0a5334165c1b18f81407d19a31ee1044"
server_key_version: "encryptionkey@0"
I20260926 23:36:26.603726 22646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:36:26.640276 22646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:36:26.646482 22646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:36:26.646886 22646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:36:26.669752 22712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40291 every 8 connection(s)
I20260926 23:36:26.669711 22646 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40291
I20260926 23:36:26.672029 22646 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:36:26.675033 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 22646
I20260926 23:36:26.675246 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:36:26.675827 388 external_mini_cluster.cc:1428] Setting key 6a9e2a3ac1f114206614f5eacc8b56c2
I20260926 23:36:26.677311 22713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:36:26.693472 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:26.678314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49123 (local address 127.0.97.62:40291)
0926 23:36:26.679468 (+ 1154us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:26.679484 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:26.679547 (+ 63us) server_negotiation.cc:407] Connection header received
0926 23:36:26.681374 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:26.681404 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:26.682031 (+ 627us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:26.682537 (+ 506us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:26.683419 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:26.685186 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:26.687434 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:26.688330 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:26.688524 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:26.690151 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:26.690160 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:26.690182 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:36:26.691105 (+ 923us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:26.691226 (+ 121us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:36:26.691264 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:26.691395 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:26.691704 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 23:36:26.692087 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":862,"thread_start_us":551,"threads_started":1}
I20260926 23:36:26.693892 22713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b05f156d450c4b4898800079476aae5d: Bootstrap starting.
I20260926 23:36:26.697671 22713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b05f156d450c4b4898800079476aae5d: Neither blocks nor log segments found. Creating new log.
I20260926 23:36:26.698228 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:36:26.698954 22713 log.cc:824] T 00000000000000000000000000000000 P b05f156d450c4b4898800079476aae5d: Log is configured to *not* fsync() on all Append() calls
I20260926 23:36:26.702764 22713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b05f156d450c4b4898800079476aae5d: No bootstrap required, opened a new log
I20260926 23:36:26.711944 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:26.703873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57023 (local address 127.0.97.62:40291)
0926 23:36:26.704130 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:26.704137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:26.704608 (+ 471us) server_negotiation.cc:407] Connection header received
0926 23:36:26.704884 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:26.704889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:26.704982 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:26.705097 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:26.705799 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:26.706736 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:26.708791 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:26.709487 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:26.709613 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:26.711034 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:26.711039 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:26.711043 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:36:26.711069 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:26.711123 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:36:26.711144 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:26.711226 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:26.711409 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 23:36:26.711509 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260926 23:36:26.713938 22713 raft_consensus.cc:348] T 00000000000000000000000000000000 P b05f156d450c4b4898800079476aae5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b05f156d450c4b4898800079476aae5d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40291 } }
I20260926 23:36:26.714401 22713 raft_consensus.cc:374] T 00000000000000000000000000000000 P b05f156d450c4b4898800079476aae5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:36:26.714525 22713 raft_consensus.cc:729] T 00000000000000000000000000000000 P b05f156d450c4b4898800079476aae5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b05f156d450c4b4898800079476aae5d, State: Initialized, Role: FOLLOWER
I20260926 23:36:26.715200 22713 consensus_queue.cc:260] T 00000000000000000000000000000000 P b05f156d450c4b4898800079476aae5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b05f156d450c4b4898800079476aae5d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40291 } }
I20260926 23:36:26.715454 22713 raft_consensus.cc:388] T 00000000000000000000000000000000 P b05f156d450c4b4898800079476aae5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:36:26.715591 22713 raft_consensus.cc:482] T 00000000000000000000000000000000 P b05f156d450c4b4898800079476aae5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:36:26.715775 22713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b05f156d450c4b4898800079476aae5d [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:36:26.718436 22713 raft_consensus.cc:504] T 00000000000000000000000000000000 P b05f156d450c4b4898800079476aae5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b05f156d450c4b4898800079476aae5d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40291 } }
I20260926 23:36:26.719126 22713 leader_election.cc:302] T 00000000000000000000000000000000 P b05f156d450c4b4898800079476aae5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b05f156d450c4b4898800079476aae5d; no voters:
I20260926 23:36:26.719887 22713 leader_election.cc:288] T 00000000000000000000000000000000 P b05f156d450c4b4898800079476aae5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:36:26.720093 22723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b05f156d450c4b4898800079476aae5d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:36:26.721271 22723 raft_consensus.cc:686] T 00000000000000000000000000000000 P b05f156d450c4b4898800079476aae5d [term 1 LEADER]: Becoming Leader. State: Replica: b05f156d450c4b4898800079476aae5d, State: Running, Role: LEADER
I20260926 23:36:26.721949 22723 consensus_queue.cc:237] T 00000000000000000000000000000000 P b05f156d450c4b4898800079476aae5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b05f156d450c4b4898800079476aae5d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40291 } }
I20260926 23:36:26.722391 22713 sys_catalog.cc:564] T 00000000000000000000000000000000 P b05f156d450c4b4898800079476aae5d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:36:26.726064 22724 sys_catalog.cc:455] T 00000000000000000000000000000000 P b05f156d450c4b4898800079476aae5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b05f156d450c4b4898800079476aae5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b05f156d450c4b4898800079476aae5d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40291 } } }
I20260926 23:36:26.726575 22724 sys_catalog.cc:458] T 00000000000000000000000000000000 P b05f156d450c4b4898800079476aae5d [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:26.725272 22725 sys_catalog.cc:455] T 00000000000000000000000000000000 P b05f156d450c4b4898800079476aae5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b05f156d450c4b4898800079476aae5d. Latest consensus state: current_term: 1 leader_uuid: "b05f156d450c4b4898800079476aae5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b05f156d450c4b4898800079476aae5d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40291 } } }
I20260926 23:36:26.727083 22725 sys_catalog.cc:458] T 00000000000000000000000000000000 P b05f156d450c4b4898800079476aae5d [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:26.729921 22731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:36:26.736937 22731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:36:26.748178 22731 catalog_manager.cc:1347] Generated new cluster ID: 17a7b3bb1b244f648884badca52515ab
I20260926 23:36:26.748381 22731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:36:26.764331 22731 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:36:26.766475 22731 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:36:26.776062 22731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b05f156d450c4b4898800079476aae5d: Generated new TSK 0
I20260926 23:36:26.777467 22731 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:36:26.794339 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 22646
W20260926 23:36:26.795697 22708 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49123
W20260926 23:36:26.796481 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 22646: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:36:26.796620 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 22646
2026-09-26T23:36:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2064 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:36:26 dist-test-slave-gwfx krb5kdc[22744](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:36:26 dist-test-slave-gwfx krb5kdc[22744](info): set up 2 sockets
Sep 26 23:36:26 dist-test-slave-gwfx krb5kdc[22744](info): commencing operation
krb5kdc: starting...
W20260926 23:36:27.851428 388 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:36:28 dist-test-slave-gwfx krb5kdc[22744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:36:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:36:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:36:28.168126 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42251
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:36:28.563472 22757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:36:28.563879 22757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:36:28.564014 22757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:36:28.572456 22757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:36:28.572628 22757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:36:28.572708 22757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:36:28.572803 22757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:36:28.572870 22757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:36:28.584583 22757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:36:28.586359 22757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:36:28.588171 22757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:36:28.602835 22762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:28.602914 22763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:28.602975 22765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:36:28.603587 22757 server_base.cc:1034] running on GCE node
I20260926 23:36:28.604691 22757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:36:28.606400 22757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:36:28.607658 22757 hybrid_clock.cc:648] HybridClock initialized: now 1790465788607590 us; error 58 us; skew 500 ppm
I20260926 23:36:28.608098 22757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:36:28 dist-test-slave-gwfx krb5kdc[22744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:36:28.615581 22757 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:36:28.617513 22757 webserver.cc:460] Webserver started at http://127.0.97.62:39647/ using document root <none> and password file <none>
I20260926 23:36:28.618286 22757 fs_manager.cc:359] Metadata directory not provided
I20260926 23:36:28.618402 22757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:36:28.618768 22757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:36:28.622027 22757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "cc1780f5df27469d840d3bff7f1b9785"
format_stamp: "Formatted at 2026-09-26 23:36:28 on dist-test-slave-gwfx"
server_key: "6e694f4b758527e8d7587db9de9ed8e1"
server_key_iv: "99fc8dcffc12a47c921ed3ac30392f05"
server_key_version: "encryptionkey@0"
I20260926 23:36:28.622866 22757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "cc1780f5df27469d840d3bff7f1b9785"
format_stamp: "Formatted at 2026-09-26 23:36:28 on dist-test-slave-gwfx"
server_key: "6e694f4b758527e8d7587db9de9ed8e1"
server_key_iv: "99fc8dcffc12a47c921ed3ac30392f05"
server_key_version: "encryptionkey@0"
I20260926 23:36:28.629086 22757 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 23:36:28.633801 22772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:36:28.635979 22757 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 23:36:28.636243 22757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "cc1780f5df27469d840d3bff7f1b9785"
format_stamp: "Formatted at 2026-09-26 23:36:28 on dist-test-slave-gwfx"
server_key: "6e694f4b758527e8d7587db9de9ed8e1"
server_key_iv: "99fc8dcffc12a47c921ed3ac30392f05"
server_key_version: "encryptionkey@0"
I20260926 23:36:28.636480 22757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:36:28.643364 22757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:36:28.648350 22757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:36:28.648676 22757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:36:28.670879 22757 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42251
I20260926 23:36:28.670992 22823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42251 every 8 connection(s)
I20260926 23:36:28.672444 22757 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:36:28.677489 22824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:36:28.680320 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 22757
I20260926 23:36:28.680486 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:36:28.680948 388 external_mini_cluster.cc:1428] Setting key 444365615faf0dc2fd725793f4b4f2cb
I20260926 23:36:28.690804 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:28.683409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43879 (local address 127.0.97.62:42251)
0926 23:36:28.684367 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:28.684381 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:28.684431 (+ 50us) server_negotiation.cc:407] Connection header received
0926 23:36:28.686426 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:28.686458 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:28.686939 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:28.687334 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:28.688743 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:28.688751 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:28.688766 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:36:28.688808 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:28.688888 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:36:28.688921 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:28.689167 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:28.689299 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 23:36:28.689542 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":436,"threads_started":1}
I20260926 23:36:28.692598 22824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc1780f5df27469d840d3bff7f1b9785: Bootstrap starting.
I20260926 23:36:28.695294 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:36:28.697024 22824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc1780f5df27469d840d3bff7f1b9785: Neither blocks nor log segments found. Creating new log.
I20260926 23:36:28.698757 22824 log.cc:824] T 00000000000000000000000000000000 P cc1780f5df27469d840d3bff7f1b9785: Log is configured to *not* fsync() on all Append() calls
I20260926 23:36:28.702937 22824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc1780f5df27469d840d3bff7f1b9785: No bootstrap required, opened a new log
I20260926 23:36:28.703899 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:28.701179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58427 (local address 127.0.97.62:42251)
0926 23:36:28.701417 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:28.701425 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:28.701926 (+ 501us) server_negotiation.cc:407] Connection header received
0926 23:36:28.702106 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:28.702110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:28.702166 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:28.702251 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:28.703234 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:28.703239 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:28.703243 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 23:36:28.703259 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:28.703308 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 23:36:28.703330 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:28.703442 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:28.703538 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 23:36:28.703605 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 23:36:28.711877 22824 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc1780f5df27469d840d3bff7f1b9785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc1780f5df27469d840d3bff7f1b9785" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42251 } }
I20260926 23:36:28.712502 22824 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc1780f5df27469d840d3bff7f1b9785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 23:36:28.712607 22824 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc1780f5df27469d840d3bff7f1b9785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc1780f5df27469d840d3bff7f1b9785, State: Initialized, Role: FOLLOWER
I20260926 23:36:28.713337 22824 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc1780f5df27469d840d3bff7f1b9785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc1780f5df27469d840d3bff7f1b9785" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42251 } }
I20260926 23:36:28.713606 22824 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc1780f5df27469d840d3bff7f1b9785 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 23:36:28.713753 22824 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc1780f5df27469d840d3bff7f1b9785 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 23:36:28.713927 22824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc1780f5df27469d840d3bff7f1b9785 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:36:28.716856 22824 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc1780f5df27469d840d3bff7f1b9785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc1780f5df27469d840d3bff7f1b9785" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42251 } }
I20260926 23:36:28.717626 22824 leader_election.cc:302] T 00000000000000000000000000000000 P cc1780f5df27469d840d3bff7f1b9785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc1780f5df27469d840d3bff7f1b9785; no voters:
I20260926 23:36:28.718369 22824 leader_election.cc:288] T 00000000000000000000000000000000 P cc1780f5df27469d840d3bff7f1b9785 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 23:36:28.718566 22834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc1780f5df27469d840d3bff7f1b9785 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:36:28.719429 22834 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc1780f5df27469d840d3bff7f1b9785 [term 1 LEADER]: Becoming Leader. State: Replica: cc1780f5df27469d840d3bff7f1b9785, State: Running, Role: LEADER
I20260926 23:36:28.720124 22834 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc1780f5df27469d840d3bff7f1b9785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc1780f5df27469d840d3bff7f1b9785" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42251 } }
I20260926 23:36:28.721028 22824 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc1780f5df27469d840d3bff7f1b9785 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:36:28.723181 22836 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc1780f5df27469d840d3bff7f1b9785 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc1780f5df27469d840d3bff7f1b9785. Latest consensus state: current_term: 1 leader_uuid: "cc1780f5df27469d840d3bff7f1b9785" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc1780f5df27469d840d3bff7f1b9785" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42251 } } }
I20260926 23:36:28.723681 22836 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc1780f5df27469d840d3bff7f1b9785 [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:28.724495 22835 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc1780f5df27469d840d3bff7f1b9785 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc1780f5df27469d840d3bff7f1b9785" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc1780f5df27469d840d3bff7f1b9785" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42251 } } }
I20260926 23:36:28.724902 22835 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc1780f5df27469d840d3bff7f1b9785 [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:28.724934 22840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:36:28.733556 22840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:36:28.746197 22840 catalog_manager.cc:1347] Generated new cluster ID: 15c671331f834a70b8ba38de0f6500c6
I20260926 23:36:28.746367 22840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:36:28.757187 22840 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:36:28.757818 22840 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:36:28.773407 22840 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc1780f5df27469d840d3bff7f1b9785: Generated new TSK 0
I20260926 23:36:28.774617 22840 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 23:36:28.792514 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 22757
W20260926 23:36:28.794191 22819 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43879
W20260926 23:36:28.795213 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 22757: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 23:36:28.795385 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 22757
2026-09-26T23:36:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1995 ms)
[----------] 18 tests from AuthTokenIssuingTest (37422 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:36:28 dist-test-slave-gwfx krb5kdc[22855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:36:28 dist-test-slave-gwfx krb5kdc[22855](info): set up 2 sockets
Sep 26 23:36:28 dist-test-slave-gwfx krb5kdc[22855](info): commencing operation
krb5kdc: starting...
W20260926 23:36:29.859694 388 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:36:30 dist-test-slave-gwfx krb5kdc[22855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:36:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:36:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:36:30.187692 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42323
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42323,127.0.97.61:42737,127.0.97.60:36151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:36:30.586396 22868 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 23:36:30.586918 22868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:36:30.587026 22868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:36:30.587167 22868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:36:30.595832 22868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:36:30.595980 22868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:36:30.596045 22868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:36:30.596122 22868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:36:30.596223 22868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:36:30.608582 22868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42323,127.0.97.61:42737,127.0.97.60:36151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:36:30.610437 22868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:36:30.612139 22868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:36:30.626542 22873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:30.626474 22876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:30.626516 22874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:36:30.627595 22868 server_base.cc:1034] running on GCE node
I20260926 23:36:30.628782 22868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:36:30.630915 22868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:36:30.632262 22868 hybrid_clock.cc:648] HybridClock initialized: now 1790465790632144 us; error 89 us; skew 500 ppm
I20260926 23:36:30.632736 22868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:36:30 dist-test-slave-gwfx krb5kdc[22855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:36:30.636899 22868 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:36:30.639001 22868 webserver.cc:460] Webserver started at http://127.0.97.62:44985/ using document root <none> and password file <none>
I20260926 23:36:30.639993 22868 fs_manager.cc:359] Metadata directory not provided
I20260926 23:36:30.640116 22868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:36:30.640523 22868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:36:30.644481 22868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "648d9ba6e75e49118bdc43157fc10425"
format_stamp: "Formatted at 2026-09-26 23:36:30 on dist-test-slave-gwfx"
server_key: "05f2b8235ee8d5158b43082bc86b15b0"
server_key_iv: "07b4ae07250a72d0718a219369aeb44a"
server_key_version: "encryptionkey@0"
I20260926 23:36:30.645534 22868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "648d9ba6e75e49118bdc43157fc10425"
format_stamp: "Formatted at 2026-09-26 23:36:30 on dist-test-slave-gwfx"
server_key: "05f2b8235ee8d5158b43082bc86b15b0"
server_key_iv: "07b4ae07250a72d0718a219369aeb44a"
server_key_version: "encryptionkey@0"
I20260926 23:36:30.652727 22868 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260926 23:36:30.657681 22883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:36:30.659832 22868 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 23:36:30.660090 22868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "648d9ba6e75e49118bdc43157fc10425"
format_stamp: "Formatted at 2026-09-26 23:36:30 on dist-test-slave-gwfx"
server_key: "05f2b8235ee8d5158b43082bc86b15b0"
server_key_iv: "07b4ae07250a72d0718a219369aeb44a"
server_key_version: "encryptionkey@0"
I20260926 23:36:30.660388 22868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:36:30.678334 22868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:36:30.684706 22868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:36:30.685096 22868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:36:30.707841 22868 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42323
I20260926 23:36:30.707845 22934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42323 every 8 connection(s)
I20260926 23:36:30.709599 22868 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:36:30.712710 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 22868
I20260926 23:36:30.712972 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:36:30.713876 388 external_mini_cluster.cc:1428] Setting key 2fd8920974c2ff3fa1692201e2413f9a
I20260926 23:36:30.715358 22935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:36:30.726712 22935 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42323 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:36:30 dist-test-slave-gwfx krb5kdc[22855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260926 23:36:30.757174 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:30.731599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42323 (local address 127.0.0.1:48964)
0926 23:36:30.733157 (+ 1558us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:30.733219 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 23:36:30.735459 (+ 2240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:36:30.736837 (+ 1378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:36:30.736844 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:36:30.737606 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:36:30.738090 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:30.738112 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:30.740476 (+ 2364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:30.740481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:30.741840 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:30.741858 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:30.743288 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:30.743295 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:30.743484 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:30.744281 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:36:30.744309 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:36:30.747730 (+ 3421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:36:30.751769 (+ 4039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:30.751783 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:30.751796 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:30.752370 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:30.752940 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:30.752946 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:30.752950 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:30.753188 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:30.754487 (+ 1299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:36:30.754495 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:36:30.754815 (+ 320us) client_negotiation.cc:769] Sending connection context
0926 23:36:30.755094 (+ 279us) client_negotiation.cc:240] Negotiation successful
0926 23:36:30.755714 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1238,"mutex_wait_us":1,"spinlock_wait_cycles":21888,"thread_start_us":942,"threads_started":1}
I20260926 23:36:30.757225 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:30.731599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48964 (local address 127.0.97.62:42323)
0926 23:36:30.732680 (+ 1081us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:30.732700 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:30.733388 (+ 688us) server_negotiation.cc:407] Connection header received
0926 23:36:30.736016 (+ 2628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:30.736025 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:30.736455 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:30.736750 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:30.738355 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:30.740278 (+ 1923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:30.742018 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:30.742998 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:30.743209 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:30.748078 (+ 4869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:30.748091 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:30.748115 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:30.748159 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:30.751567 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:30.752529 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:30.752538 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:30.752540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:30.752744 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:30.753309 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:30.753314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:30.753318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:30.754279 (+ 961us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:30.754532 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:30.755289 (+ 757us) server_negotiation.cc:299] Negotiation successful
0926 23:36:30.755869 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":284,"server-negotiator.queue_time_us":775,"thread_start_us":463,"threads_started":1}
I20260926 23:36:30.761318 22935 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42737 } has no permanent_uuid. Determining permanent_uuid...
I20260926 23:36:30.762785 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:30.762159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42737 (local address 127.0.0.1:50662)
0926 23:36:30.762419 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:30.762555 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260926 23:36:30.763109 22887 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42737: connect: Connection refused (error 111)
I20260926 23:36:30.765267 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:30.764610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42737 (local address 127.0.0.1:50668)
0926 23:36:30.764809 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:30.764911 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260926 23:36:30.765528 22935 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42737: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42737: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260926 23:36:30.795313 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:42737
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42323,127.0.97.61:42737,127.0.97.60:36151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:36:30.815002 22935 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42737 } attempt: 1
I20260926 23:36:30.816681 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:30.816000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42737 (local address 127.0.0.1:50684)
0926 23:36:30.816275 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:30.816431 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260926 23:36:30.818264 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:30.817703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42737 (local address 127.0.0.1:50688)
0926 23:36:30.817919 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:30.818032 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 23:36:30.818562 22935 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42737: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42737: connect: Connection refused (error 111)
I20260926 23:36:30.886906 22935 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42737 } attempt: 2
I20260926 23:36:30.888747 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:30.887954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42737 (local address 127.0.0.1:50692)
0926 23:36:30.888333 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:30.888507 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260926 23:36:30.890329 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:30.889734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42737 (local address 127.0.0.1:50694)
0926 23:36:30.889972 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:30.890092 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260926 23:36:30.890591 22935 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42737: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42737: connect: Connection refused (error 111)
I20260926 23:36:30.981974 22935 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42737 } attempt: 3
I20260926 23:36:30.983566 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:30.982913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42737 (local address 127.0.0.1:50706)
0926 23:36:30.983183 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:30.983330 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 23:36:30.984858 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:30.984402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42737 (local address 127.0.0.1:50720)
0926 23:36:30.984558 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:30.984663 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 23:36:30.985088 22935 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42737: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42737: connect: Connection refused (error 111)
I20260926 23:36:31.128386 22935 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42737 } attempt: 4
I20260926 23:36:31.130105 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.129347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42737 (local address 127.0.0.1:50722)
0926 23:36:31.129728 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:31.129877 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
I20260926 23:36:31.131660 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.131109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42737 (local address 127.0.0.1:50730)
0926 23:36:31.131313 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:31.131446 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 23:36:31.131943 22935 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42737: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42737: connect: Connection refused (error 111)
W20260926 23:36:31.205850 22942 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 23:36:31.206311 22942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:36:31.206390 22942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:36:31.206482 22942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:36:31.214854 22942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:36:31.215001 22942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:36:31.215062 22942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:36:31.215108 22942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:36:31.215189 22942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:36:31.226636 22942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42323,127.0.97.61:42737,127.0.97.60:36151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:42737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:36:31.228451 22942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:36:31.230103 22942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:36:31.244403 22947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:31.244464 22948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:31.244596 22950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:36:31.245326 22942 server_base.cc:1034] running on GCE node
I20260926 23:36:31.246351 22942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:36:31.248404 22942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:36:31.249691 22942 hybrid_clock.cc:648] HybridClock initialized: now 1790465791249582 us; error 75 us; skew 500 ppm
I20260926 23:36:31.250111 22942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:36:31 dist-test-slave-gwfx krb5kdc[22855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:36:31.253999 22942 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260926 23:36:31.255944 22942 webserver.cc:460] Webserver started at http://127.0.97.61:38807/ using document root <none> and password file <none>
I20260926 23:36:31.256832 22942 fs_manager.cc:359] Metadata directory not provided
I20260926 23:36:31.256935 22942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:36:31.257272 22942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:36:31.261293 22942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-1/data/instance:
uuid: "17915c1992204da5a276d42a6ef84aee"
format_stamp: "Formatted at 2026-09-26 23:36:31 on dist-test-slave-gwfx"
server_key: "8d3ae2da272be27896d8aad20a981af8"
server_key_iv: "cbc471b8c1adde7cc788304e7137d13d"
server_key_version: "encryptionkey@0"
I20260926 23:36:31.262274 22942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-1/wal/instance:
uuid: "17915c1992204da5a276d42a6ef84aee"
format_stamp: "Formatted at 2026-09-26 23:36:31 on dist-test-slave-gwfx"
server_key: "8d3ae2da272be27896d8aad20a981af8"
server_key_iv: "cbc471b8c1adde7cc788304e7137d13d"
server_key_version: "encryptionkey@0"
I20260926 23:36:31.269361 22942 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260926 23:36:31.274255 22957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:36:31.276265 22942 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 23:36:31.276526 22942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-1/wal
uuid: "17915c1992204da5a276d42a6ef84aee"
format_stamp: "Formatted at 2026-09-26 23:36:31 on dist-test-slave-gwfx"
server_key: "8d3ae2da272be27896d8aad20a981af8"
server_key_iv: "cbc471b8c1adde7cc788304e7137d13d"
server_key_version: "encryptionkey@0"
I20260926 23:36:31.276835 22942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:36:31.297847 22942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:36:31.303810 22942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:36:31.304163 22942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:36:31.328572 22942 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:42737
I20260926 23:36:31.328636 23008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:42737 every 8 connection(s)
I20260926 23:36:31.330881 22942 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-1/data/info.pb
I20260926 23:36:31.336431 23009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:36:31.340430 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 22942
I20260926 23:36:31.340662 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-1/wal/instance
I20260926 23:36:31.341172 388 external_mini_cluster.cc:1428] Setting key a710c8f00d01c852bcf280f820b230d2
I20260926 23:36:31.345911 23009 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42323 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:36:31 dist-test-slave-gwfx krb5kdc[22855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260926 23:36:31.373121 23011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.349660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48968 (local address 127.0.97.62:42323)
0926 23:36:31.350285 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:31.350291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:31.351999 (+ 1708us) server_negotiation.cc:407] Connection header received
0926 23:36:31.354958 (+ 2959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:31.354976 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:31.355114 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:31.355239 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:31.357492 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:31.358315 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:31.360178 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:31.360626 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:31.360752 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:31.365546 (+ 4794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:31.365553 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:31.365557 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:31.365622 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:31.369009 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:31.370063 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:31.370069 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:31.370071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:31.370265 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:31.370824 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:31.370830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:31.370834 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:31.371191 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:31.371369 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:31.372335 (+ 966us) server_negotiation.cc:299] Negotiation successful
0926 23:36:31.372650 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":352,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260926 23:36:31.374457 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.350602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42323 (local address 127.0.0.1:48968)
0926 23:36:31.351770 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:31.351829 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 23:36:31.354491 (+ 2662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:36:31.355630 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:36:31.355651 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:36:31.356381 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:36:31.357199 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:31.357227 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:31.358513 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:31.358518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:31.359998 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:31.360015 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:31.360902 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:31.360908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:31.361184 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:31.362045 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:36:31.362075 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:36:31.365203 (+ 3128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:36:31.369260 (+ 4057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:31.369272 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:31.369295 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:31.369886 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:31.370400 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:31.370404 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:31.370407 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:31.370656 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:31.371346 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:36:31.371356 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:36:31.371881 (+ 525us) client_negotiation.cc:769] Sending connection context
0926 23:36:31.372182 (+ 301us) client_negotiation.cc:240] Negotiation successful
0926 23:36:31.372910 (+ 728us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"thread_start_us":461,"threads_started":1}
I20260926 23:36:31.377049 23009 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42737 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:36:31 dist-test-slave-gwfx krb5kdc[22855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 23:36:31.397148 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.378185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42737 (local address 127.0.0.1:50736)
0926 23:36:31.378556 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:31.378585 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 23:36:31.378775 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:36:31.380340 (+ 1565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:36:31.380345 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:36:31.380848 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:36:31.381027 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:31.381037 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:31.383249 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:31.383255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:31.384225 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:31.384241 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:31.385570 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:31.385580 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:31.385782 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:31.386510 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:36:31.386532 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:36:31.389416 (+ 2884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:36:31.394149 (+ 4733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:31.394155 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:31.394158 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:31.394621 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:31.395257 (+ 636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:31.395261 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:31.395264 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:31.395420 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:31.396255 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:36:31.396259 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:36:31.396395 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 23:36:31.396517 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 23:36:31.396832 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260926 23:36:31.397459 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.378736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50736 (local address 127.0.97.61:42737)
0926 23:36:31.379604 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:31.379614 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:31.379653 (+ 39us) server_negotiation.cc:407] Connection header received
0926 23:36:31.379733 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:31.379745 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:31.380109 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:31.380319 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:31.381218 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:31.382995 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:31.384397 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:31.385362 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:31.385513 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:31.389752 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:31.389779 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:31.389805 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:31.389951 (+ 146us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:31.393919 (+ 3968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:31.394889 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:31.394902 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:31.394905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:31.395095 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:31.395529 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:31.395533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:31.395536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:31.396079 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:31.396321 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:31.396729 (+ 408us) server_negotiation.cc:299] Negotiation successful
0926 23:36:31.397042 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":440,"threads_started":1}
I20260926 23:36:31.400918 23009 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36151 } has no permanent_uuid. Determining permanent_uuid...
I20260926 23:36:31.402514 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.401875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36151 (local address 127.0.0.1:42410)
0926 23:36:31.402133 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:31.402309 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260926 23:36:31.402889 22961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36151: connect: Connection refused (error 111)
I20260926 23:36:31.404932 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.404475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36151 (local address 127.0.0.1:42414)
0926 23:36:31.404681 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:31.404780 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 23:36:31.405169 23009 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36151: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36151: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260926 23:36:31.419144 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:36151
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42323,127.0.97.61:42737,127.0.97.60:36151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:36:31.431237 22935 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42737 } attempt: 5
Sep 26 23:36:31 dist-test-slave-gwfx krb5kdc[22855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 23:36:31.448366 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.432396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50738 (local address 127.0.97.61:42737)
0926 23:36:31.432575 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:31.432580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:31.432602 (+ 22us) server_negotiation.cc:407] Connection header received
0926 23:36:31.432669 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:31.432674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:31.432769 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:31.432858 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:31.433937 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:31.434951 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:31.436636 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:31.437146 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:31.437258 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:31.441536 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:31.441543 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:31.441547 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:31.441600 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:31.444568 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:31.445583 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:31.445589 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:31.445592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:31.445769 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:31.446292 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:31.446298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:31.446300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:31.446789 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:31.446959 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:31.447745 (+ 786us) server_negotiation.cc:299] Negotiation successful
0926 23:36:31.448063 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 23:36:31.448403 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.432087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42737 (local address 127.0.0.1:50738)
0926 23:36:31.432354 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:31.432378 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:36:31.432525 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:36:31.432916 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:36:31.432921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:36:31.433513 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:36:31.433774 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:31.433788 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:31.435124 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:31.435130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:31.436424 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:31.436443 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:31.437336 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:31.437342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:31.437518 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:31.438327 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:36:31.438353 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:36:31.441227 (+ 2874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:36:31.444768 (+ 3541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:31.444776 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:31.444780 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:31.445412 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:31.445897 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:31.445903 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:31.445907 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:31.446107 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:31.446940 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:36:31.446946 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:36:31.447280 (+ 334us) client_negotiation.cc:769] Sending connection context
0926 23:36:31.447493 (+ 213us) client_negotiation.cc:240] Negotiation successful
0926 23:36:31.447960 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 23:36:31.450199 22935 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36151 } has no permanent_uuid. Determining permanent_uuid...
I20260926 23:36:31.451702 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.451136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36151 (local address 127.0.0.1:42420)
0926 23:36:31.451370 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:31.451483 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 23:36:31.452857 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.452457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36151 (local address 127.0.0.1:42430)
0926 23:36:31.452631 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:31.452731 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260926 23:36:31.453081 22935 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36151: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36151: connect: Connection refused (error 111)
I20260926 23:36:31.454510 23009 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36151 } attempt: 1
I20260926 23:36:31.455632 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.455180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36151 (local address 127.0.0.1:42442)
0926 23:36:31.455379 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:31.455472 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 23:36:31.456741 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.456370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36151 (local address 127.0.0.1:42448)
0926 23:36:31.456517 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:31.456614 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 23:36:31.456972 23009 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36151: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36151: connect: Connection refused (error 111)
I20260926 23:36:31.504343 22935 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36151 } attempt: 1
I20260926 23:36:31.505940 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.505332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36151 (local address 127.0.0.1:42458)
0926 23:36:31.505574 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:31.505719 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 23:36:31.507150 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.506748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36151 (local address 127.0.0.1:42460)
0926 23:36:31.506915 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:31.507004 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 23:36:31.507396 22935 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36151: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36151: connect: Connection refused (error 111)
I20260926 23:36:31.525239 23009 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36151 } attempt: 2
I20260926 23:36:31.526824 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.526158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36151 (local address 127.0.0.1:42464)
0926 23:36:31.526429 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:31.526555 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260926 23:36:31.527913 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.527559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36151 (local address 127.0.0.1:42472)
0926 23:36:31.527702 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:31.527785 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 23:36:31.528137 23009 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36151: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36151: connect: Connection refused (error 111)
I20260926 23:36:31.575677 22935 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36151 } attempt: 2
I20260926 23:36:31.577483 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.576730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36151 (local address 127.0.0.1:42482)
0926 23:36:31.577053 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:31.577230 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260926 23:36:31.578730 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.578288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36151 (local address 127.0.0.1:42496)
0926 23:36:31.578451 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:31.578531 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 23:36:31.578971 22935 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36151: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36151: connect: Connection refused (error 111)
I20260926 23:36:31.619400 23009 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36151 } attempt: 3
I20260926 23:36:31.620947 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.620352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36151 (local address 127.0.0.1:42500)
0926 23:36:31.620631 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:31.620746 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260926 23:36:31.622164 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.621689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36151 (local address 127.0.0.1:42502)
0926 23:36:31.621943 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:31.622027 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260926 23:36:31.622455 23009 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36151: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36151: connect: Connection refused (error 111)
I20260926 23:36:31.685261 22935 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36151 } attempt: 3
I20260926 23:36:31.687052 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.686340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36151 (local address 127.0.0.1:42518)
0926 23:36:31.686645 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:31.686813 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260926 23:36:31.688338 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.687922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36151 (local address 127.0.0.1:42534)
0926 23:36:31.688092 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:31.688188 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 23:36:31.688591 22935 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36151: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36151: connect: Connection refused (error 111)
I20260926 23:36:31.765709 23009 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36151 } attempt: 4
I20260926 23:36:31.767163 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.766564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36151 (local address 127.0.0.1:42548)
0926 23:36:31.766874 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:31.766977 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260926 23:36:31.768151 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.767807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36151 (local address 127.0.0.1:42556)
0926 23:36:31.767960 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:31.768033 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 23:36:31.768349 23009 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36151: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36151: connect: Connection refused (error 111)
W20260926 23:36:31.825096 23017 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 23:36:31.825587 23017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:36:31.825690 23017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:36:31.825810 23017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:36:31.834458 23017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:36:31.834609 23017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:36:31.834736 23017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:36:31.834815 23017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:36:31.834908 23017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:36:31.846638 23017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42323,127.0.97.61:42737,127.0.97.60:36151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:36151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:36:31.848460 23017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:36:31.850174 23017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:36:31.864534 23025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:31.864583 23023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:31.864791 23022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:36:31.865329 23017 server_base.cc:1034] running on GCE node
I20260926 23:36:31.865808 22935 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36151 } attempt: 4
I20260926 23:36:31.866577 23017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:36:31.867165 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.866571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36151 (local address 127.0.0.1:42560)
0926 23:36:31.866824 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:31.866936 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 23:36:31.868232 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.867896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36151 (local address 127.0.0.1:42568)
0926 23:36:31.868023 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:31.868092 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 23:36:31.868476 22935 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36151: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36151: connect: Connection refused (error 111)
I20260926 23:36:31.868834 23017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:36:31.870081 23017 hybrid_clock.cc:648] HybridClock initialized: now 1790465791870001 us; error 45 us; skew 500 ppm
I20260926 23:36:31.870565 23017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:36:31 dist-test-slave-gwfx krb5kdc[22855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:36:31.874557 23017 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260926 23:36:31.876555 23017 webserver.cc:460] Webserver started at http://127.0.97.60:45783/ using document root <none> and password file <none>
I20260926 23:36:31.877432 23017 fs_manager.cc:359] Metadata directory not provided
I20260926 23:36:31.877545 23017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:36:31.877918 23017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:36:31.881903 23017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-2/data/instance:
uuid: "12215fce93a24704886bca4459035e48"
format_stamp: "Formatted at 2026-09-26 23:36:31 on dist-test-slave-gwfx"
server_key: "4a827480c13bf3e625e9c8cefd1e1848"
server_key_iv: "35f8458a24625a750fa226f34b644d53"
server_key_version: "encryptionkey@0"
I20260926 23:36:31.883095 23017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-2/wal/instance:
uuid: "12215fce93a24704886bca4459035e48"
format_stamp: "Formatted at 2026-09-26 23:36:31 on dist-test-slave-gwfx"
server_key: "4a827480c13bf3e625e9c8cefd1e1848"
server_key_iv: "35f8458a24625a750fa226f34b644d53"
server_key_version: "encryptionkey@0"
I20260926 23:36:31.890146 23017 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.003s
I20260926 23:36:31.895141 23032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:36:31.897013 23017 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 23:36:31.897228 23017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-2/wal
uuid: "12215fce93a24704886bca4459035e48"
format_stamp: "Formatted at 2026-09-26 23:36:31 on dist-test-slave-gwfx"
server_key: "4a827480c13bf3e625e9c8cefd1e1848"
server_key_iv: "35f8458a24625a750fa226f34b644d53"
server_key_version: "encryptionkey@0"
I20260926 23:36:31.897498 23017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:36:31.917743 23017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:36:31.923235 23017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:36:31.923558 23017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:36:31.944464 23017 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:36151
I20260926 23:36:31.944484 23083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:36151 every 8 connection(s)
I20260926 23:36:31.946234 23017 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-2/data/info.pb
I20260926 23:36:31.951272 23084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:36:31.956432 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 23017
I20260926 23:36:31.956633 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790465400694755-388-0/minicluster-data/master-2/wal/instance
I20260926 23:36:31.957091 388 external_mini_cluster.cc:1428] Setting key 60a85eaaeb11d9cc0fc3e2e4d7343262
I20260926 23:36:31.959921 23084 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42323 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:36:31 dist-test-slave-gwfx krb5kdc[22855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 26 23:36:31 dist-test-slave-gwfx krb5kdc[22855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 23:36:31.981664 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.963344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48978 (local address 127.0.97.62:42323)
0926 23:36:31.964041 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:31.964048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:31.964680 (+ 632us) server_negotiation.cc:407] Connection header received
0926 23:36:31.964941 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:31.964948 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:31.965056 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:31.965195 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:31.967085 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:31.968062 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:31.969845 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:31.970298 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:31.970426 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:31.974842 (+ 4416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:31.974848 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:31.974852 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:31.974896 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:31.978021 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:31.978876 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:31.978882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:31.978885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:31.979040 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:31.979476 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:31.979481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:31.979483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:31.979798 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:31.979969 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:31.980863 (+ 894us) server_negotiation.cc:299] Negotiation successful
0926 23:36:31.981144 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":351,"threads_started":1}
I20260926 23:36:31.983027 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.963339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42323 (local address 127.0.0.1:48978)
0926 23:36:31.964478 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:31.964532 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 23:36:31.964790 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:36:31.965329 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:36:31.965334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:36:31.966192 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:36:31.966880 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:31.966901 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:31.968257 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:31.968261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:31.969655 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:31.969669 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:31.970567 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:31.970575 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:31.970779 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:31.971538 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:36:31.971568 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:36:31.974532 (+ 2964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:36:31.978215 (+ 3683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:31.978230 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:31.978244 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:31.978740 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:31.979158 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:31.979164 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:31.979167 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:31.979358 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:31.979935 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:36:31.979945 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:36:31.980404 (+ 459us) client_negotiation.cc:769] Sending connection context
0926 23:36:31.980741 (+ 337us) client_negotiation.cc:240] Negotiation successful
0926 23:36:31.981499 (+ 758us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"mutex_wait_us":261,"thread_start_us":657,"threads_started":1}
I20260926 23:36:31.983062 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.959495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42580 (local address 127.0.97.60:36151)
0926 23:36:31.960497 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:31.960514 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:31.960571 (+ 57us) server_negotiation.cc:407] Connection header received
0926 23:36:31.962651 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:31.962680 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:31.963308 (+ 628us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:31.963779 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:31.966251 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:31.968102 (+ 1851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:31.969660 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:31.970298 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:31.970502 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:31.975116 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:31.975127 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:31.975145 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:31.975184 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:31.978251 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:31.979076 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:31.979086 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:31.979088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:31.979274 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:31.979630 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:31.979633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:31.979635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:31.980323 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:31.980650 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:31.980962 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 23:36:31.981324 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":723,"thread_start_us":430,"threads_started":1}
I20260926 23:36:31.985385 23084 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42737 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:36:31 dist-test-slave-gwfx krb5kdc[22855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 26 23:36:31 dist-test-slave-gwfx krb5kdc[22855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:36:32.001933 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.986158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42737 (local address 127.0.0.1:50740)
0926 23:36:31.986356 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:31.986377 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 23:36:31.986496 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:36:31.987202 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:36:31.987207 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:36:31.987653 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:36:31.987907 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:31.987928 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:31.989148 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:31.989152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:31.990369 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:31.990388 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:31.991397 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:31.991402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:31.991585 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:31.992240 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:36:31.992260 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:36:31.994945 (+ 2685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:36:31.998747 (+ 3802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:31.998755 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:31.998759 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:31.999381 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:31.999898 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:31.999904 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:31.999908 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:32.000123 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:32.000850 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:36:32.000854 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:36:32.001016 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 23:36:32.001143 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 23:36:32.001470 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 23:36:32.001933 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.986289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50740 (local address 127.0.97.61:42737)
0926 23:36:31.986840 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:31.986846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:31.986868 (+ 22us) server_negotiation.cc:407] Connection header received
0926 23:36:31.986935 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:31.986941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:31.987032 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:31.987152 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:31.988124 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:31.988989 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:31.990563 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:31.991232 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:31.991396 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:31.995325 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:31.995336 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:31.995342 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:31.995399 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:31.998504 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:31.999572 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:31.999578 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:31.999581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:31.999754 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:32.000252 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:32.000256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:32.000258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:32.000650 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:32.000927 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:32.001270 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 23:36:32.001516 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":283,"threads_started":1}
I20260926 23:36:32.003546 23084 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36151 } has no permanent_uuid. Determining permanent_uuid...
I20260926 23:36:32.004101 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:31.987183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48990 (local address 127.0.97.62:42323)
0926 23:36:31.987377 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:31.987382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:31.987397 (+ 15us) server_negotiation.cc:407] Connection header received
0926 23:36:31.987448 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:31.987451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:31.987525 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:31.987604 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:31.989019 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:31.989757 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:31.991259 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:31.991828 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:31.991965 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:31.996983 (+ 5018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:31.996990 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:31.996993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:31.997039 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:32.000426 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:32.001471 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:32.001478 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:32.001481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:32.001657 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:32.002148 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:32.002155 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:32.002158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:32.002585 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:32.002933 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:32.003506 (+ 573us) server_negotiation.cc:299] Negotiation successful
0926 23:36:32.003735 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 23:36:32 dist-test-slave-gwfx krb5kdc[22855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 23:36:32 dist-test-slave-gwfx krb5kdc[22855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 23:36:32.019133 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:32.004403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36151 (local address 127.0.0.1:42588)
0926 23:36:32.004613 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:32.004635 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:36:32.004778 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:36:32.005267 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:36:32.005273 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:36:32.005828 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:36:32.006013 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:32.006027 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:32.007132 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:32.007136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:32.008041 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:32.008052 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:32.008991 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:32.008996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:32.009191 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:32.009845 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:36:32.009866 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:36:32.012582 (+ 2716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:36:32.016173 (+ 3591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:32.016179 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:32.016184 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:32.016594 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:32.017055 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:32.017059 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:32.017062 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:32.017269 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:32.017877 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:36:32.017883 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:36:32.018061 (+ 178us) client_negotiation.cc:769] Sending connection context
0926 23:36:32.018265 (+ 204us) client_negotiation.cc:240] Negotiation successful
0926 23:36:32.018667 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 23:36:32.019567 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:32.004838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42588 (local address 127.0.97.60:36151)
0926 23:36:32.004989 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:32.004994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:32.005008 (+ 14us) server_negotiation.cc:407] Connection header received
0926 23:36:32.005050 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:32.005054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:32.005112 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:32.005204 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:32.006174 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:32.007007 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:32.008171 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:32.008827 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:32.008969 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:32.012878 (+ 3909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:32.012885 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:32.012889 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:32.012927 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:32.015986 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:32.016728 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:32.016734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:32.016738 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:32.016906 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:32.017397 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:32.017401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:32.017403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:32.017735 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:32.017878 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:32.018418 (+ 540us) server_negotiation.cc:299] Negotiation successful
0926 23:36:32.018973 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":294,"server-negotiator.queue_time_us":50}
I20260926 23:36:32.024015 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:32.006145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50752 (local address 127.0.97.61:42737)
0926 23:36:32.006420 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:32.006426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:32.006445 (+ 19us) server_negotiation.cc:407] Connection header received
0926 23:36:32.006511 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:32.006515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:32.006582 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:32.006666 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:32.008009 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:32.009101 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:32.010402 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:32.010940 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:32.011068 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:32.016281 (+ 5213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:32.016287 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:32.016290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:32.016335 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:32.020100 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:32.021282 (+ 1182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:32.021293 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:32.021296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:32.021499 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:32.022102 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:32.022107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:32.022110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:32.022555 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:32.022891 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:32.023353 (+ 462us) server_negotiation.cc:299] Negotiation successful
0926 23:36:32.023653 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 23:36:32.026978 23084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12215fce93a24704886bca4459035e48: Bootstrap starting.
I20260926 23:36:32.030319 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:36:32.031661 23084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12215fce93a24704886bca4459035e48: Neither blocks nor log segments found. Creating new log.
I20260926 23:36:32.033107 23084 log.cc:824] T 00000000000000000000000000000000 P 12215fce93a24704886bca4459035e48: Log is configured to *not* fsync() on all Append() calls
I20260926 23:36:32.036877 23084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12215fce93a24704886bca4459035e48: No bootstrap required, opened a new log
I20260926 23:36:32.033797 22910 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "648d9ba6e75e49118bdc43157fc10425"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48990
I20260926 23:36:32.045424 23084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12215fce93a24704886bca4459035e48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648d9ba6e75e49118bdc43157fc10425" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42323 } } peers { permanent_uuid: "17915c1992204da5a276d42a6ef84aee" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42737 } } peers { permanent_uuid: "12215fce93a24704886bca4459035e48" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36151 } }
I20260926 23:36:32.045873 23084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12215fce93a24704886bca4459035e48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12215fce93a24704886bca4459035e48, State: Initialized, Role: FOLLOWER
I20260926 23:36:32.046442 23084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12215fce93a24704886bca4459035e48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648d9ba6e75e49118bdc43157fc10425" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42323 } } peers { permanent_uuid: "17915c1992204da5a276d42a6ef84aee" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42737 } } peers { permanent_uuid: "12215fce93a24704886bca4459035e48" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36151 } }
I20260926 23:36:32.047608 23092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12215fce93a24704886bca4459035e48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648d9ba6e75e49118bdc43157fc10425" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42323 } } peers { permanent_uuid: "17915c1992204da5a276d42a6ef84aee" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42737 } } peers { permanent_uuid: "12215fce93a24704886bca4459035e48" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36151 } } }
I20260926 23:36:32.048069 23092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12215fce93a24704886bca4459035e48 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:36:32.048846 23084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12215fce93a24704886bca4459035e48 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 23:36:32.059935 23103 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 12215fce93a24704886bca4459035e48: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 23:36:32.060138 23103 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 23:36:32.067569 23009 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36151 } attempt: 5
I20260926 23:36:32.057199 23017 master_runner.cc:186] Error getting master registration for 127.0.97.62:42323: OK, instance_id { permanent_uuid: "648d9ba6e75e49118bdc43157fc10425" instance_seqno: 1790465790701603 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 23:36:32.070813 23017 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 26 23:36:32 dist-test-slave-gwfx krb5kdc[22855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 23:36:32.088385 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:32.068514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36151 (local address 127.0.0.1:42602)
0926 23:36:32.068873 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:32.068898 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 23:36:32.069052 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:36:32.069928 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:36:32.069934 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:36:32.070549 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:36:32.071074 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:32.071094 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:32.072288 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:32.072295 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:32.074206 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:32.074234 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:32.075321 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:32.075328 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:32.075570 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:32.076416 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:36:32.076443 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:36:32.080183 (+ 3740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:36:32.084672 (+ 4489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:32.084680 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:32.084684 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:32.085379 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:32.085944 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:32.085951 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:32.085955 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:32.086185 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:32.087048 (+ 863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:36:32.087054 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:36:32.087315 (+ 261us) client_negotiation.cc:769] Sending connection context
0926 23:36:32.087502 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 23:36:32.087907 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207}
I20260926 23:36:32.088384 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:32.069342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42602 (local address 127.0.97.60:36151)
0926 23:36:32.069546 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:32.069551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:32.069570 (+ 19us) server_negotiation.cc:407] Connection header received
0926 23:36:32.069639 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:32.069647 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:32.069725 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:32.069842 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:32.071273 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:32.072104 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:32.074438 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:32.075117 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:32.075260 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:32.080522 (+ 5262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:32.080532 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:32.080538 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:32.080593 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:32.084435 (+ 3842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:32.085545 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:32.085552 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:32.085557 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:32.085781 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:32.086317 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:32.086322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:32.086326 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:32.086911 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:32.087067 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:32.087648 (+ 581us) server_negotiation.cc:299] Negotiation successful
0926 23:36:32.087945 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 23:36:32.094333 23009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17915c1992204da5a276d42a6ef84aee: Bootstrap starting.
I20260926 23:36:32.098593 23009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17915c1992204da5a276d42a6ef84aee: Neither blocks nor log segments found. Creating new log.
I20260926 23:36:32.100046 23009 log.cc:824] T 00000000000000000000000000000000 P 17915c1992204da5a276d42a6ef84aee: Log is configured to *not* fsync() on all Append() calls
I20260926 23:36:32.104020 23009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17915c1992204da5a276d42a6ef84aee: No bootstrap required, opened a new log
I20260926 23:36:32.116250 23009 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17915c1992204da5a276d42a6ef84aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648d9ba6e75e49118bdc43157fc10425" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42323 } } peers { permanent_uuid: "17915c1992204da5a276d42a6ef84aee" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42737 } } peers { permanent_uuid: "12215fce93a24704886bca4459035e48" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36151 } }
I20260926 23:36:32.116815 23009 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17915c1992204da5a276d42a6ef84aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17915c1992204da5a276d42a6ef84aee, State: Initialized, Role: FOLLOWER
I20260926 23:36:32.117458 23009 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17915c1992204da5a276d42a6ef84aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648d9ba6e75e49118bdc43157fc10425" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42323 } } peers { permanent_uuid: "17915c1992204da5a276d42a6ef84aee" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42737 } } peers { permanent_uuid: "12215fce93a24704886bca4459035e48" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36151 } }
I20260926 23:36:32.118575 23106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17915c1992204da5a276d42a6ef84aee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648d9ba6e75e49118bdc43157fc10425" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42323 } } peers { permanent_uuid: "17915c1992204da5a276d42a6ef84aee" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42737 } } peers { permanent_uuid: "12215fce93a24704886bca4459035e48" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36151 } } }
I20260926 23:36:32.119056 23106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17915c1992204da5a276d42a6ef84aee [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:36:32.120112 23009 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17915c1992204da5a276d42a6ef84aee [sys.catalog]: configured and running, proceeding with master startup.
W20260926 23:36:32.132275 23117 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 17915c1992204da5a276d42a6ef84aee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 23:36:32.132552 23117 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 23:36:32.128335 22942 master_runner.cc:186] Error getting master registration for 127.0.97.62:42323: OK, instance_id { permanent_uuid: "648d9ba6e75e49118bdc43157fc10425" instance_seqno: 1790465790701603 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 23:36:32.140901 22942 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 23:36:32.145745 22935 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36151 } attempt: 5
Sep 26 23:36:32 dist-test-slave-gwfx krb5kdc[22855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 23:36:32.169134 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:32.146745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36151 (local address 127.0.0.1:42612)
0926 23:36:32.147392 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:32.147420 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 23:36:32.147602 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:36:32.148111 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:36:32.148118 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:36:32.148767 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:36:32.149039 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:32.149054 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:32.150843 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:32.150850 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:32.152470 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:32.152492 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:32.153709 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:32.153718 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:32.153977 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:32.154941 (+ 964us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:36:32.154971 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:36:32.158523 (+ 3552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:36:32.164312 (+ 5789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:32.164320 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:32.164325 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:32.165074 (+ 749us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:32.166249 (+ 1175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:32.166256 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:32.166260 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:32.166505 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:32.167809 (+ 1304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:36:32.167816 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:36:32.168029 (+ 213us) client_negotiation.cc:769] Sending connection context
0926 23:36:32.168217 (+ 188us) client_negotiation.cc:240] Negotiation successful
0926 23:36:32.168624 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483}
I20260926 23:36:32.170140 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:32.147140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42612 (local address 127.0.97.60:36151)
0926 23:36:32.147391 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:32.147401 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:32.147519 (+ 118us) server_negotiation.cc:407] Connection header received
0926 23:36:32.147759 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:32.147766 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:32.147891 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:32.148033 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:32.149247 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:32.150585 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:32.152699 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:32.153510 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:32.153665 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:32.158933 (+ 5268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:32.158943 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:32.158949 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:32.159016 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:32.163998 (+ 4982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:32.165381 (+ 1383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:32.165398 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:32.165402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:32.165888 (+ 486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:32.166673 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:32.166722 (+ 49us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:32.166729 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:32.167612 (+ 883us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:32.167947 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:32.168493 (+ 546us) server_negotiation.cc:299] Negotiation successful
0926 23:36:32.169114 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 23:36:32.176294 23038 tablet.cc:2369] T 00000000000000000000000000000000 P 12215fce93a24704886bca4459035e48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:36:32.178501 22935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 648d9ba6e75e49118bdc43157fc10425: Bootstrap starting.
I20260926 23:36:32.184172 22935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 648d9ba6e75e49118bdc43157fc10425: Neither blocks nor log segments found. Creating new log.
I20260926 23:36:32.185963 22935 log.cc:824] T 00000000000000000000000000000000 P 648d9ba6e75e49118bdc43157fc10425: Log is configured to *not* fsync() on all Append() calls
I20260926 23:36:32.191381 22935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 648d9ba6e75e49118bdc43157fc10425: No bootstrap required, opened a new log
I20260926 23:36:32.200521 22935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 648d9ba6e75e49118bdc43157fc10425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648d9ba6e75e49118bdc43157fc10425" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42323 } } peers { permanent_uuid: "17915c1992204da5a276d42a6ef84aee" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42737 } } peers { permanent_uuid: "12215fce93a24704886bca4459035e48" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36151 } }
I20260926 23:36:32.201071 22935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 648d9ba6e75e49118bdc43157fc10425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 648d9ba6e75e49118bdc43157fc10425, State: Initialized, Role: FOLLOWER
I20260926 23:36:32.201740 22935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 648d9ba6e75e49118bdc43157fc10425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648d9ba6e75e49118bdc43157fc10425" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42323 } } peers { permanent_uuid: "17915c1992204da5a276d42a6ef84aee" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42737 } } peers { permanent_uuid: "12215fce93a24704886bca4459035e48" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36151 } }
I20260926 23:36:32.203539 23120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 648d9ba6e75e49118bdc43157fc10425 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648d9ba6e75e49118bdc43157fc10425" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42323 } } peers { permanent_uuid: "17915c1992204da5a276d42a6ef84aee" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42737 } } peers { permanent_uuid: "12215fce93a24704886bca4459035e48" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36151 } } }
I20260926 23:36:32.204398 23120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 648d9ba6e75e49118bdc43157fc10425 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:36:32.205042 22935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 648d9ba6e75e49118bdc43157fc10425 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:36:32.205142 22910 raft_consensus.cc:482] T 00000000000000000000000000000000 P 648d9ba6e75e49118bdc43157fc10425 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 23:36:32.205374 22910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 648d9ba6e75e49118bdc43157fc10425 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:36:32.209219 22910 raft_consensus.cc:504] T 00000000000000000000000000000000 P 648d9ba6e75e49118bdc43157fc10425 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648d9ba6e75e49118bdc43157fc10425" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42323 } } peers { permanent_uuid: "17915c1992204da5a276d42a6ef84aee" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42737 } } peers { permanent_uuid: "12215fce93a24704886bca4459035e48" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36151 } }
I20260926 23:36:32.212477 22984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "648d9ba6e75e49118bdc43157fc10425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "17915c1992204da5a276d42a6ef84aee"
I20260926 23:36:32.213147 22984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17915c1992204da5a276d42a6ef84aee [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:36:32.217741 22984 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 17915c1992204da5a276d42a6ef84aee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 648d9ba6e75e49118bdc43157fc10425 in term 1.
I20260926 23:36:32.218966 22887 leader_election.cc:302] T 00000000000000000000000000000000 P 648d9ba6e75e49118bdc43157fc10425 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17915c1992204da5a276d42a6ef84aee, 648d9ba6e75e49118bdc43157fc10425; no voters:
I20260926 23:36:32.221427 23059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "648d9ba6e75e49118bdc43157fc10425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "12215fce93a24704886bca4459035e48"
I20260926 23:36:32.222127 23059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12215fce93a24704886bca4459035e48 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:36:32.226888 23059 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 12215fce93a24704886bca4459035e48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 648d9ba6e75e49118bdc43157fc10425 in term 1.
I20260926 23:36:32.228121 22910 leader_election.cc:288] T 00000000000000000000000000000000 P 648d9ba6e75e49118bdc43157fc10425 [CANDIDATE]: Term 1 election: Requested vote from peers 17915c1992204da5a276d42a6ef84aee (127.0.97.61:42737), 12215fce93a24704886bca4459035e48 (127.0.97.60:36151)
I20260926 23:36:32.235800 23120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 648d9ba6e75e49118bdc43157fc10425 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:36:32.245990 23120 raft_consensus.cc:686] T 00000000000000000000000000000000 P 648d9ba6e75e49118bdc43157fc10425 [term 1 LEADER]: Becoming Leader. State: Replica: 648d9ba6e75e49118bdc43157fc10425, State: Running, Role: LEADER
I20260926 23:36:32.247102 23120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 648d9ba6e75e49118bdc43157fc10425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648d9ba6e75e49118bdc43157fc10425" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42323 } } peers { permanent_uuid: "17915c1992204da5a276d42a6ef84aee" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42737 } } peers { permanent_uuid: "12215fce93a24704886bca4459035e48" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36151 } }
I20260926 23:36:32.261804 23127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 648d9ba6e75e49118bdc43157fc10425 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 648d9ba6e75e49118bdc43157fc10425. Latest consensus state: current_term: 1 leader_uuid: "648d9ba6e75e49118bdc43157fc10425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648d9ba6e75e49118bdc43157fc10425" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42323 } } peers { permanent_uuid: "17915c1992204da5a276d42a6ef84aee" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42737 } } peers { permanent_uuid: "12215fce93a24704886bca4459035e48" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36151 } } }
I20260926 23:36:32.262517 23127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 648d9ba6e75e49118bdc43157fc10425 [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:32.272239 23136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:36:32.280827 23139 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260926 23:36:32.307703 22963 tablet.cc:2369] T 00000000000000000000000000000000 P 17915c1992204da5a276d42a6ef84aee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:36:32.312752 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:32.287423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42626 (local address 127.0.97.60:36151)
0926 23:36:32.287835 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:32.287842 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:32.292508 (+ 4666us) server_negotiation.cc:407] Connection header received
0926 23:36:32.292821 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:32.292828 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:32.292958 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:32.293042 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:32.295632 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:32.296844 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:32.298948 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:32.299713 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:32.299838 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:32.304387 (+ 4549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:32.304397 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:32.304403 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:32.304461 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:32.308824 (+ 4363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:32.309950 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:32.309958 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:32.309962 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:32.310168 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:32.310664 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:32.310669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:32.310673 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:32.311116 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:32.311409 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:32.311985 (+ 576us) server_negotiation.cc:299] Negotiation successful
0926 23:36:32.312308 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236}
I20260926 23:36:32.314640 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:32.274000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50760 (local address 127.0.97.61:42737)
0926 23:36:32.274226 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:32.274233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:32.282422 (+ 8189us) server_negotiation.cc:407] Connection header received
0926 23:36:32.282616 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:32.282623 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:32.282781 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:32.282910 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:32.289160 (+ 6250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:32.290428 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:32.292568 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:32.293266 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:32.293377 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:32.306971 (+ 13594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:32.306982 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:32.306987 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:32.307049 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:32.310842 (+ 3793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:32.311780 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:32.311788 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:32.311792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:32.311989 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:32.312441 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:32.312447 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:32.312450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:32.312878 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:32.313181 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:32.313885 (+ 704us) server_negotiation.cc:299] Negotiation successful
0926 23:36:32.314202 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 23:36:32.329217 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:32.258744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49000 (local address 127.0.97.62:42323)
0926 23:36:32.259022 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:32.259033 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:32.273227 (+ 14194us) server_negotiation.cc:407] Connection header received
0926 23:36:32.277937 (+ 4710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:32.277945 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:32.278073 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:32.278185 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:32.280974 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:32.282282 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:32.298840 (+ 16558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:32.300090 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:32.300252 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:32.318285 (+ 18033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:32.318294 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:32.318301 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:32.318363 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:32.323260 (+ 4897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:32.324745 (+ 1485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:32.324753 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:32.324758 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:32.324990 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:32.325603 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:32.325609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:32.325613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:32.326112 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:32.328117 (+ 2005us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:32.328340 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 23:36:32.328724 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 23:36:32.334285 23136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:36:32.412467 23059 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 12215fce93a24704886bca4459035e48 [term 1 FOLLOWER]: Refusing update from remote peer 648d9ba6e75e49118bdc43157fc10425: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:36:32.414284 23127 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 648d9ba6e75e49118bdc43157fc10425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12215fce93a24704886bca4459035e48" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:36:32.429415 22984 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 17915c1992204da5a276d42a6ef84aee [term 1 FOLLOWER]: Refusing update from remote peer 648d9ba6e75e49118bdc43157fc10425: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:36:32.431538 23127 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 648d9ba6e75e49118bdc43157fc10425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17915c1992204da5a276d42a6ef84aee" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 23:36:32.439697 22889 tablet.cc:2369] T 00000000000000000000000000000000 P 648d9ba6e75e49118bdc43157fc10425: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:36:32.475124 23106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17915c1992204da5a276d42a6ef84aee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 648d9ba6e75e49118bdc43157fc10425. Latest consensus state: current_term: 1 leader_uuid: "648d9ba6e75e49118bdc43157fc10425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648d9ba6e75e49118bdc43157fc10425" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42323 } } peers { permanent_uuid: "17915c1992204da5a276d42a6ef84aee" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42737 } } peers { permanent_uuid: "12215fce93a24704886bca4459035e48" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36151 } } }
I20260926 23:36:32.475572 23106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17915c1992204da5a276d42a6ef84aee [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:36:32.479856 23143 mvcc.cc:205] Tried to move back new op lower bound from 7333747885614772224 to 7333747885071278080. Current Snapshot: MvccSnapshot[applied={T|T < 7333747885614772224}]
I20260926 23:36:32.489295 23145 mvcc.cc:205] Tried to move back new op lower bound from 7333747885614772224 to 7333747885071278080. Current Snapshot: MvccSnapshot[applied={T|T < 7333747885614772224}]
I20260926 23:36:32.491467 23092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12215fce93a24704886bca4459035e48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 648d9ba6e75e49118bdc43157fc10425. Latest consensus state: current_term: 1 leader_uuid: "648d9ba6e75e49118bdc43157fc10425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648d9ba6e75e49118bdc43157fc10425" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42323 } } peers { permanent_uuid: "17915c1992204da5a276d42a6ef84aee" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42737 } } peers { permanent_uuid: "12215fce93a24704886bca4459035e48" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36151 } } }
I20260926 23:36:32.500564 23092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12215fce93a24704886bca4459035e48 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:36:32.511350 23120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 648d9ba6e75e49118bdc43157fc10425 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "648d9ba6e75e49118bdc43157fc10425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648d9ba6e75e49118bdc43157fc10425" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42323 } } peers { permanent_uuid: "17915c1992204da5a276d42a6ef84aee" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42737 } } peers { permanent_uuid: "12215fce93a24704886bca4459035e48" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36151 } } }
I20260926 23:36:32.511776 23120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 648d9ba6e75e49118bdc43157fc10425 [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:32.515295 23106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17915c1992204da5a276d42a6ef84aee [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "648d9ba6e75e49118bdc43157fc10425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648d9ba6e75e49118bdc43157fc10425" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42323 } } peers { permanent_uuid: "17915c1992204da5a276d42a6ef84aee" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42737 } } peers { permanent_uuid: "12215fce93a24704886bca4459035e48" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36151 } } }
I20260926 23:36:32.515633 23106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17915c1992204da5a276d42a6ef84aee [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:36:32.516849 23136 catalog_manager.cc:1347] Generated new cluster ID: 1fccde0b2792433eb98dd9bb217206f1
I20260926 23:36:32.516978 23136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:36:32.551801 23127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 648d9ba6e75e49118bdc43157fc10425 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "648d9ba6e75e49118bdc43157fc10425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648d9ba6e75e49118bdc43157fc10425" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42323 } } peers { permanent_uuid: "17915c1992204da5a276d42a6ef84aee" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42737 } } peers { permanent_uuid: "12215fce93a24704886bca4459035e48" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36151 } } }
I20260926 23:36:32.558367 23127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 648d9ba6e75e49118bdc43157fc10425 [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:32.565586 23144 mvcc.cc:205] Tried to move back new op lower bound from 7333747885614772224 to 7333747885071278080. Current Snapshot: MvccSnapshot[applied={T|T < 7333747885614772224}]
I20260926 23:36:32.583570 23092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12215fce93a24704886bca4459035e48 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "648d9ba6e75e49118bdc43157fc10425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648d9ba6e75e49118bdc43157fc10425" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42323 } } peers { permanent_uuid: "17915c1992204da5a276d42a6ef84aee" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42737 } } peers { permanent_uuid: "12215fce93a24704886bca4459035e48" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36151 } } }
I20260926 23:36:32.583961 23092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12215fce93a24704886bca4459035e48 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:36:32.621028 23136 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:36:32.629453 23136 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:36:32.678411 23136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 648d9ba6e75e49118bdc43157fc10425: Generated new TSK 0
I20260926 23:36:32.680219 23136 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 23:36:32.835085 388 mini_kdc.cc:348] Time spent kdestroy: real 0.109s user 0.000s sys 0.031s
W20260926 23:36:32.857620 23159 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:36:32.842044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49014 (local address 127.0.97.62:42323)
0926 23:36:32.853078 (+ 11034us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:32.853086 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:32.854076 (+ 990us) server_negotiation.cc:407] Connection header received
0926 23:36:32.854793 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:32.854803 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:32.854980 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:32.855112 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:32.856991 (+ 1879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49014: BlockingRecv error: recv got EOF from 127.0.0.1:49014 (error 108)
Metrics: {"server-negotiator.queue_time_us":10815,"thread_start_us":10476,"threads_started":1}
W20260926 23:36:32.867218 23160 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:36:32.854601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50764 (local address 127.0.97.61:42737)
0926 23:36:32.862604 (+ 8003us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:32.862612 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:32.862643 (+ 31us) server_negotiation.cc:407] Connection header received
0926 23:36:32.862770 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:32.862777 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:32.862909 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:32.865672 (+ 2763us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:32.866766 (+ 1094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50764: BlockingRecv error: recv got EOF from 127.0.0.1:50764 (error 108)
Metrics: {"server-negotiator.queue_time_us":7826,"thread_start_us":7565,"threads_started":1}
W20260926 23:36:32.877898 23162 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:36:32.859939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42628 (local address 127.0.97.60:36151)
0926 23:36:32.875218 (+ 15279us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:32.875224 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:32.875264 (+ 40us) server_negotiation.cc:407] Connection header received
0926 23:36:32.875350 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:32.875355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:32.875448 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:32.875616 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:32.877223 (+ 1607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42628: BlockingRecv error: recv got EOF from 127.0.0.1:42628 (error 108)
Metrics: {"server-negotiator.queue_time_us":15115,"thread_start_us":14926,"threads_started":1}
W20260926 23:36:32.885838 23160 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:36:32.883188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50776 (local address 127.0.97.61:42737)
0926 23:36:32.883437 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:32.883445 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:32.884016 (+ 571us) server_negotiation.cc:407] Connection header received
0926 23:36:32.884298 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:32.884303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:32.884448 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:32.884548 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:32.885473 (+ 925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50776: BlockingRecv error: recv got EOF from 127.0.0.1:50776 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260926 23:36:32.895489 23160 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:36:32.893023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50792 (local address 127.0.97.61:42737)
0926 23:36:32.893344 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:32.893353 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:32.893899 (+ 546us) server_negotiation.cc:407] Connection header received
0926 23:36:32.894119 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:32.894125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:32.894219 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:32.894314 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:32.895195 (+ 881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50792: BlockingRecv error: recv got EOF from 127.0.0.1:50792 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260926 23:36:32.903591 23160 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:36:32.901041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50808 (local address 127.0.97.61:42737)
0926 23:36:32.901351 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:32.901359 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:32.901926 (+ 567us) server_negotiation.cc:407] Connection header received
0926 23:36:32.902181 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:32.902188 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:32.902285 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:32.902384 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:32.903305 (+ 921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50808: BlockingRecv error: recv got EOF from 127.0.0.1:50808 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
W20260926 23:36:32.913885 23160 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:36:32.911380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50812 (local address 127.0.97.61:42737)
0926 23:36:32.911652 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:32.911660 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:32.912233 (+ 573us) server_negotiation.cc:407] Connection header received
0926 23:36:32.912472 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:32.912478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:32.912577 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:32.912679 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:32.913558 (+ 879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50812: BlockingRecv error: recv got EOF from 127.0.0.1:50812 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
W20260926 23:36:32.928534 23160 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:36:32.925919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50824 (local address 127.0.97.61:42737)
0926 23:36:32.926184 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:32.926192 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:32.926895 (+ 703us) server_negotiation.cc:407] Connection header received
0926 23:36:32.927149 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:32.927154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:32.927249 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:32.927348 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:32.928263 (+ 915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50824: BlockingRecv error: recv got EOF from 127.0.0.1:50824 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260926 23:36:32.951347 23160 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:36:32.948723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50834 (local address 127.0.97.61:42737)
0926 23:36:32.949001 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:32.949010 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:32.949661 (+ 651us) server_negotiation.cc:407] Connection header received
0926 23:36:32.949914 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:32.949920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:32.950011 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:32.950115 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:32.951030 (+ 915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50834: BlockingRecv error: recv got EOF from 127.0.0.1:50834 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260926 23:36:32.990056 23160 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:36:32.987211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50848 (local address 127.0.97.61:42737)
0926 23:36:32.987478 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:32.987486 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:32.987881 (+ 395us) server_negotiation.cc:407] Connection header received
0926 23:36:32.988052 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:32.988059 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:32.988176 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:32.988324 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:32.989611 (+ 1287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50848: BlockingRecv error: recv got EOF from 127.0.0.1:50848 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
W20260926 23:36:33.060457 23160 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:36:33.058239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50862 (local address 127.0.97.61:42737)
0926 23:36:33.058453 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:33.058458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:33.058788 (+ 330us) server_negotiation.cc:407] Connection header received
0926 23:36:33.058992 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:33.058997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:33.059089 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:33.059212 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:33.060151 (+ 939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50862: BlockingRecv error: recv got EOF from 127.0.0.1:50862 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 23:36:33.062788 23103 catalog_manager.cc:1259] Loaded cluster ID: 1fccde0b2792433eb98dd9bb217206f1
I20260926 23:36:33.062904 23103 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 12215fce93a24704886bca4459035e48: loading cluster ID for follower catalog manager: success
I20260926 23:36:33.066146 23103 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 12215fce93a24704886bca4459035e48: acquiring CA information for follower catalog manager: success
I20260926 23:36:33.069077 23103 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 12215fce93a24704886bca4459035e48: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 23:36:33.135253 23117 catalog_manager.cc:1259] Loaded cluster ID: 1fccde0b2792433eb98dd9bb217206f1
I20260926 23:36:33.135411 23117 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 17915c1992204da5a276d42a6ef84aee: loading cluster ID for follower catalog manager: success
I20260926 23:36:33.138468 23117 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 17915c1992204da5a276d42a6ef84aee: acquiring CA information for follower catalog manager: success
I20260926 23:36:33.140929 23117 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 17915c1992204da5a276d42a6ef84aee: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 23:36:33.203601 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:33.196140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50866 (local address 127.0.97.61:42737)
0926 23:36:33.196432 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:33.196439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:33.196787 (+ 348us) server_negotiation.cc:407] Connection header received
0926 23:36:33.197041 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:33.197046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:33.197269 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:33.197392 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:36:33.197941 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:33.198835 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:33.200378 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:33.200991 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:33.201172 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:33.201875 (+ 703us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:36:33.202820 (+ 945us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:36:33.202985 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:33.203135 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 23:36:33.203266 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260926 23:36:33.222383 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:33.214420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42634 (local address 127.0.97.60:36151)
0926 23:36:33.214756 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:33.214764 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:33.215089 (+ 325us) server_negotiation.cc:407] Connection header received
0926 23:36:33.215371 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:33.215376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:33.215582 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:33.215753 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:36:33.216313 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:33.217278 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:33.219236 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:33.219753 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:33.219890 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:33.220625 (+ 735us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:36:33.221462 (+ 837us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:36:33.221604 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:33.221849 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 23:36:33.221966 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260926 23:36:33.238068 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:33.231461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50878 (local address 127.0.97.61:42737)
0926 23:36:33.231669 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:33.231676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:33.231802 (+ 126us) server_negotiation.cc:407] Connection header received
0926 23:36:33.232005 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:33.232011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:33.232115 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:33.232208 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:36:33.233031 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:33.234225 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:33.236032 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:33.236491 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:33.236611 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:33.237114 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:36:33.237258 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:36:33.237334 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:33.237487 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 23:36:33.237594 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 23:36:33.239457 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:33.230429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49028 (local address 127.0.97.62:42323)
0926 23:36:33.230667 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:33.230673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:33.231098 (+ 425us) server_negotiation.cc:407] Connection header received
0926 23:36:33.231685 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:33.231693 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:33.232002 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:33.232183 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:36:33.233025 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:33.234225 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:33.236833 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:33.237416 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:33.237542 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:33.238169 (+ 627us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:36:33.238811 (+ 642us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:36:33.238913 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:33.239037 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 23:36:33.239142 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 23:36:33.239732 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:33.231963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42636 (local address 127.0.97.60:36151)
0926 23:36:33.232175 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:33.232183 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:33.232915 (+ 732us) server_negotiation.cc:407] Connection header received
0926 23:36:33.233078 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:33.233082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:33.233161 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:33.233239 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:36:33.233734 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:33.234640 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:33.237689 (+ 3049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:33.238315 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:33.238438 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:33.238898 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:36:33.239049 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:36:33.239141 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:33.239336 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 23:36:33.239441 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 23:36:33.246548 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 22868
I20260926 23:36:33.247987 22930 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:36:33.374549 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 22868
I20260926 23:36:33.400763 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 22942
I20260926 23:36:33.403139 23004 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:36:33.531173 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 22942
I20260926 23:36:33.552749 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 23017
I20260926 23:36:33.554148 23079 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:36:33.678143 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 23017
2026-09-26T23:36:33Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4889 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 23:36:33 dist-test-slave-gwfx krb5kdc[23225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 23:36:33 dist-test-slave-gwfx krb5kdc[23225](info): set up 2 sockets
Sep 26 23:36:33 dist-test-slave-gwfx krb5kdc[23225](info): commencing operation
krb5kdc: starting...
W20260926 23:36:34.775947 388 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.001s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/krb5kdc/test-user.keytab.
Sep 26 23:36:34 dist-test-slave-gwfx krb5kdc[23225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T23:36:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T23:36:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 23:36:35.102761 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37113
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37113,127.0.97.61:37569,127.0.97.60:46169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 23:36:35.498020 23238 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 23:36:35.498657 23238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:36:35.498867 23238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:36:35.498984 23238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:36:35.507876 23238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:36:35.508008 23238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:36:35.508075 23238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:36:35.508133 23238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:36:35.508242 23238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:36:35.519622 23238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37113,127.0.97.61:37569,127.0.97.60:46169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:36:35.521070 23238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:36:35.522466 23238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:36:35.537109 23244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:35.537178 23243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:35.537141 23246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:36:35.539037 23238 server_base.cc:1034] running on GCE node
I20260926 23:36:35.540014 23238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:36:35.541790 23238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:36:35.543109 23238 hybrid_clock.cc:648] HybridClock initialized: now 1790465795543005 us; error 74 us; skew 500 ppm
I20260926 23:36:35.543563 23238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:36:35 dist-test-slave-gwfx krb5kdc[23225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:36:35.547273 23238 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 23:36:35.549206 23238 webserver.cc:460] Webserver started at http://127.0.97.62:33429/ using document root <none> and password file <none>
I20260926 23:36:35.550010 23238 fs_manager.cc:359] Metadata directory not provided
I20260926 23:36:35.550113 23238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:36:35.550450 23238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:36:35.554127 23238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-0/data/instance:
uuid: "9a74a76eaa244e6e8d532141d2ed650a"
format_stamp: "Formatted at 2026-09-26 23:36:35 on dist-test-slave-gwfx"
server_key: "063ca19ffa27b1496a2229b6ef0150a5"
server_key_iv: "aded2ced00cd19f76a8358bcd2625560"
server_key_version: "encryptionkey@0"
I20260926 23:36:35.555142 23238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-0/wal/instance:
uuid: "9a74a76eaa244e6e8d532141d2ed650a"
format_stamp: "Formatted at 2026-09-26 23:36:35 on dist-test-slave-gwfx"
server_key: "063ca19ffa27b1496a2229b6ef0150a5"
server_key_iv: "aded2ced00cd19f76a8358bcd2625560"
server_key_version: "encryptionkey@0"
I20260926 23:36:35.561736 23238 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.005s
I20260926 23:36:35.566891 23253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:36:35.569000 23238 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260926 23:36:35.569252 23238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-0/wal
uuid: "9a74a76eaa244e6e8d532141d2ed650a"
format_stamp: "Formatted at 2026-09-26 23:36:35 on dist-test-slave-gwfx"
server_key: "063ca19ffa27b1496a2229b6ef0150a5"
server_key_iv: "aded2ced00cd19f76a8358bcd2625560"
server_key_version: "encryptionkey@0"
I20260926 23:36:35.569523 23238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:36:35.586012 23238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:36:35.591313 23238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:36:35.591612 23238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:36:35.613751 23238 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37113
I20260926 23:36:35.613838 23304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37113 every 8 connection(s)
I20260926 23:36:35.615469 23238 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-0/data/info.pb
I20260926 23:36:35.617026 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 23238
I20260926 23:36:35.617295 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-0/wal/instance
I20260926 23:36:35.617945 388 external_mini_cluster.cc:1428] Setting key 2c168bb5d00d9b634008039cc52b7a8f
I20260926 23:36:35.622032 23305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:36:35.633685 23305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37113 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:36:35 dist-test-slave-gwfx krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260926 23:36:35.662518 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:35.640468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45416 (local address 127.0.97.62:37113)
0926 23:36:35.641363 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:35.641372 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:35.641451 (+ 79us) server_negotiation.cc:407] Connection header received
0926 23:36:35.642770 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:35.642777 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:35.643089 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:35.643312 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:35.644735 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:35.646152 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:35.647739 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:35.648341 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:35.648530 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:35.652991 (+ 4461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:35.652999 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:35.653016 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:35.653057 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:35.656300 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:35.657418 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:35.657430 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:35.657433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:35.657704 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:35.658387 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:35.658392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:35.658394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:35.659242 (+ 848us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:35.659488 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:35.660554 (+ 1066us) server_negotiation.cc:299] Negotiation successful
0926 23:36:35.661134 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":212,"server-negotiator.queue_time_us":645,"thread_start_us":418,"threads_started":1}
I20260926 23:36:35.662582 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:35.638417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37113 (local address 127.0.0.1:45416)
0926 23:36:35.639727 (+ 1310us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:35.639793 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 23:36:35.642277 (+ 2484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:36:35.643355 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:36:35.643362 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:36:35.644137 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:36:35.644511 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:35.644531 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:35.646337 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:35.646341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:35.647583 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:35.647597 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:35.648612 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:35.648619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:35.648810 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:35.649630 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:36:35.649654 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:36:35.652690 (+ 3036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:36:35.656465 (+ 3775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:35.656479 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:35.656493 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:35.657147 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:35.657887 (+ 740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:35.657893 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:35.657897 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:35.658181 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:35.659448 (+ 1267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:36:35.659456 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:36:35.659785 (+ 329us) client_negotiation.cc:769] Sending connection context
0926 23:36:35.660252 (+ 467us) client_negotiation.cc:240] Negotiation successful
0926 23:36:35.660927 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"thread_start_us":609,"threads_started":1}
I20260926 23:36:35.666647 23305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37569 } has no permanent_uuid. Determining permanent_uuid...
I20260926 23:36:35.668377 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:35.667663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37569 (local address 127.0.0.1:39830)
0926 23:36:35.667982 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:35.668166 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260926 23:36:35.668812 23257 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37569: connect: Connection refused (error 111)
I20260926 23:36:35.671114 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:35.670532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37569 (local address 127.0.0.1:39840)
0926 23:36:35.670778 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:35.670911 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260926 23:36:35.671445 23305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37569: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37569: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260926 23:36:35.698498 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:37569
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37113,127.0.97.61:37569,127.0.97.60:46169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:36:35.720968 23305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37569 } attempt: 1
I20260926 23:36:35.722671 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:35.721974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37569 (local address 127.0.0.1:39844)
0926 23:36:35.722283 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:35.722441 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260926 23:36:35.724036 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:35.723550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37569 (local address 127.0.0.1:39854)
0926 23:36:35.723738 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:35.723851 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 23:36:35.724292 23305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37569: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37569: connect: Connection refused (error 111)
I20260926 23:36:35.792641 23305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37569 } attempt: 2
I20260926 23:36:35.794348 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:35.793619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37569 (local address 127.0.0.1:39864)
0926 23:36:35.793973 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:35.794125 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 23:36:35.796638 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:35.795786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37569 (local address 127.0.0.1:39868)
0926 23:36:35.796205 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:35.796344 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187}
W20260926 23:36:35.796972 23305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37569: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37569: connect: Connection refused (error 111)
I20260926 23:36:35.888365 23305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37569 } attempt: 3
I20260926 23:36:35.890038 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:35.889367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37569 (local address 127.0.0.1:39878)
0926 23:36:35.889667 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:35.889826 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260926 23:36:35.891347 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:35.890889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37569 (local address 127.0.0.1:39884)
0926 23:36:35.891058 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:35.891158 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 23:36:35.891638 23305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37569: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37569: connect: Connection refused (error 111)
I20260926 23:36:36.034982 23305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37569 } attempt: 4
I20260926 23:36:36.036689 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.035866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37569 (local address 127.0.0.1:39886)
0926 23:36:36.036325 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:36.036485 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265}
I20260926 23:36:36.037969 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.037480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37569 (local address 127.0.0.1:39894)
0926 23:36:36.037666 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:36.037783 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 23:36:36.038261 23305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37569: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37569: connect: Connection refused (error 111)
W20260926 23:36:36.117077 23312 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 23:36:36.117535 23312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:36:36.117652 23312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:36:36.117784 23312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:36:36.126580 23312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:36:36.126780 23312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:36:36.126849 23312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:36:36.126936 23312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:36:36.127018 23312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:36:36.139056 23312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37113,127.0.97.61:37569,127.0.97.60:46169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:37569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:36:36.140883 23312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:36:36.142558 23312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 23:36:36.156955 23317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:36.157070 23320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:36.157047 23318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:36:36.158347 23312 server_base.cc:1034] running on GCE node
I20260926 23:36:36.159539 23312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:36:36.161592 23312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:36:36.162889 23312 hybrid_clock.cc:648] HybridClock initialized: now 1790465796162806 us; error 51 us; skew 500 ppm
I20260926 23:36:36.163439 23312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:36:36 dist-test-slave-gwfx krb5kdc[23225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:36:36.167809 23312 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260926 23:36:36.169943 23312 webserver.cc:460] Webserver started at http://127.0.97.61:32887/ using document root <none> and password file <none>
I20260926 23:36:36.170948 23312 fs_manager.cc:359] Metadata directory not provided
I20260926 23:36:36.171072 23312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:36:36.171481 23312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:36:36.175838 23312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-1/data/instance:
uuid: "b8e1be79d4864a9eae6eb37f8db979e4"
format_stamp: "Formatted at 2026-09-26 23:36:36 on dist-test-slave-gwfx"
server_key: "672d7f797bb644815328cd94b0e76e58"
server_key_iv: "23f87ce05dd720c22310b7576b354bb3"
server_key_version: "encryptionkey@0"
I20260926 23:36:36.176973 23312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-1/wal/instance:
uuid: "b8e1be79d4864a9eae6eb37f8db979e4"
format_stamp: "Formatted at 2026-09-26 23:36:36 on dist-test-slave-gwfx"
server_key: "672d7f797bb644815328cd94b0e76e58"
server_key_iv: "23f87ce05dd720c22310b7576b354bb3"
server_key_version: "encryptionkey@0"
I20260926 23:36:36.184108 23312 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260926 23:36:36.189018 23327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:36:36.191277 23312 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 23:36:36.191542 23312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-1/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-1/wal
uuid: "b8e1be79d4864a9eae6eb37f8db979e4"
format_stamp: "Formatted at 2026-09-26 23:36:36 on dist-test-slave-gwfx"
server_key: "672d7f797bb644815328cd94b0e76e58"
server_key_iv: "23f87ce05dd720c22310b7576b354bb3"
server_key_version: "encryptionkey@0"
I20260926 23:36:36.191836 23312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:36:36.213354 23312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:36:36.218781 23312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:36:36.219108 23312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:36:36.241624 23312 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:37569
I20260926 23:36:36.241668 23378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:37569 every 8 connection(s)
I20260926 23:36:36.243428 23312 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-1/data/info.pb
I20260926 23:36:36.247814 23379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:36:36.251060 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 23312
I20260926 23:36:36.251281 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-1/wal/instance
I20260926 23:36:36.251856 388 external_mini_cluster.cc:1428] Setting key 4d075553519c6eab7902e7be9acd4472
I20260926 23:36:36.256690 23379 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37113 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:36:36 dist-test-slave-gwfx krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260926 23:36:36.286424 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.260424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45424 (local address 127.0.97.62:37113)
0926 23:36:36.261138 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:36.261146 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:36.262456 (+ 1310us) server_negotiation.cc:407] Connection header received
0926 23:36:36.265092 (+ 2636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:36.265097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:36.265213 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:36.265345 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:36.268024 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.268799 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.271287 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.271756 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.271894 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:36.277520 (+ 5626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:36.277530 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:36.277535 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:36.277623 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:36.281937 (+ 4314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:36.283284 (+ 1347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:36.283292 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:36.283297 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:36.283499 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:36.284056 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:36.284062 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:36.284065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:36.284514 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:36.284715 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:36.285728 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0926 23:36:36.285959 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":359,"threads_started":1}
I20260926 23:36:36.287380 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.261246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37113 (local address 127.0.0.1:45424)
0926 23:36:36.262233 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:36.262287 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 23:36:36.264650 (+ 2363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:36:36.265710 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:36:36.265725 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:36:36.266623 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:36:36.267642 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:36.267674 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.269009 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.269016 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:36.271022 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:36.271045 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.272051 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.272059 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:36.272391 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:36.273451 (+ 1060us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:36:36.273493 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:36:36.277040 (+ 3547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:36:36.282268 (+ 5228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:36.282279 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:36.282296 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:36.283069 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:36.283662 (+ 593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:36.283665 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:36.283668 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:36.283897 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:36.284718 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:36:36.284726 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:36:36.285175 (+ 449us) client_negotiation.cc:769] Sending connection context
0926 23:36:36.285514 (+ 339us) client_negotiation.cc:240] Negotiation successful
0926 23:36:36.286000 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":413,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260926 23:36:36.289783 23379 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37569 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:36:36 dist-test-slave-gwfx krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 23:36:36.306911 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.290911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37569 (local address 127.0.0.1:39910)
0926 23:36:36.291143 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:36.291167 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:36:36.291328 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:36:36.292262 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:36:36.292267 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:36:36.292786 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:36:36.292947 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:36.292959 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.294450 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.294454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:36.295418 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:36.295431 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.296357 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.296362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:36.296583 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:36.297376 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:36:36.297399 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:36:36.300214 (+ 2815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:36:36.304108 (+ 3894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:36.304114 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:36.304119 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:36.304554 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:36.304955 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:36.304960 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:36.304974 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:36.305124 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:36.305787 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:36:36.305792 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:36:36.305929 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 23:36:36.306043 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 23:36:36.306388 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":23}
I20260926 23:36:36.306945 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.290988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39910 (local address 127.0.97.61:37569)
0926 23:36:36.291730 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:36.291737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:36.291767 (+ 30us) server_negotiation.cc:407] Connection header received
0926 23:36:36.291821 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:36.291827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:36.292083 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:36.292251 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:36.293137 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.294297 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.295556 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.296216 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.296340 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:36.300501 (+ 4161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:36.300518 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:36.300539 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:36.300575 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:36.303919 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:36.304709 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:36.304718 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:36.304721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:36.304849 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:36.305231 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:36.305234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:36.305236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:36.305664 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:36.305854 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:36.306256 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 23:36:36.306510 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":561,"thread_start_us":363,"threads_started":1}
I20260926 23:36:36.310190 23379 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46169 } has no permanent_uuid. Determining permanent_uuid...
I20260926 23:36:36.311975 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.311267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46169 (local address 127.0.0.1:57440)
0926 23:36:36.311542 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:36.311691 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260926 23:36:36.312348 23330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46169: connect: Connection refused (error 111)
I20260926 23:36:36.314216 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.313803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46169 (local address 127.0.0.1:57448)
0926 23:36:36.314001 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:36.314088 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 23:36:36.314503 23379 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46169: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46169: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260926 23:36:36.323464 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
/tmp/dist-test-tasklIDZWX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46169
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37113,127.0.97.61:37569,127.0.97.60:46169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 23:36:36.337577 23305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37569 } attempt: 5
Sep 26 23:36:36 dist-test-slave-gwfx krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 23:36:36.354146 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.338616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37569 (local address 127.0.0.1:39916)
0926 23:36:36.338964 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:36.338996 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 23:36:36.339169 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:36:36.339518 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:36:36.339522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:36:36.340040 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:36:36.340228 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:36.340238 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.341262 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.341266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:36.342153 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:36.342164 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.342944 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.342950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:36.343126 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:36.343728 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:36:36.343746 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:36:36.346475 (+ 2729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:36:36.351466 (+ 4991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:36.351472 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:36.351475 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:36.351996 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:36.352509 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:36.352512 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:36.352515 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:36.352655 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:36.353342 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:36:36.353345 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:36:36.353496 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 23:36:36.353641 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 23:36:36.353868 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203}
I20260926 23:36:36.354425 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.338654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39916 (local address 127.0.97.61:37569)
0926 23:36:36.339028 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:36.339034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:36.339099 (+ 65us) server_negotiation.cc:407] Connection header received
0926 23:36:36.339302 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:36.339306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:36.339377 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:36.339463 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:36.340387 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.341105 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.342295 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.342788 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.342908 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:36.346925 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:36.346935 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:36.346941 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:36.347012 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:36.351200 (+ 4188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:36.352172 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:36.352179 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:36.352183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:36.352383 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:36.352771 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:36.352777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:36.352781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:36.353223 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:36.353394 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:36.353788 (+ 394us) server_negotiation.cc:299] Negotiation successful
0926 23:36:36.354063 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260926 23:36:36.355937 23305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46169 } has no permanent_uuid. Determining permanent_uuid...
I20260926 23:36:36.357175 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.356717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46169 (local address 127.0.0.1:57456)
0926 23:36:36.356899 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:36.357022 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 23:36:36.358458 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.358041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46169 (local address 127.0.0.1:57464)
0926 23:36:36.358238 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:36.358322 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260926 23:36:36.358719 23305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46169: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46169: connect: Connection refused (error 111)
I20260926 23:36:36.363870 23379 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46169 } attempt: 1
I20260926 23:36:36.365020 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.364556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46169 (local address 127.0.0.1:57470)
0926 23:36:36.364760 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:36.364846 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 23:36:36.366245 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.365863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46169 (local address 127.0.0.1:57476)
0926 23:36:36.366048 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:36.366132 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 23:36:36.366505 23379 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46169: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46169: connect: Connection refused (error 111)
I20260926 23:36:36.410044 23305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46169 } attempt: 1
I20260926 23:36:36.411906 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.411148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46169 (local address 127.0.0.1:57478)
0926 23:36:36.411425 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:36.411551 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260926 23:36:36.413103 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.412730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46169 (local address 127.0.0.1:57482)
0926 23:36:36.412895 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:36.412974 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 23:36:36.413331 23305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46169: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46169: connect: Connection refused (error 111)
I20260926 23:36:36.434803 23379 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46169 } attempt: 2
I20260926 23:36:36.436406 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.435775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46169 (local address 127.0.0.1:57496)
0926 23:36:36.436099 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:36.436233 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260926 23:36:36.437924 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.437444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46169 (local address 127.0.0.1:57510)
0926 23:36:36.437628 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:36.437707 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260926 23:36:36.438169 23379 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46169: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46169: connect: Connection refused (error 111)
I20260926 23:36:36.481606 23305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46169 } attempt: 2
I20260926 23:36:36.483383 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.482377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46169 (local address 127.0.0.1:34854)
0926 23:36:36.482852 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:36.483054 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260926 23:36:36.485329 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.484683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46169 (local address 127.0.0.1:34862)
0926 23:36:36.484973 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:36.485116 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260926 23:36:36.485708 23305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46169: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46169: connect: Connection refused (error 111)
I20260926 23:36:36.529441 23379 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46169 } attempt: 3
I20260926 23:36:36.530599 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.530149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46169 (local address 127.0.0.1:34870)
0926 23:36:36.530357 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:36.530440 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 23:36:36.532245 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.531634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46169 (local address 127.0.0.1:34878)
0926 23:36:36.531999 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:36.532101 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200}
W20260926 23:36:36.532482 23379 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46169: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46169: connect: Connection refused (error 111)
I20260926 23:36:36.592000 23305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46169 } attempt: 3
I20260926 23:36:36.593624 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.592899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46169 (local address 127.0.0.1:34894)
0926 23:36:36.593197 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:36.593354 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260926 23:36:36.595096 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.594510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46169 (local address 127.0.0.1:34910)
0926 23:36:36.594819 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:36.594953 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260926 23:36:36.595367 23305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46169: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46169: connect: Connection refused (error 111)
I20260926 23:36:36.675763 23379 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46169 } attempt: 4
I20260926 23:36:36.677273 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.676682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46169 (local address 127.0.0.1:34916)
0926 23:36:36.676967 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:36.677071 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260926 23:36:36.678795 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.678305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46169 (local address 127.0.0.1:34924)
0926 23:36:36.678478 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:36.678573 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 23:36:36.679070 23379 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46169: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46169: connect: Connection refused (error 111)
W20260926 23:36:36.742974 23387 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 23:36:36.743353 23387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 23:36:36.743434 23387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 23:36:36.743521 23387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 23:36:36.751128 23387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 23:36:36.751253 23387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 23:36:36.751309 23387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 23:36:36.751372 23387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 23:36:36.751430 23387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 23:36:36.761897 23387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37113,127.0.97.61:37569,127.0.97.60:46169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 22:24:36 UTC on ccycloud.quasar-qtpyyo.root.comops.site
build id 4675482
ASAN enabled
I20260926 23:36:36.763463 23387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 23:36:36.764984 23387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 23:36:36.772594 23305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46169 } attempt: 4
I20260926 23:36:36.774953 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.774206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46169 (local address 127.0.0.1:34940)
0926 23:36:36.774534 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:36.774673 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260926 23:36:36.776412 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.775865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46169 (local address 127.0.0.1:34944)
0926 23:36:36.776080 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:36.776242 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260926 23:36:36.776700 23305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46169: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46169: connect: Connection refused (error 111)
W20260926 23:36:36.781085 23393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:36.781322 23392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 23:36:36.781157 23395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 23:36:36.781926 23387 server_base.cc:1034] running on GCE node
I20260926 23:36:36.783078 23387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 23:36:36.785075 23387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 23:36:36.786413 23387 hybrid_clock.cc:648] HybridClock initialized: now 1790465796786306 us; error 64 us; skew 500 ppm
I20260926 23:36:36.786953 23387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 23:36:36 dist-test-slave-gwfx krb5kdc[23225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 23:36:36.790948 23387 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260926 23:36:36.793027 23387 webserver.cc:460] Webserver started at http://127.0.97.60:33077/ using document root <none> and password file <none>
I20260926 23:36:36.794029 23387 fs_manager.cc:359] Metadata directory not provided
I20260926 23:36:36.794167 23387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 23:36:36.794559 23387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 23:36:36.798652 23387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-2/data/instance:
uuid: "eee0b16644b54d92a39f34da23feec03"
format_stamp: "Formatted at 2026-09-26 23:36:36 on dist-test-slave-gwfx"
server_key: "701708c02dcc0db94c11dabad2dd3a3e"
server_key_iv: "9f397252dce9c24599274a02c519f1dd"
server_key_version: "encryptionkey@0"
I20260926 23:36:36.799707 23387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-2/wal/instance:
uuid: "eee0b16644b54d92a39f34da23feec03"
format_stamp: "Formatted at 2026-09-26 23:36:36 on dist-test-slave-gwfx"
server_key: "701708c02dcc0db94c11dabad2dd3a3e"
server_key_iv: "9f397252dce9c24599274a02c519f1dd"
server_key_version: "encryptionkey@0"
I20260926 23:36:36.806725 23387 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260926 23:36:36.811797 23402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 23:36:36.814041 23387 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260926 23:36:36.814325 23387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-2/data,/tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-2/wal
uuid: "eee0b16644b54d92a39f34da23feec03"
format_stamp: "Formatted at 2026-09-26 23:36:36 on dist-test-slave-gwfx"
server_key: "701708c02dcc0db94c11dabad2dd3a3e"
server_key_iv: "9f397252dce9c24599274a02c519f1dd"
server_key_version: "encryptionkey@0"
I20260926 23:36:36.814603 23387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 23:36:36.835198 23387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 23:36:36.841624 23387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 23:36:36.841972 23387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 23:36:36.864485 23387 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:46169
I20260926 23:36:36.864506 23453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:46169 every 8 connection(s)
I20260926 23:36:36.866361 23387 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-2/data/info.pb
I20260926 23:36:36.867146 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu as pid 23387
I20260926 23:36:36.867388 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklIDZWX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790465400694755-388-0/minicluster-data/master-2/wal/instance
I20260926 23:36:36.868106 388 external_mini_cluster.cc:1428] Setting key 5a3d22ea07e62793663bf090f8f71014
I20260926 23:36:36.874154 23454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 23:36:36.883638 23454 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37113 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:36:36 dist-test-slave-gwfx krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 23:36:36 dist-test-slave-gwfx krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 23:36:36.899016 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.870417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34948 (local address 127.0.97.60:46169)
0926 23:36:36.871796 (+ 1379us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:36.871818 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:36.871889 (+ 71us) server_negotiation.cc:407] Connection header received
0926 23:36:36.874611 (+ 2722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:36.874651 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:36.875318 (+ 667us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:36.875844 (+ 526us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:36.877728 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.879887 (+ 2159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.881679 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.882846 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.883098 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:36.888469 (+ 5371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:36.888482 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:36.888508 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:36.888574 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:36.892981 (+ 4407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:36.894093 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:36.894103 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:36.894115 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:36.894508 (+ 393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:36.895125 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:36.895131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:36.895135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:36.896088 (+ 953us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:36.896450 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:36.896962 (+ 512us) server_negotiation.cc:299] Negotiation successful
0926 23:36:36.897456 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1104,"spinlock_wait_cycles":78208,"thread_start_us":842,"threads_started":1}
I20260926 23:36:36.906622 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.887173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37113 (local address 127.0.0.1:48592)
0926 23:36:36.888315 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:36.888366 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 23:36:36.888573 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:36:36.889028 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:36:36.889036 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:36:36.890000 (+ 964us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:36:36.890375 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:36.890389 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.891569 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.891573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:36.892843 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:36.892860 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.893818 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.893824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:36.894028 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:36.895058 (+ 1030us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:36:36.895088 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:36:36.898388 (+ 3300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:36:36.902945 (+ 4557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:36.902955 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:36.902961 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:36.903660 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:36.904316 (+ 656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:36.904321 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:36.904324 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:36.904521 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:36.905422 (+ 901us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:36:36.905429 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:36:36.905654 (+ 225us) client_negotiation.cc:769] Sending connection context
0926 23:36:36.905911 (+ 257us) client_negotiation.cc:240] Negotiation successful
0926 23:36:36.906208 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"thread_start_us":568,"threads_started":1}
I20260926 23:36:36.906867 23458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.887172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48592 (local address 127.0.97.62:37113)
0926 23:36:36.887816 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:36.887824 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:36.888486 (+ 662us) server_negotiation.cc:407] Connection header received
0926 23:36:36.888709 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:36.888714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:36.888812 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:36.888906 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:36.890571 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.891382 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.893024 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.893595 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.893744 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:36.898731 (+ 4987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:36.898741 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:36.898746 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:36.898811 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:36.902721 (+ 3910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:36.903880 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:36.903887 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:36.903893 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:36.904160 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:36.904662 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:36.904668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:36.904671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:36.905242 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:36.905446 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:36.906079 (+ 633us) server_negotiation.cc:299] Negotiation successful
0926 23:36:36.906360 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":291,"threads_started":1}
I20260926 23:36:36.908478 23454 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37569 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:36:36 dist-test-slave-gwfx krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 26 23:36:36 dist-test-slave-gwfx krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 23:36:36.922303 23460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.903646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48606 (local address 127.0.97.62:37113)
0926 23:36:36.904380 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:36.904386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:36.904410 (+ 24us) server_negotiation.cc:407] Connection header received
0926 23:36:36.904471 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:36.904475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:36.904557 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:36.904669 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:36.906500 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.907840 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.909423 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.910193 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.910321 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:36.915937 (+ 5616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:36.915946 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:36.915951 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:36.915998 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:36.919317 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:36.920232 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:36.920237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:36.920239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:36.920387 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:36.920748 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:36.920752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:36.920754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:36.921064 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:36.921345 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:36.921643 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 23:36:36.921888 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":355,"threads_started":1}
I20260926 23:36:36.925323 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.909479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37569 (local address 127.0.0.1:34206)
0926 23:36:36.909692 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:36.909716 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:36:36.909830 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:36:36.910648 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:36:36.910655 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:36:36.911168 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:36:36.911375 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:36.911388 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.912619 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.912625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:36.914084 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:36.914101 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.915128 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.915136 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:36.915334 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:36.916002 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:36:36.916025 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:36:36.918794 (+ 2769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:36:36.922072 (+ 3278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:36.922079 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:36.922084 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:36.922598 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:36.923217 (+ 619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:36.923222 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:36.923225 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:36.923425 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:36.924163 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:36:36.924168 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:36:36.924364 (+ 196us) client_negotiation.cc:769] Sending connection context
0926 23:36:36.924521 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 23:36:36.924889 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 23:36:36.925520 23461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.909406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34206 (local address 127.0.97.61:37569)
0926 23:36:36.910300 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:36.910306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:36.910324 (+ 18us) server_negotiation.cc:407] Connection header received
0926 23:36:36.910385 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:36.910390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:36.910461 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:36.910575 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:36.911649 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.912430 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.914306 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.914923 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.915042 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:36.919053 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:36.919059 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:36.919063 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:36.919120 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:36.921896 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:36.922821 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:36.922826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:36.922829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:36.923045 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:36.923575 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:36.923578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:36.923580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:36.923953 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:36.924097 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:36.924691 (+ 594us) server_negotiation.cc:299] Negotiation successful
0926 23:36:36.924951 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774,"thread_start_us":613,"threads_started":1}
I20260926 23:36:36.927047 23454 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46169 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 23:36:36 dist-test-slave-gwfx krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 26 23:36:36 dist-test-slave-gwfx krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 23:36:36.942793 23462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.924570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34212 (local address 127.0.97.61:37569)
0926 23:36:36.925551 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:36.925557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:36.925580 (+ 23us) server_negotiation.cc:407] Connection header received
0926 23:36:36.925631 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:36.925635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:36.925711 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:36.925839 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:36.927567 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.928734 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.930120 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.930565 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.930679 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:36.936362 (+ 5683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:36.936372 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:36.936376 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:36.936429 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:36.939608 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:36.940523 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:36.940527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:36.940530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:36.940670 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:36.941066 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:36.941070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:36.941095 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:36.941425 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:36.941694 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:36.942043 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 23:36:36.942315 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":829,"thread_start_us":657,"threads_started":1}
I20260926 23:36:36.942920 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.927932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46169 (local address 127.0.0.1:34956)
0926 23:36:36.928132 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:36.928155 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 23:36:36.928251 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:36:36.928673 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:36:36.928676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:36:36.929103 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:36:36.929267 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:36.929274 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.930281 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.930287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:36.931295 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:36.931310 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.932456 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.932462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:36.932669 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:36.933309 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:36:36.933328 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:36:36.936269 (+ 2941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:36:36.939849 (+ 3580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:36.939853 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:36.939857 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:36.940309 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:36.940772 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:36.940777 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:36.940781 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:36.940934 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:36.941583 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:36:36.941588 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:36:36.941869 (+ 281us) client_negotiation.cc:769] Sending connection context
0926 23:36:36.942106 (+ 237us) client_negotiation.cc:240] Negotiation successful
0926 23:36:36.942478 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 23:36:36.943413 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.928088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34956 (local address 127.0.97.60:46169)
0926 23:36:36.928343 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:36.928349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:36.928369 (+ 20us) server_negotiation.cc:407] Connection header received
0926 23:36:36.928428 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:36.928433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:36.928515 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:36.928629 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:36.929411 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.930160 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.931508 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.932196 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.932357 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:36.936580 (+ 4223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:36.936586 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:36.936590 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:36.936632 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:36.939597 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:36.940459 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:36.940466 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:36.940470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:36.940641 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:36.941051 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:36.941056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:36.941060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:36.941447 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:36.941599 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:36.942242 (+ 643us) server_negotiation.cc:299] Negotiation successful
0926 23:36:36.942821 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":230,"server-negotiator.queue_time_us":83}
I20260926 23:36:36.949939 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 23:36:36.950992 23454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eee0b16644b54d92a39f34da23feec03: Bootstrap starting.
I20260926 23:36:36.956275 23454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eee0b16644b54d92a39f34da23feec03: Neither blocks nor log segments found. Creating new log.
I20260926 23:36:36.952795 23280 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9a74a76eaa244e6e8d532141d2ed650a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48606
I20260926 23:36:36.958151 23454 log.cc:824] T 00000000000000000000000000000000 P eee0b16644b54d92a39f34da23feec03: Log is configured to *not* fsync() on all Append() calls
I20260926 23:36:36.962849 23454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eee0b16644b54d92a39f34da23feec03: No bootstrap required, opened a new log
I20260926 23:36:36.975970 23454 raft_consensus.cc:348] T 00000000000000000000000000000000 P eee0b16644b54d92a39f34da23feec03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a74a76eaa244e6e8d532141d2ed650a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37113 } } peers { permanent_uuid: "b8e1be79d4864a9eae6eb37f8db979e4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37569 } } peers { permanent_uuid: "eee0b16644b54d92a39f34da23feec03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46169 } }
I20260926 23:36:36.976578 23454 raft_consensus.cc:729] T 00000000000000000000000000000000 P eee0b16644b54d92a39f34da23feec03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eee0b16644b54d92a39f34da23feec03, State: Initialized, Role: FOLLOWER
I20260926 23:36:36.977284 23454 consensus_queue.cc:260] T 00000000000000000000000000000000 P eee0b16644b54d92a39f34da23feec03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a74a76eaa244e6e8d532141d2ed650a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37113 } } peers { permanent_uuid: "b8e1be79d4864a9eae6eb37f8db979e4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37569 } } peers { permanent_uuid: "eee0b16644b54d92a39f34da23feec03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46169 } }
I20260926 23:36:36.978353 23379 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46169 } attempt: 5
I20260926 23:36:36.978866 23464 sys_catalog.cc:455] T 00000000000000000000000000000000 P eee0b16644b54d92a39f34da23feec03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a74a76eaa244e6e8d532141d2ed650a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37113 } } peers { permanent_uuid: "b8e1be79d4864a9eae6eb37f8db979e4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37569 } } peers { permanent_uuid: "eee0b16644b54d92a39f34da23feec03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46169 } } }
I20260926 23:36:36.979449 23464 sys_catalog.cc:458] T 00000000000000000000000000000000 P eee0b16644b54d92a39f34da23feec03 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:36:36.980489 23454 sys_catalog.cc:564] T 00000000000000000000000000000000 P eee0b16644b54d92a39f34da23feec03 [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 23:36:36 dist-test-slave-gwfx krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260926 23:36:36.994956 23475 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eee0b16644b54d92a39f34da23feec03: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 23:36:36.995149 23475 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 23:36:36.998706 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.979218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46169 (local address 127.0.0.1:34962)
0926 23:36:36.979543 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:36.979567 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 23:36:36.979691 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:36:36.980171 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:36:36.980177 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:36:36.980710 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:36:36.980953 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:36.980967 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.982616 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.982622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:36.983798 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:36.983814 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.984906 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.984913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:36.985160 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:36.986015 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:36:36.986043 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:36:36.989050 (+ 3007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:36:36.995124 (+ 6074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:36.995131 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:36.995136 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:36.995725 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:36.996230 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:36.996243 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:36.996248 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:36.996435 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:36.997333 (+ 898us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:36:36.997340 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:36:36.997605 (+ 265us) client_negotiation.cc:769] Sending connection context
0926 23:36:36.997809 (+ 204us) client_negotiation.cc:240] Negotiation successful
0926 23:36:36.998219 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173}
I20260926 23:36:36.999212 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:36.979360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34962 (local address 127.0.97.60:46169)
0926 23:36:36.979586 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:36.979594 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:36.979679 (+ 85us) server_negotiation.cc:407] Connection header received
0926 23:36:36.979872 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:36.979879 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:36.979981 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:36.980112 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:36.981152 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.982416 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.983959 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:36.984681 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:36.984865 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:36.989388 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:36.989395 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:36.989399 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:36.989445 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:36.994800 (+ 5355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:36.995889 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:36.995895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:36.995898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:36.996090 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:36.996580 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:36.996586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:36.996589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:36.997157 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:36.997380 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:36.998046 (+ 666us) server_negotiation.cc:299] Negotiation successful
0926 23:36:36.998564 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 23:36:36.989663 23387 master_runner.cc:186] Error getting master registration for 127.0.97.62:37113: OK, instance_id { permanent_uuid: "9a74a76eaa244e6e8d532141d2ed650a" instance_seqno: 1790465795607221 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 23:36:37.004446 23387 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 23:36:37.006265 23379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8e1be79d4864a9eae6eb37f8db979e4: Bootstrap starting.
I20260926 23:36:37.012219 23379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8e1be79d4864a9eae6eb37f8db979e4: Neither blocks nor log segments found. Creating new log.
I20260926 23:36:37.014087 23379 log.cc:824] T 00000000000000000000000000000000 P b8e1be79d4864a9eae6eb37f8db979e4: Log is configured to *not* fsync() on all Append() calls
I20260926 23:36:37.019748 23379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8e1be79d4864a9eae6eb37f8db979e4: No bootstrap required, opened a new log
I20260926 23:36:37.034111 23379 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8e1be79d4864a9eae6eb37f8db979e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a74a76eaa244e6e8d532141d2ed650a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37113 } } peers { permanent_uuid: "b8e1be79d4864a9eae6eb37f8db979e4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37569 } } peers { permanent_uuid: "eee0b16644b54d92a39f34da23feec03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46169 } }
I20260926 23:36:37.034762 23379 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8e1be79d4864a9eae6eb37f8db979e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8e1be79d4864a9eae6eb37f8db979e4, State: Initialized, Role: FOLLOWER
I20260926 23:36:37.035485 23379 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8e1be79d4864a9eae6eb37f8db979e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a74a76eaa244e6e8d532141d2ed650a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37113 } } peers { permanent_uuid: "b8e1be79d4864a9eae6eb37f8db979e4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37569 } } peers { permanent_uuid: "eee0b16644b54d92a39f34da23feec03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46169 } }
I20260926 23:36:37.036868 23478 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8e1be79d4864a9eae6eb37f8db979e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a74a76eaa244e6e8d532141d2ed650a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37113 } } peers { permanent_uuid: "b8e1be79d4864a9eae6eb37f8db979e4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37569 } } peers { permanent_uuid: "eee0b16644b54d92a39f34da23feec03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46169 } } }
I20260926 23:36:37.037489 23478 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8e1be79d4864a9eae6eb37f8db979e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:36:37.038589 23379 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8e1be79d4864a9eae6eb37f8db979e4 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 23:36:37.051851 23489 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b8e1be79d4864a9eae6eb37f8db979e4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 23:36:37.052212 23489 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 23:36:37.053959 23305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46169 } attempt: 5
I20260926 23:36:37.048897 23312 master_runner.cc:186] Error getting master registration for 127.0.97.62:37113: OK, instance_id { permanent_uuid: "9a74a76eaa244e6e8d532141d2ed650a" instance_seqno: 1790465795607221 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 23:36:37.063558 23312 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 26 23:36:37 dist-test-slave-gwfx krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790465795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 23:36:37.076284 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:37.054920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46169 (local address 127.0.0.1:34964)
0926 23:36:37.055393 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0926 23:36:37.055415 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 23:36:37.055548 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 23:36:37.056849 (+ 1301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 23:36:37.056856 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 23:36:37.057481 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0926 23:36:37.057766 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:37.057781 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:37.059661 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:37.059668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:37.061189 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 23:36:37.061211 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 23:36:37.062652 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 23:36:37.062659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 23:36:37.062934 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:37.063899 (+ 965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 23:36:37.063927 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 23:36:37.067418 (+ 3491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 23:36:37.072564 (+ 5146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:37.072571 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:37.072575 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:37.073246 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:37.073852 (+ 606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 23:36:37.073858 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 23:36:37.073862 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 23:36:37.074129 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 23:36:37.075005 (+ 876us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 23:36:37.075010 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 23:36:37.075230 (+ 220us) client_negotiation.cc:769] Sending connection context
0926 23:36:37.075406 (+ 176us) client_negotiation.cc:240] Negotiation successful
0926 23:36:37.075789 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322}
I20260926 23:36:37.077540 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:37.055785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34964 (local address 127.0.97.60:46169)
0926 23:36:37.056130 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:37.056139 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:37.056170 (+ 31us) server_negotiation.cc:407] Connection header received
0926 23:36:37.056263 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:37.056270 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:37.056420 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:37.056578 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:37.058050 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:37.059417 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:37.061446 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:37.062235 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:37.062376 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:37.067796 (+ 5420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:37.067805 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:37.067811 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:37.067868 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:37.072247 (+ 4379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:37.073464 (+ 1217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:37.073472 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:37.073476 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:37.073686 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:37.074280 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:37.074286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:37.074289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:37.074754 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:37.074933 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:37.076791 (+ 1858us) server_negotiation.cc:299] Negotiation successful
0926 23:36:37.077127 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260926 23:36:37.084671 23305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a74a76eaa244e6e8d532141d2ed650a: Bootstrap starting.
I20260926 23:36:37.089736 23305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a74a76eaa244e6e8d532141d2ed650a: Neither blocks nor log segments found. Creating new log.
I20260926 23:36:37.091378 23305 log.cc:824] T 00000000000000000000000000000000 P 9a74a76eaa244e6e8d532141d2ed650a: Log is configured to *not* fsync() on all Append() calls
W20260926 23:36:37.096530 23408 tablet.cc:2369] T 00000000000000000000000000000000 P eee0b16644b54d92a39f34da23feec03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:36:37.097210 23305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a74a76eaa244e6e8d532141d2ed650a: No bootstrap required, opened a new log
I20260926 23:36:37.105930 23305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a74a76eaa244e6e8d532141d2ed650a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a74a76eaa244e6e8d532141d2ed650a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37113 } } peers { permanent_uuid: "b8e1be79d4864a9eae6eb37f8db979e4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37569 } } peers { permanent_uuid: "eee0b16644b54d92a39f34da23feec03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46169 } }
I20260926 23:36:37.106446 23305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a74a76eaa244e6e8d532141d2ed650a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a74a76eaa244e6e8d532141d2ed650a, State: Initialized, Role: FOLLOWER
I20260926 23:36:37.107162 23305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a74a76eaa244e6e8d532141d2ed650a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a74a76eaa244e6e8d532141d2ed650a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37113 } } peers { permanent_uuid: "b8e1be79d4864a9eae6eb37f8db979e4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37569 } } peers { permanent_uuid: "eee0b16644b54d92a39f34da23feec03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46169 } }
I20260926 23:36:37.109081 23492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a74a76eaa244e6e8d532141d2ed650a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a74a76eaa244e6e8d532141d2ed650a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37113 } } peers { permanent_uuid: "b8e1be79d4864a9eae6eb37f8db979e4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37569 } } peers { permanent_uuid: "eee0b16644b54d92a39f34da23feec03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46169 } } }
I20260926 23:36:37.110208 23305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a74a76eaa244e6e8d532141d2ed650a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 23:36:37.110347 23280 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a74a76eaa244e6e8d532141d2ed650a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 23:36:37.110634 23280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a74a76eaa244e6e8d532141d2ed650a [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:36:37.115268 23280 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a74a76eaa244e6e8d532141d2ed650a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a74a76eaa244e6e8d532141d2ed650a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37113 } } peers { permanent_uuid: "b8e1be79d4864a9eae6eb37f8db979e4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37569 } } peers { permanent_uuid: "eee0b16644b54d92a39f34da23feec03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46169 } }
I20260926 23:36:37.119231 23354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9a74a76eaa244e6e8d532141d2ed650a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b8e1be79d4864a9eae6eb37f8db979e4"
I20260926 23:36:37.119947 23354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8e1be79d4864a9eae6eb37f8db979e4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:36:37.124756 23354 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b8e1be79d4864a9eae6eb37f8db979e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a74a76eaa244e6e8d532141d2ed650a in term 1.
I20260926 23:36:37.125999 23257 leader_election.cc:302] T 00000000000000000000000000000000 P 9a74a76eaa244e6e8d532141d2ed650a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a74a76eaa244e6e8d532141d2ed650a, b8e1be79d4864a9eae6eb37f8db979e4; no voters:
I20260926 23:36:37.131348 23429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9a74a76eaa244e6e8d532141d2ed650a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "eee0b16644b54d92a39f34da23feec03"
I20260926 23:36:37.132071 23429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eee0b16644b54d92a39f34da23feec03 [term 0 FOLLOWER]: Advancing to term 1
I20260926 23:36:37.136528 23429 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eee0b16644b54d92a39f34da23feec03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a74a76eaa244e6e8d532141d2ed650a in term 1.
I20260926 23:36:37.137980 23280 leader_election.cc:288] T 00000000000000000000000000000000 P 9a74a76eaa244e6e8d532141d2ed650a [CANDIDATE]: Term 1 election: Requested vote from peers b8e1be79d4864a9eae6eb37f8db979e4 (127.0.97.61:37569), eee0b16644b54d92a39f34da23feec03 (127.0.97.60:46169)
I20260926 23:36:37.140517 23492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a74a76eaa244e6e8d532141d2ed650a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:36:37.141124 23492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a74a76eaa244e6e8d532141d2ed650a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 23:36:37.141738 23492 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a74a76eaa244e6e8d532141d2ed650a [term 1 LEADER]: Becoming Leader. State: Replica: 9a74a76eaa244e6e8d532141d2ed650a, State: Running, Role: LEADER
I20260926 23:36:37.143631 23492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a74a76eaa244e6e8d532141d2ed650a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a74a76eaa244e6e8d532141d2ed650a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37113 } } peers { permanent_uuid: "b8e1be79d4864a9eae6eb37f8db979e4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37569 } } peers { permanent_uuid: "eee0b16644b54d92a39f34da23feec03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46169 } }
I20260926 23:36:37.153716 23494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a74a76eaa244e6e8d532141d2ed650a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a74a76eaa244e6e8d532141d2ed650a. Latest consensus state: current_term: 1 leader_uuid: "9a74a76eaa244e6e8d532141d2ed650a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a74a76eaa244e6e8d532141d2ed650a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37113 } } peers { permanent_uuid: "b8e1be79d4864a9eae6eb37f8db979e4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37569 } } peers { permanent_uuid: "eee0b16644b54d92a39f34da23feec03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46169 } } }
I20260926 23:36:37.154389 23494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a74a76eaa244e6e8d532141d2ed650a [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:37.158851 23501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 23:36:37.184293 23501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 23:36:37.209506 23460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:37.177498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48614 (local address 127.0.97.62:37113)
0926 23:36:37.178058 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:37.178064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:37.178247 (+ 183us) server_negotiation.cc:407] Connection header received
0926 23:36:37.178626 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:37.178634 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:37.178792 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:37.178896 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:37.181436 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:37.190872 (+ 9436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:37.192952 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:37.193637 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:37.194114 (+ 477us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:37.198780 (+ 4666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:37.198788 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:37.198793 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:37.198865 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:37.202957 (+ 4092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:37.204023 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:37.204030 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:37.204033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:37.204208 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:37.207203 (+ 2995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:37.207210 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:37.207214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:37.207687 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:37.207982 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:37.208808 (+ 826us) server_negotiation.cc:299] Negotiation successful
0926 23:36:37.209113 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 23:36:37.222436 23429 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eee0b16644b54d92a39f34da23feec03 [term 1 FOLLOWER]: Refusing update from remote peer 9a74a76eaa244e6e8d532141d2ed650a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260926 23:36:37.222510 23333 tablet.cc:2369] T 00000000000000000000000000000000 P b8e1be79d4864a9eae6eb37f8db979e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 23:36:37.222870 23354 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b8e1be79d4864a9eae6eb37f8db979e4 [term 1 FOLLOWER]: Refusing update from remote peer 9a74a76eaa244e6e8d532141d2ed650a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 23:36:37.224396 23492 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a74a76eaa244e6e8d532141d2ed650a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eee0b16644b54d92a39f34da23feec03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:36:37.230403 23462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:37.178288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34220 (local address 127.0.97.61:37569)
0926 23:36:37.178510 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:37.178515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:37.182167 (+ 3652us) server_negotiation.cc:407] Connection header received
0926 23:36:37.182419 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:37.182425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:37.182517 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:37.182593 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:37.185868 (+ 3275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:37.187101 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:37.189039 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:37.189790 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:37.189900 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:37.209962 (+ 20062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:37.209969 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:37.209974 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:37.210025 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:37.214624 (+ 4599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:37.227639 (+ 13015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:37.227646 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:37.227650 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:37.227856 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:37.228391 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:37.228396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:37.228400 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:37.228829 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:37.229099 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:37.229702 (+ 603us) server_negotiation.cc:299] Negotiation successful
0926 23:36:37.230001 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 23:36:37.231230 23492 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a74a76eaa244e6e8d532141d2ed650a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8e1be79d4864a9eae6eb37f8db979e4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 23:36:37.242440 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:37.176886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34978 (local address 127.0.97.60:46169)
0926 23:36:37.177244 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:37.177252 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:37.181351 (+ 4099us) server_negotiation.cc:407] Connection header received
0926 23:36:37.182235 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:37.182240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:37.182338 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:37.182412 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:37.206948 (+ 24536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:37.208055 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:37.210520 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:37.211185 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:37.211343 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:37.218734 (+ 7391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 23:36:37.218742 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 23:36:37.218748 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 23:36:37.218807 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 23:36:37.226146 (+ 7339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:37.233213 (+ 7067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:37.233221 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:37.233225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:37.233441 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 23:36:37.239260 (+ 5819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 23:36:37.239269 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 23:36:37.239274 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 23:36:37.240367 (+ 1093us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 23:36:37.240874 (+ 507us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:37.241664 (+ 790us) server_negotiation.cc:299] Negotiation successful
0926 23:36:37.242003 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260926 23:36:37.284243 23478 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8e1be79d4864a9eae6eb37f8db979e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a74a76eaa244e6e8d532141d2ed650a. Latest consensus state: current_term: 1 leader_uuid: "9a74a76eaa244e6e8d532141d2ed650a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a74a76eaa244e6e8d532141d2ed650a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37113 } } peers { permanent_uuid: "b8e1be79d4864a9eae6eb37f8db979e4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37569 } } peers { permanent_uuid: "eee0b16644b54d92a39f34da23feec03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46169 } } }
I20260926 23:36:37.284734 23478 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8e1be79d4864a9eae6eb37f8db979e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:36:37.310429 23513 mvcc.cc:205] Tried to move back new op lower bound from 7333747905402253312 to 7333747905115209728. Current Snapshot: MvccSnapshot[applied={T|T < 7333747905402253312}]
I20260926 23:36:37.310542 23516 mvcc.cc:205] Tried to move back new op lower bound from 7333747905402253312 to 7333747905115209728. Current Snapshot: MvccSnapshot[applied={T|T < 7333747905402253312}]
I20260926 23:36:37.313860 23464 sys_catalog.cc:455] T 00000000000000000000000000000000 P eee0b16644b54d92a39f34da23feec03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a74a76eaa244e6e8d532141d2ed650a. Latest consensus state: current_term: 1 leader_uuid: "9a74a76eaa244e6e8d532141d2ed650a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a74a76eaa244e6e8d532141d2ed650a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37113 } } peers { permanent_uuid: "b8e1be79d4864a9eae6eb37f8db979e4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37569 } } peers { permanent_uuid: "eee0b16644b54d92a39f34da23feec03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46169 } } }
I20260926 23:36:37.314462 23464 sys_catalog.cc:458] T 00000000000000000000000000000000 P eee0b16644b54d92a39f34da23feec03 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:36:37.319250 23492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a74a76eaa244e6e8d532141d2ed650a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a74a76eaa244e6e8d532141d2ed650a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a74a76eaa244e6e8d532141d2ed650a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37113 } } peers { permanent_uuid: "b8e1be79d4864a9eae6eb37f8db979e4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37569 } } peers { permanent_uuid: "eee0b16644b54d92a39f34da23feec03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46169 } } }
I20260926 23:36:37.320307 23492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a74a76eaa244e6e8d532141d2ed650a [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:37.329875 23464 sys_catalog.cc:455] T 00000000000000000000000000000000 P eee0b16644b54d92a39f34da23feec03 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a74a76eaa244e6e8d532141d2ed650a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a74a76eaa244e6e8d532141d2ed650a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37113 } } peers { permanent_uuid: "b8e1be79d4864a9eae6eb37f8db979e4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37569 } } peers { permanent_uuid: "eee0b16644b54d92a39f34da23feec03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46169 } } }
I20260926 23:36:37.330214 23464 sys_catalog.cc:458] T 00000000000000000000000000000000 P eee0b16644b54d92a39f34da23feec03 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:36:37.331151 23515 mvcc.cc:205] Tried to move back new op lower bound from 7333747905402253312 to 7333747905115209728. Current Snapshot: MvccSnapshot[applied={T|T < 7333747905402253312}]
I20260926 23:36:37.333407 23478 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8e1be79d4864a9eae6eb37f8db979e4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a74a76eaa244e6e8d532141d2ed650a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a74a76eaa244e6e8d532141d2ed650a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37113 } } peers { permanent_uuid: "b8e1be79d4864a9eae6eb37f8db979e4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37569 } } peers { permanent_uuid: "eee0b16644b54d92a39f34da23feec03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46169 } } }
I20260926 23:36:37.333735 23478 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8e1be79d4864a9eae6eb37f8db979e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 23:36:37.340780 23501 catalog_manager.cc:1347] Generated new cluster ID: 90fc9cb4dc1e478fa5b6a81ed2f01193
I20260926 23:36:37.340921 23501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 23:36:37.340226 23494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a74a76eaa244e6e8d532141d2ed650a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a74a76eaa244e6e8d532141d2ed650a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a74a76eaa244e6e8d532141d2ed650a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37113 } } peers { permanent_uuid: "b8e1be79d4864a9eae6eb37f8db979e4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37569 } } peers { permanent_uuid: "eee0b16644b54d92a39f34da23feec03" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46169 } } }
I20260926 23:36:37.358454 23494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a74a76eaa244e6e8d532141d2ed650a [sys.catalog]: This master's current role is: LEADER
I20260926 23:36:37.418022 23501 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 23:36:37.420534 23501 catalog_manager.cc:1504] Loading token signing keys...
I20260926 23:36:37.450214 23501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a74a76eaa244e6e8d532141d2ed650a: Generated new TSK 0
I20260926 23:36:37.451584 23501 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 23:36:37.609795 23460 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:36:37.582597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48622 (local address 127.0.97.62:37113)
0926 23:36:37.582877 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:37.582883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:37.604274 (+ 21391us) server_negotiation.cc:407] Connection header received
0926 23:36:37.605900 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:37.605907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:37.606022 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:37.606105 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:37.609453 (+ 3348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48622: BlockingRecv error: recv got EOF from 127.0.0.1:48622 (error 108)
Metrics: {"server-negotiator.queue_time_us":140}
W20260926 23:36:37.615780 23462 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:36:37.601831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34232 (local address 127.0.97.61:37569)
0926 23:36:37.602047 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:37.602053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:37.612923 (+ 10870us) server_negotiation.cc:407] Connection header received
0926 23:36:37.613659 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:37.613665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:37.613781 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:37.613861 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:37.615499 (+ 1638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34232: BlockingRecv error: recv got EOF from 127.0.0.1:34232 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260926 23:36:37.618289 23457 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:36:37.589122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34984 (local address 127.0.97.60:46169)
0926 23:36:37.589337 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:37.589345 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:37.612249 (+ 22904us) server_negotiation.cc:407] Connection header received
0926 23:36:37.614175 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:37.614180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:37.614285 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:37.614363 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:37.617997 (+ 3634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34984: BlockingRecv error: recv got EOF from 127.0.0.1:34984 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 23:36:37.662452 23462 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:36:37.652268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34248 (local address 127.0.97.61:37569)
0926 23:36:37.652484 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:37.652491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:37.659592 (+ 7101us) server_negotiation.cc:407] Connection header received
0926 23:36:37.660615 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:37.660621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:37.660792 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:37.660894 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:37.662187 (+ 1293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34248: BlockingRecv error: recv got EOF from 127.0.0.1:34248 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260926 23:36:37.709664 23462 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:36:37.699294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34250 (local address 127.0.97.61:37569)
0926 23:36:37.699747 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:37.699756 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:37.707112 (+ 7356us) server_negotiation.cc:407] Connection header received
0926 23:36:37.707797 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:37.707806 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:37.707930 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:37.708016 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:37.709334 (+ 1318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34250: BlockingRecv error: recv got EOF from 127.0.0.1:34250 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
W20260926 23:36:37.742097 23462 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:36:37.738732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34254 (local address 127.0.97.61:37569)
0926 23:36:37.738961 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:37.738970 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:37.739710 (+ 740us) server_negotiation.cc:407] Connection header received
0926 23:36:37.740120 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:37.740128 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:37.740251 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:37.740350 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:37.741788 (+ 1438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34254: BlockingRecv error: recv got EOF from 127.0.0.1:34254 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260926 23:36:37.755076 23462 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:36:37.751740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34270 (local address 127.0.97.61:37569)
0926 23:36:37.752013 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:37.752020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:37.752688 (+ 668us) server_negotiation.cc:407] Connection header received
0926 23:36:37.753146 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:37.753156 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:37.753359 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:37.753466 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:37.754776 (+ 1310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34270: BlockingRecv error: recv got EOF from 127.0.0.1:34270 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
W20260926 23:36:37.771688 23462 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:36:37.768776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34284 (local address 127.0.97.61:37569)
0926 23:36:37.768996 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:37.769003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:37.769639 (+ 636us) server_negotiation.cc:407] Connection header received
0926 23:36:37.769977 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:37.769984 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:37.770114 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:37.770195 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:37.771411 (+ 1216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34284: BlockingRecv error: recv got EOF from 127.0.0.1:34284 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260926 23:36:37.795495 23462 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:36:37.793044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34298 (local address 127.0.97.61:37569)
0926 23:36:37.793303 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:37.793313 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:37.793862 (+ 549us) server_negotiation.cc:407] Connection header received
0926 23:36:37.794110 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:37.794115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:37.794203 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:37.794292 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:37.795167 (+ 875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34298: BlockingRecv error: recv got EOF from 127.0.0.1:34298 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260926 23:36:37.834275 23462 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:36:37.831494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34300 (local address 127.0.97.61:37569)
0926 23:36:37.831778 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:37.831788 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:37.832372 (+ 584us) server_negotiation.cc:407] Connection header received
0926 23:36:37.832657 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:37.832663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:37.832770 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:37.832892 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:37.833940 (+ 1048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34300: BlockingRecv error: recv got EOF from 127.0.0.1:34300 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260926 23:36:37.906294 23462 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:36:37.903757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34310 (local address 127.0.97.61:37569)
0926 23:36:37.904072 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:37.904079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:37.904436 (+ 357us) server_negotiation.cc:407] Connection header received
0926 23:36:37.904636 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:37.904640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:37.904792 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:37.904899 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:37.905990 (+ 1091us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34310: BlockingRecv error: recv got EOF from 127.0.0.1:34310 (error 108)
Metrics: {"server-negotiator.queue_time_us":139}
I20260926 23:36:37.997848 23475 catalog_manager.cc:1259] Loaded cluster ID: 90fc9cb4dc1e478fa5b6a81ed2f01193
I20260926 23:36:37.997987 23475 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eee0b16644b54d92a39f34da23feec03: loading cluster ID for follower catalog manager: success
I20260926 23:36:38.001134 23475 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eee0b16644b54d92a39f34da23feec03: acquiring CA information for follower catalog manager: success
I20260926 23:36:38.003715 23475 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eee0b16644b54d92a39f34da23feec03: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 23:36:38.041406 23462 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 23:36:38.039176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34326 (local address 127.0.97.61:37569)
0926 23:36:38.039368 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:38.039374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:38.039785 (+ 411us) server_negotiation.cc:407] Connection header received
0926 23:36:38.040014 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:38.040018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:38.040110 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:38.040281 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0926 23:36:38.041130 (+ 849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34326: BlockingRecv error: recv got EOF from 127.0.0.1:34326 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 23:36:38.054852 23489 catalog_manager.cc:1259] Loaded cluster ID: 90fc9cb4dc1e478fa5b6a81ed2f01193
I20260926 23:36:38.054986 23489 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b8e1be79d4864a9eae6eb37f8db979e4: loading cluster ID for follower catalog manager: success
I20260926 23:36:38.057869 23489 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b8e1be79d4864a9eae6eb37f8db979e4: acquiring CA information for follower catalog manager: success
I20260926 23:36:38.060196 23489 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b8e1be79d4864a9eae6eb37f8db979e4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 23:36:38.309417 23462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:38.301801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34332 (local address 127.0.97.61:37569)
0926 23:36:38.302027 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:38.302032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:38.302314 (+ 282us) server_negotiation.cc:407] Connection header received
0926 23:36:38.302565 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:38.302570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:38.302837 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:38.302970 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:36:38.303619 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:38.304708 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:38.306353 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:38.306899 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:38.307029 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:38.307810 (+ 781us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:36:38.308595 (+ 785us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:36:38.308742 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:38.308931 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 23:36:38.309098 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 23:36:38.326220 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:38.316850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34992 (local address 127.0.97.60:46169)
0926 23:36:38.319283 (+ 2433us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:38.319289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:38.319313 (+ 24us) server_negotiation.cc:407] Connection header received
0926 23:36:38.319451 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:38.319456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:38.319812 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:38.320052 (+ 240us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:36:38.320621 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:38.321647 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:38.323429 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:38.323939 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:38.324076 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:38.324718 (+ 642us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:36:38.325445 (+ 727us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:36:38.325597 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:38.325711 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 23:36:38.325823 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2293,"thread_start_us":2124,"threads_started":1}
I20260926 23:36:38.343689 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:38.334163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35000 (local address 127.0.97.60:46169)
0926 23:36:38.334381 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:38.334389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:38.336027 (+ 1638us) server_negotiation.cc:407] Connection header received
0926 23:36:38.336621 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:38.336626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:38.336729 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:38.336829 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:36:38.338044 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:38.338992 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:38.341134 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:38.342115 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:38.342263 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:38.342834 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:36:38.342978 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:36:38.343089 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:38.343259 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 23:36:38.343357 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 23:36:38.343685 23462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:38.334026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34344 (local address 127.0.97.61:37569)
0926 23:36:38.334252 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:38.334258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:38.335821 (+ 1563us) server_negotiation.cc:407] Connection header received
0926 23:36:38.336152 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:38.336159 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:38.336271 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:38.336384 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:36:38.336953 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:38.338201 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:38.340790 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:38.341406 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:38.341858 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:38.342667 (+ 809us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:36:38.342890 (+ 223us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:36:38.342987 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:38.343196 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 23:36:38.343310 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 23:36:38.344420 23595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 23:36:38.334026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48628 (local address 127.0.97.62:37113)
0926 23:36:38.334653 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0926 23:36:38.334659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 23:36:38.335446 (+ 787us) server_negotiation.cc:407] Connection header received
0926 23:36:38.335917 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 23:36:38.335923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 23:36:38.336197 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 23:36:38.336391 (+ 194us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 23:36:38.337269 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:38.338298 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:38.340737 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 23:36:38.341632 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 23:36:38.341783 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 23:36:38.342793 (+ 1010us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 23:36:38.343380 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 23:36:38.343498 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 23:36:38.343770 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 23:36:38.344080 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":361,"threads_started":1}
I20260926 23:36:38.350585 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 23238
I20260926 23:36:38.352115 23300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:36:38.480792 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 23238
W20260926 23:36:38.508930 23331 connection.cc:535] server connection from 127.0.0.1:39916 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 23:36:38.509373 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 23312
I20260926 23:36:38.510874 23374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:36:38.635787 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 23312
I20260926 23:36:38.657631 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu pid 23387
I20260926 23:36:38.659020 23449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 23:36:38.781279 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklIDZWX/build/asan/bin/kudu with pid 23387
2026-09-26T23:36:38Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5103 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9992 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (398033 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST