Diagnosed failure

MultiThreadedRpcTest.ReactorMaxIOLatencyStatsEnabled: Unrecognized error type. Please see the error log for more information.

Full log

[==========] Running 8 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 8 tests from MultiThreadedRpcTest
[ RUN      ] MultiThreadedRpcTest.TestShutdownDuringService
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260809 13:45:42.690347 16359 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.211s	user 0.459s	sys 0.749s
W20260809 13:45:42.690985 16359 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.212s	user 0.459s	sys 0.752s
I20260809 13:45:42.699251 16391 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 0.0.0.0:35005 every 8 connection(s)
I20260809 13:45:42.701227 16392 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 0.0.0.0:35005 every 8 connection(s)
I20260809 13:45:42.809653 16397 mt-rpc-test.cc:111] Connecting to 0.0.0.0:35005
I20260809 13:45:42.847951 16400 mt-rpc-test.cc:111] Connecting to 0.0.0.0:35005
I20260809 13:45:42.855584 16400 mt-rpc-test.cc:121] Call failed. Shutting down client thread. Ran 1 calls: Network error: Client connection negotiation failed: client connection to 0.0.0.0:35005: connect: Connection refused (error 111)
I20260809 13:45:42.891901 16404 mt-rpc-test.cc:111] Connecting to 0.0.0.0:35005
I20260809 13:45:42.894201 16409 mt-rpc-test.cc:111] Connecting to 0.0.0.0:35005
I20260809 13:45:42.894486 16404 mt-rpc-test.cc:121] Call failed. Shutting down client thread. Ran 1 calls: Network error: Client connection negotiation failed: client connection to 0.0.0.0:35005: connect: Connection refused (error 111)
I20260809 13:45:42.898077 16409 mt-rpc-test.cc:121] Call failed. Shutting down client thread. Ran 1 calls: Network error: Client connection negotiation failed: client connection to 0.0.0.0:35005: connect: Connection refused (error 111)
W20260809 13:45:42.906442 16387 connection.cc:466] server connection from 127.0.0.1:39872 torn down before Call kudu.rpc.GenericCalculatorService.Add from 127.0.0.1:39872 (request call id 0) could send its response
I20260809 13:45:42.907768 16397 mt-rpc-test.cc:121] Call failed. Shutting down client thread. Ran 1 calls: Network error: recv got EOF from 127.0.0.1:35005 (error 108)
W20260809 13:45:42.911919 16359 rpc-test-base.h:478] error unregistering service: Service unavailable: service kudu.rpc.GenericCalculatorService not registered on TestServer
[       OK ] MultiThreadedRpcTest.TestShutdownDuringService (1489 ms)
[ RUN      ] MultiThreadedRpcTest.TestShutdownClientWhileCallsPending
I20260809 13:45:42.930500 16482 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 0.0.0.0:36439 every 8 connection(s)
I20260809 13:45:42.931639 16483 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 0.0.0.0:36439 every 8 connection(s)
I20260809 13:45:42.945031 16501 mt-rpc-test.cc:111] Connecting to 0.0.0.0:36439
I20260809 13:45:42.947597 16501 mt-rpc-test.cc:121] Call failed. Shutting down client thread. Ran 1 calls: Service unavailable: Unable to start connection negotiation thread: The pool has been shut down.
W20260809 13:45:42.951104 16477 negotiation.cc:336] Failed RPC negotiation. Trace:
0809 13:45:42.947575 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:53644 (local address 127.0.0.1:36439)
0809 13:45:42.948427 (+   852us) server_negotiation.cc:207] Beginning negotiation
0809 13:45:42.948447 (+    20us) server_negotiation.cc:400] Waiting for connection header
0809 13:45:42.948812 (+   365us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53644: BlockingRecv error: recv got EOF from 127.0.0.1:53644 (error 108)
Metrics: {"server-negotiator.queue_time_us":225}
[       OK ] MultiThreadedRpcTest.TestShutdownClientWhileCallsPending (40 ms)
[ RUN      ] MultiThreadedRpcTest.TestBlowOutServiceQueue
I20260809 13:45:42.962884 16506 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 0.0.0.0:34069 every 8 connection(s)
I20260809 13:45:42.963459 16507 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 0.0.0.0:34069 every 8 connection(s)
I20260809 13:45:42.965631 16508 mt-rpc-test.cc:91] Connecting to 0.0.0.0:34069
I20260809 13:45:42.966487 16512 mt-rpc-test.cc:91] Connecting to 0.0.0.0:34069
I20260809 13:45:42.967739 16513 mt-rpc-test.cc:91] Connecting to 0.0.0.0:34069
W20260809 13:45:43.003994 16503 service_pool.cc:136] Add request on kudu.rpc.GenericCalculatorService from 127.0.0.1:40622 dropped due to backpressure: service queue is full with 2 items
I20260809 13:45:43.004319 16503 service_pool.cc:137] Add request on kudu.rpc.GenericCalculatorService from 127.0.0.1:40622 dropped due to backpressure: service queue is full with 2 items Contents of service queue:
Call kudu.rpc.GenericCalculatorService.Add from 127.0.0.1:40606 (request call id 0)
Call kudu.rpc.GenericCalculatorService.Add from 127.0.0.1:40614 (request call id 0)
W20260809 13:45:43.023151 16359 rpc-test-base.h:478] error unregistering service: Service unavailable: service kudu.rpc.GenericCalculatorService not registered on messenger1
[       OK ] MultiThreadedRpcTest.TestBlowOutServiceQueue (68 ms)
[ RUN      ] MultiThreadedRpcTest.ReactorMaxIOLatencyStatsDisabled
I20260809 13:45:43.038760 16589 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 0.0.0.0:35971 every 8 connection(s)
I20260809 13:45:43.039654 16590 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 0.0.0.0:35971 every 8 connection(s)
[       OK ] MultiThreadedRpcTest.ReactorMaxIOLatencyStatsDisabled (1161 ms)
[ RUN      ] MultiThreadedRpcTest.ReactorMaxIOLatencyStatsEnabled
I20260809 13:45:44.198025 16624 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 0.0.0.0:39499 every 8 connection(s)
I20260809 13:45:44.199064 16626 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 0.0.0.0:39499 every 8 connection(s)
I20260809 13:45:50.082865 16628 rpcz_store.cc:275] Call kudu.rpc.GenericCalculatorService.SendTwoStrings from 127.0.0.1:36052 (request call id 0) took 5770 ms. Trace:
I20260809 13:45:50.090901 16628 rpcz_store.cc:276] 0809 13:45:44.312527 (+     0us) service_pool.cc:167] Inserting onto call queue
0809 13:45:44.312713 (+   186us) service_pool.cc:224] Handling call
0809 13:45:50.082742 (+5770029us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260809 13:45:50.573115 16636 rpcz_store.cc:275] Call kudu.rpc.GenericCalculatorService.SendTwoStrings from 127.0.0.1:36030 (request call id 0) took 6289 ms. Trace:
I20260809 13:45:50.573901 16636 rpcz_store.cc:276] 0809 13:45:44.283812 (+     0us) service_pool.cc:167] Inserting onto call queue
0809 13:45:44.283976 (+   164us) service_pool.cc:224] Handling call
0809 13:45:50.573032 (+6289056us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260809 13:45:50.630704 16633 rpcz_store.cc:275] Call kudu.rpc.GenericCalculatorService.SendTwoStrings from 127.0.0.1:36040 (request call id 0) took 6345 ms. Trace:
I20260809 13:45:50.631618 16633 rpcz_store.cc:276] 0809 13:45:44.284960 (+     0us) service_pool.cc:167] Inserting onto call queue
0809 13:45:44.285296 (+   336us) service_pool.cc:224] Handling call
0809 13:45:50.630587 (+6345291us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260809 13:45:50.930045 16632 rpcz_store.cc:275] Call kudu.rpc.GenericCalculatorService.SendTwoStrings from 127.0.0.1:36054 (request call id 0) took 6632 ms. Trace:
I20260809 13:45:50.931044 16632 rpcz_store.cc:276] 0809 13:45:44.297144 (+     0us) service_pool.cc:167] Inserting onto call queue
0809 13:45:44.297335 (+   191us) service_pool.cc:224] Handling call
0809 13:45:50.929910 (+6632575us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260809 13:45:51.607024 16627 rpcz_store.cc:275] Call kudu.rpc.GenericCalculatorService.SendTwoStrings from 127.0.0.1:36070 (request call id 0) took 7250 ms. Trace:
I20260809 13:45:51.608287 16627 rpcz_store.cc:276] 0809 13:45:44.355758 (+     0us) service_pool.cc:167] Inserting onto call queue
0809 13:45:44.355963 (+   205us) service_pool.cc:224] Handling call
0809 13:45:51.606431 (+7250468us) inbound_call.cc:177] Queueing success response
Metrics: {}
F20260809 13:45:54.345415 16647 rpc-test-base.h:557] Check failed: _s.ok() Bad status: Timed out: SendTwoStrings RPC to 0.0.0.0:39499 timed out after 10.000s (SENT)
*** Check failure stack trace: ***
*** Aborted at 1786283154 (unix time) try "date -d @1786283154" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGABRT (@0x3e800003fe7) received by PID 16359 (TID 0x7f47e94f9700) from PID 16359; stack trace: ***
    @           0x29e030 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
    @           0x2a0513 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
    @     0x7f47fdfce980 (unknown) at ??:0
    @     0x7f47fd9f1e87 gsignal at ??:0
    @     0x7f47fd9f37f1 abort at ??:0
    @           0x2a4026 __interceptor_abort at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
    @     0x7f480102cd74 google::LogMessage::SendToLog() at ??:0
    @     0x7f480102d920 google::LogMessage::Flush() at ??:0
    @     0x7f4801032a5b google::LogMessageFatal::~LogMessageFatal() at ??:0
    @           0x359f3d kudu::rpc::RpcTestBase::DoTestSidecar() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/rpc-test-base.h:?
    @           0x353685 kudu::rpc::MultiThreadedRpcTest_ReactorMaxIOLatencyStatsEnabled_Test::TestBody()::$_5::operator()() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/mt-rpc-test.cc:?
    @           0x35345a _ZNSt3__18__invokeIZN4kudu3rpc57MultiThreadedRpcTest_ReactorMaxIOLatencyStatsEnabled_Test8TestBodyEvE3$_5JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS5_DpOS6_ at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3905
    @           0x3533c2 _ZNSt3__116__thread_executeINS_10unique_ptrINS_15__thread_structENS_14default_deleteIS2_EEEEZN4kudu3rpc57MultiThreadedRpcTest_ReactorMaxIOLatencyStatsEnabled_Test8TestBodyEvE3$_5JEJEEEvRNS_5tupleIJT_T0_DpT1_EEENS_15__tuple_indicesIJXspT2_EEEE at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:281
    @           0x3530db std::__1::__thread_proxy<>() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:?
    @           0x29def0 __tsan_thread_start_func at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:959
    @     0x7f47fdfc36db start_thread at ??:0
    @     0x7f47fdad461f clone at ??:0