Diagnosed failure
MaintenanceManagerTest.TestRunningInstances: /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager-test.cc:629: Failure
Expected equality of these values:
status_pb.running_operations_size()
Which is: 0
2
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_util.cc:401: Failure
Failed
Timed out waiting for assertion to pass.
LLVMSymbolizer: error reading file: No such file or directory
==================
WARNING: ThreadSanitizer: data race on vptr (ctor/dtor vs virtual call) (pid=25470)
Write of size 8 at 0x7ffc34c71ff0 by main thread:
#0 kudu::MaintenanceOp::~MaintenanceOp() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager.cc:173:33 (libkudu_util.so+0x386aae)
#1 kudu::TestMaintenanceOp::~TestMaintenanceOp() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager-test.cc:119:41 (maintenance_manager-test+0x374d83)
#2 kudu::MaintenanceManagerTest_TestRunningInstances_Test::TestBody() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager-test.cc:643:1 (maintenance_manager-test+0x35e7c6)
#3 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x64e2f)
#4 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x64e2f)
#5 testing::Test::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2674:5 (libgtest.so.1.12.1+0x42a31)
#6 testing::TestInfo::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2853:11 (libgtest.so.1.12.1+0x43d48)
#7 testing::TestSuite::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012:30 (libgtest.so.1.12.1+0x44d24)
#8 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5870:44 (libgtest.so.1.12.1+0x59814)
#9 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x65cef)
#10 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x65cef)
#11 testing::UnitTest::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5444:10 (libgtest.so.1.12.1+0x58dcc)
#12 RUN_ALL_TESTS() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/gtest/gtest.h:2293:73 (maintenance_manager-test+0x393c2b)
#13 main /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_main.cc:109:10 (maintenance_manager-test+0x392b2c)
Previous read of size 8 at 0x7ffc34c71ff0 by thread T11 (mutexes: write M698):
#0 kudu::MaintenanceManager::FindBestOp() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager.cc:474:9 (libkudu_util.so+0x3895cc)
#1 kudu::MaintenanceManager::RunSchedulerThread() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager.cc:382:32 (libkudu_util.so+0x388649)
#2 kudu::MaintenanceManager::Start()::$_2::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager.cc:238:42 (libkudu_util.so+0x38d721)
#3 decltype(std::__1::forward<kudu::MaintenanceManager::Start()::$_2&>(fp)()) std::__1::__invoke<kudu::MaintenanceManager::Start()::$_2&>(kudu::MaintenanceManager::Start()::$_2&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3899:1 (libkudu_util.so+0x38d6d9)
#4 void std::__1::__invoke_void_return_wrapper<void>::__call<kudu::MaintenanceManager::Start()::$_2&>(kudu::MaintenanceManager::Start()::$_2&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__functional_base:348:9 (libkudu_util.so+0x38d669)
#5 std::__1::__function::__alloc_func<kudu::MaintenanceManager::Start()::$_2, std::__1::allocator<kudu::MaintenanceManager::Start()::$_2>, void ()>::operator()() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1557:16 (libkudu_util.so+0x38d631)
#6 std::__1::__function::__func<kudu::MaintenanceManager::Start()::$_2, std::__1::allocator<kudu::MaintenanceManager::Start()::$_2>, void ()>::operator()() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1731:12 (libkudu_util.so+0x38c92d)
#7 std::__1::__function::__value_func<void ()>::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1884:16 (libkudu_test_util.so+0x36e44)
#8 std::__1::function<void ()>::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:2556:12 (libkudu_test_util.so+0x288c9)
#9 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:693:3 (libkudu_util.so+0x448fe6)
Location is stack of main thread.
Location is global '??' at 0x7ffc34c54000 ([stack]+0x00000001dff0)
Mutex M698 (0x7b5000000270) created at:
#0 pthread_mutex_init /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1224 (maintenance_manager-test+0x2dfea2)
#1 kudu::Mutex::Mutex() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/mutex.cc:65:8 (libkudu_util.so+0x3dd87f)
#2 kudu::MaintenanceManager::MaintenanceManager(kudu::MaintenanceManager::Options const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, scoped_refptr<kudu::MetricEntity> const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager.cc:201:21 (libkudu_util.so+0x387469)
#3 kudu::MaintenanceManagerTest::StartManager(int) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager-test.cc:343:24 (maintenance_manager-test+0x3741d4)
#4 kudu::MaintenanceManagerTest::SetUp() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager-test.cc:331:5 (maintenance_manager-test+0x3757ee)
#5 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x64e2f)
#6 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x64e2f)
#7 testing::Test::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2669:3 (libgtest.so.1.12.1+0x4280b)
#8 testing::TestInfo::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2853:11 (libgtest.so.1.12.1+0x43d48)
#9 testing::TestSuite::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012:30 (libgtest.so.1.12.1+0x44d24)
#10 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5870:44 (libgtest.so.1.12.1+0x59814)
#11 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x65cef)
#12 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x65cef)
#13 testing::UnitTest::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5444:10 (libgtest.so.1.12.1+0x58dcc)
#14 RUN_ALL_TESTS() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/gtest/gtest.h:2293:73 (maintenance_manager-test+0x393c2b)
#15 main /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_main.cc:109:10 (maintenance_manager-test+0x392b2c)
Thread T11 'maintenance_sch' (tid=25503, running) created by main thread at:
#0 pthread_create /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (maintenance_manager-test+0x2df935)
#1 kudu::Thread::StartThread(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<void ()>, unsigned long, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:637:15 (libkudu_util.so+0x44884a)
#2 kudu::Thread::Create(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<void ()>, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.h:146:12 (maintenance_manager-test+0x37e409)
#3 kudu::MaintenanceManager::Start() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager.cc:237:10 (libkudu_util.so+0x387bed)
#4 kudu::MaintenanceManagerTest::StartManager(int) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager-test.cc:359:5 (maintenance_manager-test+0x374238)
#5 kudu::MaintenanceManagerTest::SetUp() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager-test.cc:331:5 (maintenance_manager-test+0x3757ee)
#6 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x64e2f)
#7 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x64e2f)
#8 testing::Test::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2669:3 (libgtest.so.1.12.1+0x4280b)
#9 testing::TestInfo::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2853:11 (libgtest.so.1.12.1+0x43d48)
#10 testing::TestSuite::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012:30 (libgtest.so.1.12.1+0x44d24)
#11 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5870:44 (libgtest.so.1.12.1+0x59814)
#12 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x65cef)
#13 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x65cef)
#14 testing::UnitTest::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5444:10 (libgtest.so.1.12.1+0x58dcc)
#15 RUN_ALL_TESTS() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/gtest/gtest.h:2293:73 (maintenance_manager-test+0x393c2b)
#16 main /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_main.cc:109:10 (maintenance_manager-test+0x392b2c)
SUMMARY: ThreadSanitizer: data race on vptr (ctor/dtor vs virtual call) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager.cc:173:33 in kudu::MaintenanceOp::~MaintenanceOp()
==================
F20250114 09:00:41.489043 25470 maintenance_manager.cc:174] Check failed: !manager_.get() You must unregister the op Op before destroying it.
libc++abi: Pure virtual function called!
*** Aborted at 1736845241 (unix time) try "date -d @1736845241" if you are using GNU date ***
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e80000637e) received by PID 25470 (TID 0x7fdca95fe700) from PID 25470; stack trace: ***
@ 0x2dbfc0 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2de4a3 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
@ 0x7fdcafa2c980 (unknown) at ??:0
@ 0x7fdcaf44ffb7 gsignal at ??:0
@ 0x7fdcaf451921 abort at ??:0
@ 0x2e1fb6 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
@ 0x7fdcb00039a3 abort_message at ??:0
@ 0x7fdcb0003372 __cxa_pure_virtual at ??:0
@ 0x7fdcb2e455d9 kudu::MaintenanceManager::FindBestOp() at ??:0
@ 0x7fdcb2e4464a kudu::MaintenanceManager::RunSchedulerThread() at ??:0
@ 0x7fdcb2e49722 kudu::MaintenanceManager::Start()::$_2::operator()() at ??:0
@ 0x7fdcb2e496da _ZNSt3__18__invokeIRZN4kudu18MaintenanceManager5StartEvE3$_2JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS5_DpOS6_ at ??:0
@ 0x7fdcb2e4966a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7fdcb2e49632 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7fdcb2e4892e std::__1::__function::__func<>::operator()() at ??:0
@ 0x7fdcb31d0e45 std::__1::__function::__value_func<>::operator()() at ??:0
@ 0x7fdcb31c28ca std::__1::function<>::operator()() at ??:0
@ 0x7fdcb2f04fe7 kudu::Thread::SuperviseThread() at ??:0
@ 0x2dbe80 __tsan_thread_start_func at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:959
@ 0x7fdcafa216db start_thread at ??:0
@ 0x7fdcaf53271f clone at ??:0
Full log
[==========] Running 16 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 16 tests from MaintenanceManagerTest
[ RUN ] MaintenanceManagerTest.TestCreateAndShutdown
[ OK ] MaintenanceManagerTest.TestCreateAndShutdown (9 ms)
[ RUN ] MaintenanceManagerTest.TestRegisterUnregister
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250114 09:00:11.261719 25481 maintenance_manager-test.cc:129] Prepared op 1
I20250114 09:00:11.262045 25481 maintenance_manager.cc:418] P 12345: Scheduling 1: perf score=10.000000
I20250114 09:00:11.262956 25480 maintenance_manager-test.cc:136] Performing op 1
I20250114 09:00:11.263249 25480 maintenance_manager.cc:642] P 12345: 1 complete. Timing: real 0.000s user 0.001s sys 0.000s Metrics: {}
[ OK ] MaintenanceManagerTest.TestRegisterUnregister (11 ms)
[ RUN ] MaintenanceManagerTest.TestRegisterUnregisterWithContention
[ OK ] MaintenanceManagerTest.TestRegisterUnregisterWithContention (7 ms)
[ RUN ] MaintenanceManagerTest.TestNewOpsDontGetScheduledDuringUnregister
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager-test.cc:449: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] MaintenanceManagerTest.TestNewOpsDontGetScheduledDuringUnregister (5 ms)
[ RUN ] MaintenanceManagerTest.TestMemoryPressurePrioritizesMemory
I20250114 09:00:11.304504 25491 maintenance_manager-test.cc:129] Prepared op op
I20250114 09:00:11.304683 25491 maintenance_manager.cc:418] P 12345: Scheduling op: under memory pressure (0.00% used), 0 bytes log retention, and flush 100 bytes memory
I20250114 09:00:11.305289 25490 maintenance_manager-test.cc:136] Performing op op
I20250114 09:00:11.305646 25490 maintenance_manager.cc:642] P 12345: op complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
[ OK ] MaintenanceManagerTest.TestMemoryPressurePrioritizesMemory (30 ms)
[ RUN ] MaintenanceManagerTest.TestMemoryPressurePerformsNoMemoryOp
I20250114 09:00:11.335124 25494 maintenance_manager-test.cc:129] Prepared op op
I20250114 09:00:11.335302 25494 maintenance_manager.cc:418] P 12345: Scheduling op: perf score=1.000000
I20250114 09:00:11.335875 25493 maintenance_manager-test.cc:136] Performing op op
I20250114 09:00:11.336131 25493 maintenance_manager.cc:642] P 12345: op complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
[ OK ] MaintenanceManagerTest.TestMemoryPressurePerformsNoMemoryOp (30 ms)
[ RUN ] MaintenanceManagerTest.TestLogRetentionPrioritization
[ OK ] MaintenanceManagerTest.TestLogRetentionPrioritization (5 ms)
[ RUN ] MaintenanceManagerTest.TestPrioritizeLogRetentionUnderMemoryPressure
[ OK ] MaintenanceManagerTest.TestPrioritizeLogRetentionUnderMemoryPressure (4 ms)
[ RUN ] MaintenanceManagerTest.TestRunningInstances
I20250114 09:00:11.354168 25503 maintenance_manager-test.cc:129] Prepared op op
I20250114 09:00:11.354329 25503 maintenance_manager.cc:418] P 12345: Scheduling op: perf score=10.000000
I20250114 09:00:11.354789 25503 maintenance_manager-test.cc:129] Prepared op op
I20250114 09:00:11.354979 25502 maintenance_manager-test.cc:136] Performing op op
I20250114 09:00:11.354974 25503 maintenance_manager.cc:418] P 12345: Scheduling op: perf score=10.000000
I20250114 09:00:12.437857 25502 maintenance_manager.cc:642] P 12345: op complete. Timing: real 1.083s user 0.001s sys 0.000s Metrics: {"spinlock_wait_cycles":65920}
I20250114 09:00:12.438779 25501 maintenance_manager-test.cc:136] Performing op op
I20250114 09:00:13.439128 25501 maintenance_manager.cc:642] P 12345: op complete. Timing: real 1.000s user 0.001s sys 0.000s Metrics: {}
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager-test.cc:629: Failure
Expected equality of these values:
status_pb.running_operations_size()
Which is: 0
2
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_util.cc:401: Failure
Failed
Timed out waiting for assertion to pass.
LLVMSymbolizer: error reading file: No such file or directory
==================
WARNING: ThreadSanitizer: data race on vptr (ctor/dtor vs virtual call) (pid=25470)
Write of size 8 at 0x7ffc34c71ff0 by main thread:
#0 kudu::MaintenanceOp::~MaintenanceOp() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager.cc:173:33 (libkudu_util.so+0x386aae)
#1 kudu::TestMaintenanceOp::~TestMaintenanceOp() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager-test.cc:119:41 (maintenance_manager-test+0x374d83)
#2 kudu::MaintenanceManagerTest_TestRunningInstances_Test::TestBody() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager-test.cc:643:1 (maintenance_manager-test+0x35e7c6)
#3 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x64e2f)
#4 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x64e2f)
#5 testing::Test::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2674:5 (libgtest.so.1.12.1+0x42a31)
#6 testing::TestInfo::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2853:11 (libgtest.so.1.12.1+0x43d48)
#7 testing::TestSuite::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012:30 (libgtest.so.1.12.1+0x44d24)
#8 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5870:44 (libgtest.so.1.12.1+0x59814)
#9 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x65cef)
#10 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x65cef)
#11 testing::UnitTest::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5444:10 (libgtest.so.1.12.1+0x58dcc)
#12 RUN_ALL_TESTS() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/gtest/gtest.h:2293:73 (maintenance_manager-test+0x393c2b)
#13 main /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_main.cc:109:10 (maintenance_manager-test+0x392b2c)
Previous read of size 8 at 0x7ffc34c71ff0 by thread T11 (mutexes: write M698):
#0 kudu::MaintenanceManager::FindBestOp() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager.cc:474:9 (libkudu_util.so+0x3895cc)
#1 kudu::MaintenanceManager::RunSchedulerThread() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager.cc:382:32 (libkudu_util.so+0x388649)
#2 kudu::MaintenanceManager::Start()::$_2::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager.cc:238:42 (libkudu_util.so+0x38d721)
#3 decltype(std::__1::forward<kudu::MaintenanceManager::Start()::$_2&>(fp)()) std::__1::__invoke<kudu::MaintenanceManager::Start()::$_2&>(kudu::MaintenanceManager::Start()::$_2&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3899:1 (libkudu_util.so+0x38d6d9)
#4 void std::__1::__invoke_void_return_wrapper<void>::__call<kudu::MaintenanceManager::Start()::$_2&>(kudu::MaintenanceManager::Start()::$_2&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__functional_base:348:9 (libkudu_util.so+0x38d669)
#5 std::__1::__function::__alloc_func<kudu::MaintenanceManager::Start()::$_2, std::__1::allocator<kudu::MaintenanceManager::Start()::$_2>, void ()>::operator()() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1557:16 (libkudu_util.so+0x38d631)
#6 std::__1::__function::__func<kudu::MaintenanceManager::Start()::$_2, std::__1::allocator<kudu::MaintenanceManager::Start()::$_2>, void ()>::operator()() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1731:12 (libkudu_util.so+0x38c92d)
#7 std::__1::__function::__value_func<void ()>::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1884:16 (libkudu_test_util.so+0x36e44)
#8 std::__1::function<void ()>::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:2556:12 (libkudu_test_util.so+0x288c9)
#9 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:693:3 (libkudu_util.so+0x448fe6)
Location is stack of main thread.
Location is global '??' at 0x7ffc34c54000 ([stack]+0x00000001dff0)
Mutex M698 (0x7b5000000270) created at:
#0 pthread_mutex_init /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1224 (maintenance_manager-test+0x2dfea2)
#1 kudu::Mutex::Mutex() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/mutex.cc:65:8 (libkudu_util.so+0x3dd87f)
#2 kudu::MaintenanceManager::MaintenanceManager(kudu::MaintenanceManager::Options const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, scoped_refptr<kudu::MetricEntity> const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager.cc:201:21 (libkudu_util.so+0x387469)
#3 kudu::MaintenanceManagerTest::StartManager(int) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager-test.cc:343:24 (maintenance_manager-test+0x3741d4)
#4 kudu::MaintenanceManagerTest::SetUp() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager-test.cc:331:5 (maintenance_manager-test+0x3757ee)
#5 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x64e2f)
#6 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x64e2f)
#7 testing::Test::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2669:3 (libgtest.so.1.12.1+0x4280b)
#8 testing::TestInfo::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2853:11 (libgtest.so.1.12.1+0x43d48)
#9 testing::TestSuite::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012:30 (libgtest.so.1.12.1+0x44d24)
#10 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5870:44 (libgtest.so.1.12.1+0x59814)
#11 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x65cef)
#12 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x65cef)
#13 testing::UnitTest::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5444:10 (libgtest.so.1.12.1+0x58dcc)
#14 RUN_ALL_TESTS() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/gtest/gtest.h:2293:73 (maintenance_manager-test+0x393c2b)
#15 main /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_main.cc:109:10 (maintenance_manager-test+0x392b2c)
Thread T11 'maintenance_sch' (tid=25503, running) created by main thread at:
#0 pthread_create /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (maintenance_manager-test+0x2df935)
#1 kudu::Thread::StartThread(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<void ()>, unsigned long, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:637:15 (libkudu_util.so+0x44884a)
#2 kudu::Thread::Create(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<void ()>, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.h:146:12 (maintenance_manager-test+0x37e409)
#3 kudu::MaintenanceManager::Start() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager.cc:237:10 (libkudu_util.so+0x387bed)
#4 kudu::MaintenanceManagerTest::StartManager(int) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager-test.cc:359:5 (maintenance_manager-test+0x374238)
#5 kudu::MaintenanceManagerTest::SetUp() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager-test.cc:331:5 (maintenance_manager-test+0x3757ee)
#6 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x64e2f)
#7 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x64e2f)
#8 testing::Test::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2669:3 (libgtest.so.1.12.1+0x4280b)
#9 testing::TestInfo::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2853:11 (libgtest.so.1.12.1+0x43d48)
#10 testing::TestSuite::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012:30 (libgtest.so.1.12.1+0x44d24)
#11 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5870:44 (libgtest.so.1.12.1+0x59814)
#12 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x65cef)
#13 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x65cef)
#14 testing::UnitTest::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5444:10 (libgtest.so.1.12.1+0x58dcc)
#15 RUN_ALL_TESTS() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/gtest/gtest.h:2293:73 (maintenance_manager-test+0x393c2b)
#16 main /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_main.cc:109:10 (maintenance_manager-test+0x392b2c)
SUMMARY: ThreadSanitizer: data race on vptr (ctor/dtor vs virtual call) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager.cc:173:33 in kudu::MaintenanceOp::~MaintenanceOp()
==================
F20250114 09:00:41.489043 25470 maintenance_manager.cc:174] Check failed: !manager_.get() You must unregister the op Op before destroying it.
libc++abi: Pure virtual function called!
*** Aborted at 1736845241 (unix time) try "date -d @1736845241" if you are using GNU date ***
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e80000637e) received by PID 25470 (TID 0x7fdca95fe700) from PID 25470; stack trace: ***
@ 0x2dbfc0 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2de4a3 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
@ 0x7fdcafa2c980 (unknown) at ??:0
@ 0x7fdcaf44ffb7 gsignal at ??:0
@ 0x7fdcaf451921 abort at ??:0
@ 0x2e1fb6 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
@ 0x7fdcb00039a3 abort_message at ??:0
@ 0x7fdcb0003372 __cxa_pure_virtual at ??:0
@ 0x7fdcb2e455d9 kudu::MaintenanceManager::FindBestOp() at ??:0
@ 0x7fdcb2e4464a kudu::MaintenanceManager::RunSchedulerThread() at ??:0
@ 0x7fdcb2e49722 kudu::MaintenanceManager::Start()::$_2::operator()() at ??:0
@ 0x7fdcb2e496da _ZNSt3__18__invokeIRZN4kudu18MaintenanceManager5StartEvE3$_2JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS5_DpOS6_ at ??:0
@ 0x7fdcb2e4966a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7fdcb2e49632 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7fdcb2e4892e std::__1::__function::__func<>::operator()() at ??:0
@ 0x7fdcb31d0e45 std::__1::__function::__value_func<>::operator()() at ??:0
@ 0x7fdcb31c28ca std::__1::function<>::operator()() at ??:0
@ 0x7fdcb2f04fe7 kudu::Thread::SuperviseThread() at ??:0
@ 0x2dbe80 __tsan_thread_start_func at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:959
@ 0x7fdcafa216db start_thread at ??:0
@ 0x7fdcaf53271f clone at ??:0