Diagnosed failure
MaintenanceManagerTest.TestNewOpsDontGetScheduledDuringUnregister: /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/maintenance_manager-test.cc:468: Failure
Expected: (op1.DurationHistogram()->TotalCount()) <= (2), actual: 3 vs 2
I20260928 03:44:41.275900 24280 test_util.cc:175] -----------------------------------------------
I20260928 03:44:41.276062 24280 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskyXmWXi/test-tmp/maintenance_manager-test.0.MaintenanceManagerTest.TestNewOpsDontGetScheduledDuringUnregister.1790567079098751-24280-0
Full log
[==========] Running 16 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 16 tests from MaintenanceManagerTest
[ RUN ] MaintenanceManagerTest.TestCreateAndShutdown
[ OK ] MaintenanceManagerTest.TestCreateAndShutdown (14 ms)
[ RUN ] MaintenanceManagerTest.TestRegisterUnregister
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260928 03:44:39.146443 24291 maintenance_manager-test.cc:130] Prepared op 1
I20260928 03:44:39.146824 24291 maintenance_manager.cc:419] P 12345: Scheduling 1: perf score=10.000000
I20260928 03:44:39.147936 24290 maintenance_manager-test.cc:137] Performing op 1
I20260928 03:44:39.148227 24290 maintenance_manager.cc:643] P 12345: 1 complete. Timing: real 0.000s user 0.001s sys 0.000s Metrics: {}
[ OK ] MaintenanceManagerTest.TestRegisterUnregister (16 ms)
[ RUN ] MaintenanceManagerTest.TestRegisterUnregisterWithContention
[ OK ] MaintenanceManagerTest.TestRegisterUnregisterWithContention (7 ms)
[ RUN ] MaintenanceManagerTest.TestNewOpsDontGetScheduledDuringUnregister
I20260928 03:44:39.163584 24298 maintenance_manager-test.cc:130] Prepared op 1
I20260928 03:44:39.163750 24298 maintenance_manager.cc:419] P 12345: Scheduling 1: perf score=10.000000
I20260928 03:44:39.164342 24298 maintenance_manager-test.cc:130] Prepared op 1
I20260928 03:44:39.164551 24297 maintenance_manager-test.cc:137] Performing op 1
I20260928 03:44:39.164552 24298 maintenance_manager.cc:419] P 12345: Scheduling 1: perf score=10.000000
I20260928 03:44:40.270503 24297 maintenance_manager.cc:643] P 12345: 1 complete. Timing: real 1.106s user 0.001s sys 0.000s Metrics: {"spinlock_wait_cycles":316288}
I20260928 03:44:40.271436 24296 maintenance_manager-test.cc:137] Performing op 1
I20260928 03:44:40.271783 24298 maintenance_manager-test.cc:130] Prepared op 1
I20260928 03:44:40.271935 24298 maintenance_manager.cc:419] P 12345: Scheduling 1: perf score=10.000000
I20260928 03:44:40.272435 24297 maintenance_manager-test.cc:137] Performing op 1
I20260928 03:44:41.271833 24296 maintenance_manager.cc:643] P 12345: 1 complete. Timing: real 1.000s user 0.001s sys 0.000s Metrics: {}
I20260928 03:44:41.272758 24297 maintenance_manager.cc:643] P 12345: 1 complete. Timing: real 1.000s user 0.000s sys 0.000s Metrics: {}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/maintenance_manager-test.cc:468: Failure
Expected: (op1.DurationHistogram()->TotalCount()) <= (2), actual: 3 vs 2
I20260928 03:44:41.275900 24280 test_util.cc:175] -----------------------------------------------
I20260928 03:44:41.276062 24280 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskyXmWXi/test-tmp/maintenance_manager-test.0.MaintenanceManagerTest.TestNewOpsDontGetScheduledDuringUnregister.1790567079098751-24280-0
[ FAILED ] MaintenanceManagerTest.TestNewOpsDontGetScheduledDuringUnregister (2117 ms)
[ RUN ] MaintenanceManagerTest.TestMemoryPressurePrioritizesMemory
I20260928 03:44:41.302551 24302 maintenance_manager-test.cc:130] Prepared op op
I20260928 03:44:41.302757 24302 maintenance_manager.cc:419] P 12345: Scheduling op: under memory pressure (0.00% used), 0 bytes log retention, and flush 100 bytes memory
I20260928 03:44:41.303311 24301 maintenance_manager-test.cc:137] Performing op op
I20260928 03:44:41.303589 24301 maintenance_manager.cc:643] P 12345: op complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
[ OK ] MaintenanceManagerTest.TestMemoryPressurePrioritizesMemory (31 ms)
[ RUN ] MaintenanceManagerTest.TestMemoryPressurePerformsNoMemoryOp
I20260928 03:44:41.333165 24305 maintenance_manager-test.cc:130] Prepared op op
I20260928 03:44:41.333359 24305 maintenance_manager.cc:419] P 12345: Scheduling op: perf score=1.000000
I20260928 03:44:41.334052 24304 maintenance_manager-test.cc:137] Performing op op
I20260928 03:44:41.334396 24304 maintenance_manager.cc:643] P 12345: op complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
[ OK ] MaintenanceManagerTest.TestMemoryPressurePerformsNoMemoryOp (31 ms)
[ RUN ] MaintenanceManagerTest.TestLogRetentionPrioritization
[ OK ] MaintenanceManagerTest.TestLogRetentionPrioritization (6 ms)
[ RUN ] MaintenanceManagerTest.TestPrioritizeLogRetentionUnderMemoryPressure
[ OK ] MaintenanceManagerTest.TestPrioritizeLogRetentionUnderMemoryPressure (5 ms)
[ RUN ] MaintenanceManagerTest.TestRunningInstances
I20260928 03:44:41.355897 24314 maintenance_manager-test.cc:130] Prepared op op
I20260928 03:44:41.356112 24314 maintenance_manager.cc:419] P 12345: Scheduling op: perf score=10.000000
I20260928 03:44:41.356601 24314 maintenance_manager-test.cc:130] Prepared op op
I20260928 03:44:41.356775 24314 maintenance_manager.cc:419] P 12345: Scheduling op: perf score=10.000000
I20260928 03:44:41.356786 24313 maintenance_manager-test.cc:137] Performing op op
I20260928 03:44:41.357352 24312 maintenance_manager-test.cc:137] Performing op op
I20260928 03:44:42.357345 24313 maintenance_manager.cc:643] P 12345: op complete. Timing: real 1.001s user 0.000s sys 0.001s Metrics: {"spinlock_wait_cycles":65408}
I20260928 03:44:42.357659 24312 maintenance_manager.cc:643] P 12345: op complete. Timing: real 1.000s user 0.000s sys 0.000s Metrics: {}
[ OK ] MaintenanceManagerTest.TestRunningInstances (1008 ms)
[ RUN ] MaintenanceManagerTest.TestCompletedOpsHistory
I20260928 03:44:42.364454 24317 maintenance_manager-test.cc:130] Prepared op op0
I20260928 03:44:42.364611 24317 maintenance_manager.cc:419] P 12345: Scheduling op0: perf score=1.000000
I20260928 03:44:42.365152 24316 maintenance_manager-test.cc:137] Performing op op0
I20260928 03:44:42.365438 24316 maintenance_manager.cc:643] P 12345: op0 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260928 03:44:42.368170 24317 maintenance_manager-test.cc:130] Prepared op op1
I20260928 03:44:42.368310 24317 maintenance_manager.cc:419] P 12345: Scheduling op1: perf score=1.000000
I20260928 03:44:42.368772 24316 maintenance_manager-test.cc:137] Performing op op1
I20260928 03:44:42.369023 24316 maintenance_manager.cc:643] P 12345: op1 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260928 03:44:42.371796 24317 maintenance_manager-test.cc:130] Prepared op op2
I20260928 03:44:42.371946 24317 maintenance_manager.cc:419] P 12345: Scheduling op2: perf score=1.000000
I20260928 03:44:42.372401 24316 maintenance_manager-test.cc:137] Performing op op2
I20260928 03:44:42.372653 24316 maintenance_manager.cc:643] P 12345: op2 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260928 03:44:42.374213 24317 maintenance_manager-test.cc:130] Prepared op op3
I20260928 03:44:42.374385 24317 maintenance_manager.cc:419] P 12345: Scheduling op3: perf score=1.000000
I20260928 03:44:42.374882 24316 maintenance_manager-test.cc:137] Performing op op3
I20260928 03:44:42.375195 24316 maintenance_manager.cc:643] P 12345: op3 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260928 03:44:42.377864 24317 maintenance_manager-test.cc:130] Prepared op op4
I20260928 03:44:42.378062 24317 maintenance_manager.cc:419] P 12345: Scheduling op4: perf score=1.000000
I20260928 03:44:42.378515 24316 maintenance_manager-test.cc:137] Performing op op4
I20260928 03:44:42.378773 24316 maintenance_manager.cc:643] P 12345: op4 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260928 03:44:42.382689 24317 maintenance_manager-test.cc:130] Prepared op op5
I20260928 03:44:42.382861 24317 maintenance_manager.cc:419] P 12345: Scheduling op5: perf score=1.000000
I20260928 03:44:42.383373 24316 maintenance_manager-test.cc:137] Performing op op5
I20260928 03:44:42.383613 24316 maintenance_manager.cc:643] P 12345: op5 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260928 03:44:42.386322 24317 maintenance_manager-test.cc:130] Prepared op op6
I20260928 03:44:42.386472 24317 maintenance_manager.cc:419] P 12345: Scheduling op6: perf score=1.000000
I20260928 03:44:42.386909 24316 maintenance_manager-test.cc:137] Performing op op6
I20260928 03:44:42.387141 24316 maintenance_manager.cc:643] P 12345: op6 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260928 03:44:42.391007 24317 maintenance_manager-test.cc:130] Prepared op op7
I20260928 03:44:42.391152 24317 maintenance_manager.cc:419] P 12345: Scheduling op7: perf score=1.000000
I20260928 03:44:42.391667 24316 maintenance_manager-test.cc:137] Performing op op7
I20260928 03:44:42.391942 24316 maintenance_manager.cc:643] P 12345: op7 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260928 03:44:42.394656 24317 maintenance_manager-test.cc:130] Prepared op op8
I20260928 03:44:42.394804 24317 maintenance_manager.cc:419] P 12345: Scheduling op8: perf score=1.000000
I20260928 03:44:42.395231 24316 maintenance_manager-test.cc:137] Performing op op8
I20260928 03:44:42.395478 24316 maintenance_manager.cc:643] P 12345: op8 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260928 03:44:42.399464 24317 maintenance_manager-test.cc:130] Prepared op op9
I20260928 03:44:42.399636 24317 maintenance_manager.cc:419] P 12345: Scheduling op9: perf score=1.000000
I20260928 03:44:42.400101 24316 maintenance_manager-test.cc:137] Performing op op9
I20260928 03:44:42.400344 24316 maintenance_manager.cc:643] P 12345: op9 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260928 03:44:42.402985 24317 maintenance_manager-test.cc:130] Prepared op op10
I20260928 03:44:42.403151 24317 maintenance_manager.cc:419] P 12345: Scheduling op10: perf score=1.000000
I20260928 03:44:42.403604 24316 maintenance_manager-test.cc:137] Performing op op10
I20260928 03:44:42.403924 24316 maintenance_manager.cc:643] P 12345: op10 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
[ OK ] MaintenanceManagerTest.TestCompletedOpsHistory (47 ms)
[ RUN ] MaintenanceManagerTest.TestOpFactors
[ OK ] MaintenanceManagerTest.TestOpFactors (4 ms)
[ RUN ] MaintenanceManagerTest.TestPriorityOpLaunch
I20260928 03:44:42.418476 24325 maintenance_manager-test.cc:130] Prepared op canary
I20260928 03:44:42.418634 24325 maintenance_manager.cc:419] P 12345: Scheduling canary: perf score=1.000000
I20260928 03:44:42.419154 24324 maintenance_manager-test.cc:137] Performing op canary
I20260928 03:44:42.419394 24324 maintenance_manager.cc:643] P 12345: canary complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260928 03:44:42.424589 24325 maintenance_manager-test.cc:130] Prepared op op7
I20260928 03:44:42.424774 24325 maintenance_manager.cc:419] P 12345: Scheduling op7: perf score=19.000000
I20260928 03:44:42.425242 24324 maintenance_manager-test.cc:137] Performing op op7
I20260928 03:44:42.426501 24324 maintenance_manager.cc:643] P 12345: op7 complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20260928 03:44:42.426959 24325 maintenance_manager-test.cc:130] Prepared op op6
I20260928 03:44:42.427125 24325 maintenance_manager.cc:419] P 12345: Scheduling op6: perf score=16.105100
I20260928 03:44:42.427558 24324 maintenance_manager-test.cc:137] Performing op op6
I20260928 03:44:42.428798 24324 maintenance_manager.cc:643] P 12345: op6 complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20260928 03:44:42.429291 24325 maintenance_manager-test.cc:130] Prepared op op5
I20260928 03:44:42.429452 24325 maintenance_manager.cc:419] P 12345: Scheduling op5: perf score=12.000000
I20260928 03:44:42.429877 24324 maintenance_manager-test.cc:137] Performing op op5
I20260928 03:44:42.431185 24324 maintenance_manager.cc:643] P 12345: op5 complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20260928 03:44:42.431658 24325 maintenance_manager-test.cc:130] Prepared op op4
I20260928 03:44:42.431823 24325 maintenance_manager.cc:419] P 12345: Scheduling op4: perf score=11.000000
I20260928 03:44:42.432371 24324 maintenance_manager-test.cc:137] Performing op op4
I20260928 03:44:42.433713 24324 maintenance_manager.cc:643] P 12345: op4 complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20260928 03:44:42.434213 24325 maintenance_manager-test.cc:130] Prepared op op3
I20260928 03:44:42.434396 24325 maintenance_manager.cc:419] P 12345: Scheduling op3: perf score=10.000000
I20260928 03:44:42.434862 24324 maintenance_manager-test.cc:137] Performing op op3
I20260928 03:44:42.436105 24324 maintenance_manager.cc:643] P 12345: op3 complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20260928 03:44:42.436585 24325 maintenance_manager-test.cc:130] Prepared op op2
I20260928 03:44:42.436754 24325 maintenance_manager.cc:419] P 12345: Scheduling op2: perf score=9.090909
I20260928 03:44:42.437183 24324 maintenance_manager-test.cc:137] Performing op op2
I20260928 03:44:42.438423 24324 maintenance_manager.cc:643] P 12345: op2 complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20260928 03:44:42.438876 24325 maintenance_manager-test.cc:130] Prepared op op1
I20260928 03:44:42.439037 24325 maintenance_manager.cc:419] P 12345: Scheduling op1: perf score=6.209213
I20260928 03:44:42.439497 24324 maintenance_manager-test.cc:137] Performing op op1
I20260928 03:44:42.440738 24324 maintenance_manager.cc:643] P 12345: op1 complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
[ OK ] MaintenanceManagerTest.TestPriorityOpLaunch (30 ms)
[ RUN ] MaintenanceManagerTest.VerifyMetrics
I20260928 03:44:42.457324 24328 maintenance_manager.cc:411] P 12345: Prepare failed for oppf. Re-running scheduler.
I20260928 03:44:42.467706 24328 maintenance_manager-test.cc:130] Prepared op op0
I20260928 03:44:42.467897 24328 maintenance_manager.cc:419] P 12345: Scheduling op0: perf score=4.000000
I20260928 03:44:42.468485 24327 maintenance_manager-test.cc:137] Performing op op0
I20260928 03:44:42.468729 24327 maintenance_manager.cc:643] P 12345: op0 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260928 03:44:42.478436 24328 maintenance_manager-test.cc:130] Prepared op op1
I20260928 03:44:42.478631 24328 maintenance_manager.cc:419] P 12345: Scheduling op1: perf score=2.000000
I20260928 03:44:42.479164 24327 maintenance_manager-test.cc:137] Performing op op1
I20260928 03:44:42.479485 24327 maintenance_manager.cc:643] P 12345: op1 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
[ OK ] MaintenanceManagerTest.VerifyMetrics (47 ms)
[ RUN ] MaintenanceManagerTest.ManyOperationsHeavyUpdateStats
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/maintenance_manager-test.cc:890: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] MaintenanceManagerTest.ManyOperationsHeavyUpdateStats (4 ms)
[ RUN ] MaintenanceManagerTest.TestUnregisterWhileScheduling
I20260928 03:44:42.499341 24334 maintenance_manager-test.cc:130] Prepared op 1
I20260928 03:44:42.499504 24334 maintenance_manager.cc:419] P 12345: Scheduling 1: perf score=10.000000
I20260928 03:44:42.500064 24334 maintenance_manager-test.cc:130] Prepared op 1
I20260928 03:44:42.500239 24334 maintenance_manager.cc:419] P 12345: Scheduling 1: perf score=10.000000
I20260928 03:44:42.500267 24333 maintenance_manager-test.cc:137] Performing op 1
I20260928 03:44:42.500752 24332 maintenance_manager-test.cc:137] Performing op 1
I20260928 03:44:42.500700 24333 maintenance_manager.cc:643] P 12345: 1 complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"spinlock_wait_cycles":354560}
I20260928 03:44:42.501068 24332 maintenance_manager.cc:643] P 12345: 1 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260928 03:44:42.501394 24334 maintenance_manager-test.cc:130] Prepared op 1
I20260928 03:44:42.501560 24334 maintenance_manager.cc:419] P 12345: Scheduling 1: perf score=10.000000
I20260928 03:44:42.501976 24334 maintenance_manager-test.cc:130] Prepared op 1
I20260928 03:44:42.502187 24332 maintenance_manager-test.cc:137] Performing op 1
I20260928 03:44:42.502389 24334 maintenance_manager.cc:419] P 12345: Scheduling 1: perf score=10.000000
I20260928 03:44:42.502472 24332 maintenance_manager.cc:643] P 12345: 1 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"spinlock_wait_cycles":317696}
I20260928 03:44:42.502897 24333 maintenance_manager-test.cc:137] Performing op 1
I20260928 03:44:42.503183 24333 maintenance_manager.cc:643] P 12345: 1 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
[ OK ] MaintenanceManagerTest.TestUnregisterWhileScheduling (9 ms)
[ RUN ] MaintenanceManagerTest.ComprehensiveTest
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/maintenance_manager-test.cc:973: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] MaintenanceManagerTest.ComprehensiveTest (4 ms)
[----------] 16 tests from MaintenanceManagerTest (3388 ms total)
[----------] Global test environment tear-down
[==========] 16 tests from 1 test suite ran. (3389 ms total)
[ PASSED ] 13 tests.
[ SKIPPED ] 2 tests, listed below:
[ SKIPPED ] MaintenanceManagerTest.ManyOperationsHeavyUpdateStats
[ SKIPPED ] MaintenanceManagerTest.ComprehensiveTest
[ FAILED ] 1 test, listed below:
[ FAILED ] MaintenanceManagerTest.TestNewOpsDontGetScheduledDuringUnregister
1 FAILED TEST