Diagnosed failure
MaintenanceManagerTest.TestPriorityOpLaunch: /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager-test.cc:867: Failure
Expected equality of these values:
ordered_ops.front()
Which is: "op6"
instance.name()
Which is: "op7"
I20250625 03:33:20.882462 9863 test_util.cc:183] -----------------------------------------------
I20250625 03:33:20.882624 9863 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskQ70Ztu/test-tmp/maintenance_manager-test.0.MaintenanceManagerTest.TestPriorityOpLaunch.1750822398657305-9863-0
Full log
[==========] Running 16 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 16 tests from MaintenanceManagerTest
[ RUN ] MaintenanceManagerTest.TestCreateAndShutdown
[ OK ] MaintenanceManagerTest.TestCreateAndShutdown (8 ms)
[ RUN ] MaintenanceManagerTest.TestRegisterUnregister
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250625 03:33:18.684485 9874 maintenance_manager-test.cc:131] Prepared op 1
I20250625 03:33:18.684787 9874 maintenance_manager.cc:419] P 12345: Scheduling 1: perf score=10.000000
I20250625 03:33:18.685544 9873 maintenance_manager-test.cc:138] Performing op 1
I20250625 03:33:18.685801 9873 maintenance_manager.cc:643] P 12345: 1 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
[ OK ] MaintenanceManagerTest.TestRegisterUnregister (10 ms)
[ RUN ] MaintenanceManagerTest.TestRegisterUnregisterWithContention
[ OK ] MaintenanceManagerTest.TestRegisterUnregisterWithContention (6 ms)
[ RUN ] MaintenanceManagerTest.TestNewOpsDontGetScheduledDuringUnregister
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager-test.cc:453: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] MaintenanceManagerTest.TestNewOpsDontGetScheduledDuringUnregister (4 ms)
[ RUN ] MaintenanceManagerTest.TestMemoryPressurePrioritizesMemory
I20250625 03:33:18.724933 9884 maintenance_manager-test.cc:131] Prepared op op
I20250625 03:33:18.725072 9884 maintenance_manager.cc:419] P 12345: Scheduling op: under memory pressure (0.00% used), 0 bytes log retention, and flush 100 bytes memory
I20250625 03:33:18.725550 9883 maintenance_manager-test.cc:138] Performing op op
I20250625 03:33:18.725792 9883 maintenance_manager.cc:643] P 12345: op complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
[ OK ] MaintenanceManagerTest.TestMemoryPressurePrioritizesMemory (29 ms)
[ RUN ] MaintenanceManagerTest.TestMemoryPressurePerformsNoMemoryOp
I20250625 03:33:18.753624 9887 maintenance_manager-test.cc:131] Prepared op op
I20250625 03:33:18.753778 9887 maintenance_manager.cc:419] P 12345: Scheduling op: perf score=1.000000
I20250625 03:33:18.754276 9886 maintenance_manager-test.cc:138] Performing op op
I20250625 03:33:18.754546 9886 maintenance_manager.cc:643] P 12345: op complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
[ OK ] MaintenanceManagerTest.TestMemoryPressurePerformsNoMemoryOp (28 ms)
[ RUN ] MaintenanceManagerTest.TestLogRetentionPrioritization
[ OK ] MaintenanceManagerTest.TestLogRetentionPrioritization (4 ms)
[ RUN ] MaintenanceManagerTest.TestPrioritizeLogRetentionUnderMemoryPressure
[ OK ] MaintenanceManagerTest.TestPrioritizeLogRetentionUnderMemoryPressure (4 ms)
[ RUN ] MaintenanceManagerTest.RunningInstances
I20250625 03:33:18.770062 9896 maintenance_manager-test.cc:131] Prepared op op
I20250625 03:33:18.770200 9896 maintenance_manager.cc:419] P 12345: Scheduling op: perf score=10.000000
I20250625 03:33:18.770691 9896 maintenance_manager-test.cc:131] Prepared op op
I20250625 03:33:18.770865 9896 maintenance_manager.cc:419] P 12345: Scheduling op: perf score=10.000000
I20250625 03:33:18.770877 9895 maintenance_manager-test.cc:138] Performing op op
I20250625 03:33:19.795712 9894 maintenance_manager-test.cc:138] Performing op op
I20250625 03:33:20.796415 9895 maintenance_manager.cc:643] P 12345: op complete. Timing: real 2.026s user 0.005s sys 0.005s Metrics: {"spinlock_wait_cycles":300416}
I20250625 03:33:20.798065 9894 maintenance_manager.cc:643] P 12345: op complete. Timing: real 2.027s user 0.006s sys 0.000s Metrics: {"spinlock_wait_cycles":2355444096}
[ OK ] MaintenanceManagerTest.RunningInstances (2039 ms)
[ RUN ] MaintenanceManagerTest.TestCompletedOpsHistory
I20250625 03:33:20.810304 9900 maintenance_manager-test.cc:131] Prepared op op0
I20250625 03:33:20.810456 9900 maintenance_manager.cc:419] P 12345: Scheduling op0: perf score=1.000000
I20250625 03:33:20.811062 9899 maintenance_manager-test.cc:138] Performing op op0
I20250625 03:33:20.811286 9899 maintenance_manager.cc:643] P 12345: op0 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250625 03:33:20.813939 9900 maintenance_manager-test.cc:131] Prepared op op1
I20250625 03:33:20.814071 9900 maintenance_manager.cc:419] P 12345: Scheduling op1: perf score=1.000000
I20250625 03:33:20.814432 9899 maintenance_manager-test.cc:138] Performing op op1
I20250625 03:33:20.814640 9899 maintenance_manager.cc:643] P 12345: op1 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250625 03:33:20.818337 9900 maintenance_manager-test.cc:131] Prepared op op2
I20250625 03:33:20.818461 9900 maintenance_manager.cc:419] P 12345: Scheduling op2: perf score=1.000000
I20250625 03:33:20.818856 9899 maintenance_manager-test.cc:138] Performing op op2
I20250625 03:33:20.819147 9899 maintenance_manager.cc:643] P 12345: op2 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250625 03:33:20.821741 9900 maintenance_manager-test.cc:131] Prepared op op3
I20250625 03:33:20.821888 9900 maintenance_manager.cc:419] P 12345: Scheduling op3: perf score=1.000000
I20250625 03:33:20.822263 9899 maintenance_manager-test.cc:138] Performing op op3
I20250625 03:33:20.822530 9899 maintenance_manager.cc:643] P 12345: op3 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250625 03:33:20.824015 9900 maintenance_manager-test.cc:131] Prepared op op4
I20250625 03:33:20.824163 9900 maintenance_manager.cc:419] P 12345: Scheduling op4: perf score=1.000000
I20250625 03:33:20.824569 9899 maintenance_manager-test.cc:138] Performing op op4
I20250625 03:33:20.824887 9899 maintenance_manager.cc:643] P 12345: op4 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250625 03:33:20.827450 9900 maintenance_manager-test.cc:131] Prepared op op5
I20250625 03:33:20.827589 9900 maintenance_manager.cc:419] P 12345: Scheduling op5: perf score=1.000000
I20250625 03:33:20.827952 9899 maintenance_manager-test.cc:138] Performing op op5
I20250625 03:33:20.828171 9899 maintenance_manager.cc:643] P 12345: op5 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250625 03:33:20.829619 9900 maintenance_manager-test.cc:131] Prepared op op6
I20250625 03:33:20.829769 9900 maintenance_manager.cc:419] P 12345: Scheduling op6: perf score=1.000000
I20250625 03:33:20.830179 9899 maintenance_manager-test.cc:138] Performing op op6
I20250625 03:33:20.830487 9899 maintenance_manager.cc:643] P 12345: op6 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250625 03:33:20.833168 9900 maintenance_manager-test.cc:131] Prepared op op7
I20250625 03:33:20.833308 9900 maintenance_manager.cc:419] P 12345: Scheduling op7: perf score=1.000000
I20250625 03:33:20.833662 9899 maintenance_manager-test.cc:138] Performing op op7
I20250625 03:33:20.833909 9899 maintenance_manager.cc:643] P 12345: op7 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250625 03:33:20.835357 9900 maintenance_manager-test.cc:131] Prepared op op8
I20250625 03:33:20.835518 9900 maintenance_manager.cc:419] P 12345: Scheduling op8: perf score=1.000000
I20250625 03:33:20.835909 9899 maintenance_manager-test.cc:138] Performing op op8
I20250625 03:33:20.836175 9899 maintenance_manager.cc:643] P 12345: op8 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250625 03:33:20.839931 9900 maintenance_manager-test.cc:131] Prepared op op9
I20250625 03:33:20.840078 9900 maintenance_manager.cc:419] P 12345: Scheduling op9: perf score=1.000000
I20250625 03:33:20.840493 9899 maintenance_manager-test.cc:138] Performing op op9
I20250625 03:33:20.840705 9899 maintenance_manager.cc:643] P 12345: op9 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250625 03:33:20.843281 9900 maintenance_manager-test.cc:131] Prepared op op10
I20250625 03:33:20.843405 9900 maintenance_manager.cc:419] P 12345: Scheduling op10: perf score=1.000000
I20250625 03:33:20.843837 9899 maintenance_manager-test.cc:138] Performing op op10
I20250625 03:33:20.844060 9899 maintenance_manager.cc:643] P 12345: op10 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
[ OK ] MaintenanceManagerTest.TestCompletedOpsHistory (42 ms)
[ RUN ] MaintenanceManagerTest.TestOpFactors
[ OK ] MaintenanceManagerTest.TestOpFactors (4 ms)
[ RUN ] MaintenanceManagerTest.TestPriorityOpLaunch
I20250625 03:33:20.858287 9908 maintenance_manager-test.cc:131] Prepared op canary
I20250625 03:33:20.858435 9908 maintenance_manager.cc:419] P 12345: Scheduling canary: perf score=1.000000
I20250625 03:33:20.858870 9907 maintenance_manager-test.cc:138] Performing op canary
I20250625 03:33:20.859102 9907 maintenance_manager.cc:643] P 12345: canary complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250625 03:33:20.863086 9908 maintenance_manager-test.cc:131] Prepared op op6
I20250625 03:33:20.863302 9908 maintenance_manager.cc:419] P 12345: Scheduling op6: perf score=16.105100
I20250625 03:33:20.863853 9907 maintenance_manager-test.cc:138] Performing op op6
I20250625 03:33:20.865099 9907 maintenance_manager.cc:643] P 12345: op6 complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250625 03:33:20.865496 9908 maintenance_manager-test.cc:131] Prepared op op7
I20250625 03:33:20.865634 9908 maintenance_manager.cc:419] P 12345: Scheduling op7: perf score=19.000000
I20250625 03:33:20.866106 9907 maintenance_manager-test.cc:138] Performing op op7
I20250625 03:33:20.867337 9907 maintenance_manager.cc:643] P 12345: op7 complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250625 03:33:20.867794 9908 maintenance_manager-test.cc:131] Prepared op op5
I20250625 03:33:20.867952 9908 maintenance_manager.cc:419] P 12345: Scheduling op5: perf score=12.000000
I20250625 03:33:20.868324 9907 maintenance_manager-test.cc:138] Performing op op5
I20250625 03:33:20.869567 9907 maintenance_manager.cc:643] P 12345: op5 complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250625 03:33:20.869983 9908 maintenance_manager-test.cc:131] Prepared op op4
I20250625 03:33:20.870113 9908 maintenance_manager.cc:419] P 12345: Scheduling op4: perf score=11.000000
I20250625 03:33:20.870538 9907 maintenance_manager-test.cc:138] Performing op op4
I20250625 03:33:20.871764 9907 maintenance_manager.cc:643] P 12345: op4 complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250625 03:33:20.872224 9908 maintenance_manager-test.cc:131] Prepared op op3
I20250625 03:33:20.872360 9908 maintenance_manager.cc:419] P 12345: Scheduling op3: perf score=10.000000
I20250625 03:33:20.872735 9907 maintenance_manager-test.cc:138] Performing op op3
I20250625 03:33:20.873966 9907 maintenance_manager.cc:643] P 12345: op3 complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250625 03:33:20.874330 9908 maintenance_manager-test.cc:131] Prepared op op2
I20250625 03:33:20.874449 9908 maintenance_manager.cc:419] P 12345: Scheduling op2: perf score=9.090909
I20250625 03:33:20.874792 9907 maintenance_manager-test.cc:138] Performing op op2
I20250625 03:33:20.876020 9907 maintenance_manager.cc:643] P 12345: op2 complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250625 03:33:20.876410 9908 maintenance_manager-test.cc:131] Prepared op op1
I20250625 03:33:20.876541 9908 maintenance_manager.cc:419] P 12345: Scheduling op1: perf score=6.209213
I20250625 03:33:20.876925 9907 maintenance_manager-test.cc:138] Performing op op1
I20250625 03:33:20.878141 9907 maintenance_manager.cc:643] P 12345: op1 complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager-test.cc:867: Failure
Expected equality of these values:
ordered_ops.front()
Which is: "op6"
instance.name()
Which is: "op7"
I20250625 03:33:20.882462 9863 test_util.cc:183] -----------------------------------------------
I20250625 03:33:20.882624 9863 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskQ70Ztu/test-tmp/maintenance_manager-test.0.MaintenanceManagerTest.TestPriorityOpLaunch.1750822398657305-9863-0
[ FAILED ] MaintenanceManagerTest.TestPriorityOpLaunch (29 ms)
[ RUN ] MaintenanceManagerTest.VerifyMetrics
I20250625 03:33:20.896391 9911 maintenance_manager.cc:411] P 12345: Prepare failed for oppf. Re-running scheduler.
I20250625 03:33:20.906749 9911 maintenance_manager-test.cc:131] Prepared op op0
I20250625 03:33:20.906898 9911 maintenance_manager.cc:419] P 12345: Scheduling op0: perf score=4.000000
I20250625 03:33:20.907335 9910 maintenance_manager-test.cc:138] Performing op op0
I20250625 03:33:20.907613 9910 maintenance_manager.cc:643] P 12345: op0 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250625 03:33:20.917323 9911 maintenance_manager-test.cc:131] Prepared op op1
I20250625 03:33:20.917465 9911 maintenance_manager.cc:419] P 12345: Scheduling op1: perf score=2.000000
I20250625 03:33:20.917886 9910 maintenance_manager-test.cc:138] Performing op op1
I20250625 03:33:20.918165 9910 maintenance_manager.cc:643] P 12345: op1 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
[ OK ] MaintenanceManagerTest.VerifyMetrics (46 ms)
[ RUN ] MaintenanceManagerTest.ManyOperationsHeavyUpdateStats
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager-test.cc:949: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] MaintenanceManagerTest.ManyOperationsHeavyUpdateStats (4 ms)
[ RUN ] MaintenanceManagerTest.TestUnregisterWhileScheduling
I20250625 03:33:20.936952 9917 maintenance_manager-test.cc:131] Prepared op 1
I20250625 03:33:20.937080 9917 maintenance_manager.cc:419] P 12345: Scheduling 1: perf score=10.000000
I20250625 03:33:20.937464 9917 maintenance_manager-test.cc:131] Prepared op 1
I20250625 03:33:20.937635 9917 maintenance_manager.cc:419] P 12345: Scheduling 1: perf score=10.000000
I20250625 03:33:20.937641 9916 maintenance_manager-test.cc:138] Performing op 1
I20250625 03:33:20.938174 9915 maintenance_manager-test.cc:138] Performing op 1
I20250625 03:33:20.938105 9916 maintenance_manager.cc:643] P 12345: 1 complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"spinlock_wait_cycles":131840}
I20250625 03:33:20.938416 9915 maintenance_manager.cc:643] P 12345: 1 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250625 03:33:20.938557 9917 maintenance_manager-test.cc:131] Prepared op 1
I20250625 03:33:20.938738 9917 maintenance_manager.cc:419] P 12345: Scheduling 1: perf score=10.000000
I20250625 03:33:20.939100 9917 maintenance_manager-test.cc:131] Prepared op 1
I20250625 03:33:20.939227 9917 maintenance_manager.cc:419] P 12345: Scheduling 1: perf score=10.000000
I20250625 03:33:20.939246 9915 maintenance_manager-test.cc:138] Performing op 1
I20250625 03:33:20.939697 9916 maintenance_manager-test.cc:138] Performing op 1
I20250625 03:33:20.939671 9915 maintenance_manager.cc:643] P 12345: 1 complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"spinlock_wait_cycles":265984}
I20250625 03:33:20.940017 9916 maintenance_manager.cc:643] P 12345: 1 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
[ OK ] MaintenanceManagerTest.TestUnregisterWhileScheduling (8 ms)
[ RUN ] MaintenanceManagerTest.ComprehensiveTest
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager-test.cc:1032: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] MaintenanceManagerTest.ComprehensiveTest (3 ms)
[----------] 16 tests from MaintenanceManagerTest (2275 ms total)
[----------] Global test environment tear-down
[==========] 16 tests from 1 test suite ran. (2276 ms total)
[ PASSED ] 12 tests.
[ SKIPPED ] 3 tests, listed below:
[ SKIPPED ] MaintenanceManagerTest.TestNewOpsDontGetScheduledDuringUnregister
[ SKIPPED ] MaintenanceManagerTest.ManyOperationsHeavyUpdateStats
[ SKIPPED ] MaintenanceManagerTest.ComprehensiveTest
[ FAILED ] 1 test, listed below:
[ FAILED ] MaintenanceManagerTest.TestPriorityOpLaunch
1 FAILED TEST