Diagnosed failure
MaintenanceManagerTest.TestPriorityOpLaunch: /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager-test.cc:816: Failure
Expected equality of these values:
ordered_ops.front()
Which is: "op6"
instance.name()
Which is: "op7"
I20250115 10:20:44.743307 25454 test_util.cc:181] -----------------------------------------------
I20250115 10:20:44.743464 25454 test_util.cc:182] Had failures, leaving test files at /tmp/dist-test-taskCtqw8B/test-tmp/maintenance_manager-test.0.MaintenanceManagerTest.TestPriorityOpLaunch.1736936442490582-25454-0
Full log
[==========] Running 16 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 16 tests from MaintenanceManagerTest
[ RUN ] MaintenanceManagerTest.TestCreateAndShutdown
[ OK ] MaintenanceManagerTest.TestCreateAndShutdown (9 ms)
[ RUN ] MaintenanceManagerTest.TestRegisterUnregister
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250115 10:20:42.520123 25465 maintenance_manager-test.cc:129] Prepared op 1
I20250115 10:20:42.520452 25465 maintenance_manager.cc:418] P 12345: Scheduling 1: perf score=10.000000
I20250115 10:20:42.521356 25464 maintenance_manager-test.cc:136] Performing op 1
I20250115 10:20:42.521624 25464 maintenance_manager.cc:642] P 12345: 1 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
[ OK ] MaintenanceManagerTest.TestRegisterUnregister (10 ms)
[ RUN ] MaintenanceManagerTest.TestRegisterUnregisterWithContention
[ OK ] MaintenanceManagerTest.TestRegisterUnregisterWithContention (6 ms)
[ RUN ] MaintenanceManagerTest.TestNewOpsDontGetScheduledDuringUnregister
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager-test.cc:449: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] MaintenanceManagerTest.TestNewOpsDontGetScheduledDuringUnregister (4 ms)
[ RUN ] MaintenanceManagerTest.TestMemoryPressurePrioritizesMemory
I20250115 10:20:42.561401 25475 maintenance_manager-test.cc:129] Prepared op op
I20250115 10:20:42.561592 25475 maintenance_manager.cc:418] P 12345: Scheduling op: under memory pressure (0.00% used), 0 bytes log retention, and flush 100 bytes memory
I20250115 10:20:42.562284 25474 maintenance_manager-test.cc:136] Performing op op
I20250115 10:20:42.562551 25474 maintenance_manager.cc:642] P 12345: op complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
[ OK ] MaintenanceManagerTest.TestMemoryPressurePrioritizesMemory (29 ms)
[ RUN ] MaintenanceManagerTest.TestMemoryPressurePerformsNoMemoryOp
I20250115 10:20:42.591506 25478 maintenance_manager-test.cc:129] Prepared op op
I20250115 10:20:42.591693 25478 maintenance_manager.cc:418] P 12345: Scheduling op: perf score=1.000000
I20250115 10:20:42.592358 25477 maintenance_manager-test.cc:136] Performing op op
I20250115 10:20:42.592612 25477 maintenance_manager.cc:642] P 12345: op complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
[ OK ] MaintenanceManagerTest.TestMemoryPressurePerformsNoMemoryOp (30 ms)
[ RUN ] MaintenanceManagerTest.TestLogRetentionPrioritization
[ OK ] MaintenanceManagerTest.TestLogRetentionPrioritization (5 ms)
[ RUN ] MaintenanceManagerTest.TestPrioritizeLogRetentionUnderMemoryPressure
[ OK ] MaintenanceManagerTest.TestPrioritizeLogRetentionUnderMemoryPressure (4 ms)
[ RUN ] MaintenanceManagerTest.TestRunningInstances
I20250115 10:20:42.609781 25487 maintenance_manager-test.cc:129] Prepared op op
I20250115 10:20:42.609956 25487 maintenance_manager.cc:418] P 12345: Scheduling op: perf score=10.000000
I20250115 10:20:42.610402 25487 maintenance_manager-test.cc:129] Prepared op op
I20250115 10:20:42.610558 25487 maintenance_manager.cc:418] P 12345: Scheduling op: perf score=10.000000
I20250115 10:20:42.610571 25486 maintenance_manager-test.cc:136] Performing op op
I20250115 10:20:43.657117 25485 maintenance_manager-test.cc:136] Performing op op
I20250115 10:20:44.657307 25486 maintenance_manager.cc:642] P 12345: op complete. Timing: real 2.047s user 0.003s sys 0.008s Metrics: {"spinlock_wait_cycles":82176}
I20250115 10:20:44.659544 25485 maintenance_manager.cc:642] P 12345: op complete. Timing: real 2.048s user 0.006s sys 0.000s Metrics: {"spinlock_wait_cycles":2405250816}
[ OK ] MaintenanceManagerTest.TestRunningInstances (2056 ms)
[ RUN ] MaintenanceManagerTest.TestCompletedOpsHistory
I20250115 10:20:44.667181 25491 maintenance_manager-test.cc:129] Prepared op op0
I20250115 10:20:44.667337 25491 maintenance_manager.cc:418] P 12345: Scheduling op0: perf score=1.000000
I20250115 10:20:44.667842 25490 maintenance_manager-test.cc:136] Performing op op0
I20250115 10:20:44.668115 25490 maintenance_manager.cc:642] P 12345: op0 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250115 10:20:44.670817 25491 maintenance_manager-test.cc:129] Prepared op op1
I20250115 10:20:44.670957 25491 maintenance_manager.cc:418] P 12345: Scheduling op1: perf score=1.000000
I20250115 10:20:44.671370 25490 maintenance_manager-test.cc:136] Performing op op1
I20250115 10:20:44.671588 25490 maintenance_manager.cc:642] P 12345: op1 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250115 10:20:44.675374 25491 maintenance_manager-test.cc:129] Prepared op op2
I20250115 10:20:44.675504 25491 maintenance_manager.cc:418] P 12345: Scheduling op2: perf score=1.000000
I20250115 10:20:44.675961 25490 maintenance_manager-test.cc:136] Performing op op2
I20250115 10:20:44.676193 25490 maintenance_manager.cc:642] P 12345: op2 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250115 10:20:44.678836 25491 maintenance_manager-test.cc:129] Prepared op op3
I20250115 10:20:44.678984 25491 maintenance_manager.cc:418] P 12345: Scheduling op3: perf score=1.000000
I20250115 10:20:44.679432 25490 maintenance_manager-test.cc:136] Performing op op3
I20250115 10:20:44.679690 25490 maintenance_manager.cc:642] P 12345: op3 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250115 10:20:44.681231 25491 maintenance_manager-test.cc:129] Prepared op op4
I20250115 10:20:44.681372 25491 maintenance_manager.cc:418] P 12345: Scheduling op4: perf score=1.000000
I20250115 10:20:44.681851 25490 maintenance_manager-test.cc:136] Performing op op4
I20250115 10:20:44.682157 25490 maintenance_manager.cc:642] P 12345: op4 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250115 10:20:44.684772 25491 maintenance_manager-test.cc:129] Prepared op op5
I20250115 10:20:44.684916 25491 maintenance_manager.cc:418] P 12345: Scheduling op5: perf score=1.000000
I20250115 10:20:44.685305 25490 maintenance_manager-test.cc:136] Performing op op5
I20250115 10:20:44.685547 25490 maintenance_manager.cc:642] P 12345: op5 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250115 10:20:44.689376 25491 maintenance_manager-test.cc:129] Prepared op op6
I20250115 10:20:44.689538 25491 maintenance_manager.cc:418] P 12345: Scheduling op6: perf score=1.000000
I20250115 10:20:44.690012 25490 maintenance_manager-test.cc:136] Performing op op6
I20250115 10:20:44.690312 25490 maintenance_manager.cc:642] P 12345: op6 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250115 10:20:44.692929 25491 maintenance_manager-test.cc:129] Prepared op op7
I20250115 10:20:44.693076 25491 maintenance_manager.cc:418] P 12345: Scheduling op7: perf score=1.000000
I20250115 10:20:44.693476 25490 maintenance_manager-test.cc:136] Performing op op7
I20250115 10:20:44.693713 25490 maintenance_manager.cc:642] P 12345: op7 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250115 10:20:44.695202 25491 maintenance_manager-test.cc:129] Prepared op op8
I20250115 10:20:44.695346 25491 maintenance_manager.cc:418] P 12345: Scheduling op8: perf score=1.000000
I20250115 10:20:44.695739 25490 maintenance_manager-test.cc:136] Performing op op8
I20250115 10:20:44.695960 25490 maintenance_manager.cc:642] P 12345: op8 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250115 10:20:44.699885 25491 maintenance_manager-test.cc:129] Prepared op op9
I20250115 10:20:44.700062 25491 maintenance_manager.cc:418] P 12345: Scheduling op9: perf score=1.000000
I20250115 10:20:44.700508 25490 maintenance_manager-test.cc:136] Performing op op9
I20250115 10:20:44.700723 25490 maintenance_manager.cc:642] P 12345: op9 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250115 10:20:44.703357 25491 maintenance_manager-test.cc:129] Prepared op op10
I20250115 10:20:44.703478 25491 maintenance_manager.cc:418] P 12345: Scheduling op10: perf score=1.000000
I20250115 10:20:44.703851 25490 maintenance_manager-test.cc:136] Performing op op10
I20250115 10:20:44.704072 25490 maintenance_manager.cc:642] P 12345: op10 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
[ OK ] MaintenanceManagerTest.TestCompletedOpsHistory (46 ms)
[ RUN ] MaintenanceManagerTest.TestOpFactors
[ OK ] MaintenanceManagerTest.TestOpFactors (4 ms)
[ RUN ] MaintenanceManagerTest.TestPriorityOpLaunch
I20250115 10:20:44.718564 25499 maintenance_manager-test.cc:129] Prepared op canary
I20250115 10:20:44.718731 25499 maintenance_manager.cc:418] P 12345: Scheduling canary: perf score=1.000000
I20250115 10:20:44.719332 25498 maintenance_manager-test.cc:136] Performing op canary
I20250115 10:20:44.719571 25498 maintenance_manager.cc:642] P 12345: canary complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250115 10:20:44.723662 25499 maintenance_manager-test.cc:129] Prepared op op6
I20250115 10:20:44.723822 25499 maintenance_manager.cc:418] P 12345: Scheduling op6: perf score=16.105100
I20250115 10:20:44.724300 25498 maintenance_manager-test.cc:136] Performing op op6
I20250115 10:20:44.725529 25498 maintenance_manager.cc:642] P 12345: op6 complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250115 10:20:44.725973 25499 maintenance_manager-test.cc:129] Prepared op op7
I20250115 10:20:44.726111 25499 maintenance_manager.cc:418] P 12345: Scheduling op7: perf score=19.000000
I20250115 10:20:44.726469 25498 maintenance_manager-test.cc:136] Performing op op7
I20250115 10:20:44.727691 25498 maintenance_manager.cc:642] P 12345: op7 complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250115 10:20:44.728142 25499 maintenance_manager-test.cc:129] Prepared op op5
I20250115 10:20:44.728271 25499 maintenance_manager.cc:418] P 12345: Scheduling op5: perf score=12.000000
I20250115 10:20:44.728732 25498 maintenance_manager-test.cc:136] Performing op op5
I20250115 10:20:44.729974 25498 maintenance_manager.cc:642] P 12345: op5 complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250115 10:20:44.730367 25499 maintenance_manager-test.cc:129] Prepared op op4
I20250115 10:20:44.730496 25499 maintenance_manager.cc:418] P 12345: Scheduling op4: perf score=11.000000
I20250115 10:20:44.730850 25498 maintenance_manager-test.cc:136] Performing op op4
I20250115 10:20:44.732082 25498 maintenance_manager.cc:642] P 12345: op4 complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250115 10:20:44.732528 25499 maintenance_manager-test.cc:129] Prepared op op3
I20250115 10:20:44.732661 25499 maintenance_manager.cc:418] P 12345: Scheduling op3: perf score=10.000000
I20250115 10:20:44.733033 25498 maintenance_manager-test.cc:136] Performing op op3
I20250115 10:20:44.734287 25498 maintenance_manager.cc:642] P 12345: op3 complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250115 10:20:44.734715 25499 maintenance_manager-test.cc:129] Prepared op op2
I20250115 10:20:44.734853 25499 maintenance_manager.cc:418] P 12345: Scheduling op2: perf score=9.090909
I20250115 10:20:44.735229 25498 maintenance_manager-test.cc:136] Performing op op2
I20250115 10:20:44.736454 25498 maintenance_manager.cc:642] P 12345: op2 complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250115 10:20:44.736862 25499 maintenance_manager-test.cc:129] Prepared op op1
I20250115 10:20:44.736986 25499 maintenance_manager.cc:418] P 12345: Scheduling op1: perf score=6.209213
I20250115 10:20:44.737350 25498 maintenance_manager-test.cc:136] Performing op op1
I20250115 10:20:44.738591 25498 maintenance_manager.cc:642] P 12345: op1 complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager-test.cc:816: Failure
Expected equality of these values:
ordered_ops.front()
Which is: "op6"
instance.name()
Which is: "op7"
I20250115 10:20:44.743307 25454 test_util.cc:181] -----------------------------------------------
I20250115 10:20:44.743464 25454 test_util.cc:182] Had failures, leaving test files at /tmp/dist-test-taskCtqw8B/test-tmp/maintenance_manager-test.0.MaintenanceManagerTest.TestPriorityOpLaunch.1736936442490582-25454-0
[ FAILED ] MaintenanceManagerTest.TestPriorityOpLaunch (29 ms)
[ RUN ] MaintenanceManagerTest.VerifyMetrics
I20250115 10:20:44.757529 25502 maintenance_manager.cc:410] P 12345: Prepare failed for oppf. Re-running scheduler.
I20250115 10:20:44.767838 25502 maintenance_manager-test.cc:129] Prepared op op0
I20250115 10:20:44.768023 25502 maintenance_manager.cc:418] P 12345: Scheduling op0: perf score=4.000000
I20250115 10:20:44.768611 25501 maintenance_manager-test.cc:136] Performing op op0
I20250115 10:20:44.768895 25501 maintenance_manager.cc:642] P 12345: op0 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250115 10:20:44.778581 25502 maintenance_manager-test.cc:129] Prepared op op1
I20250115 10:20:44.778766 25502 maintenance_manager.cc:418] P 12345: Scheduling op1: perf score=2.000000
I20250115 10:20:44.779263 25501 maintenance_manager-test.cc:136] Performing op op1
I20250115 10:20:44.779523 25501 maintenance_manager.cc:642] P 12345: op1 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
[ OK ] MaintenanceManagerTest.VerifyMetrics (47 ms)
[ RUN ] MaintenanceManagerTest.ManyOperationsHeavyUpdateStats
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager-test.cc:898: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] MaintenanceManagerTest.ManyOperationsHeavyUpdateStats (4 ms)
[ RUN ] MaintenanceManagerTest.TestUnregisterWhileScheduling
I20250115 10:20:44.799332 25508 maintenance_manager-test.cc:129] Prepared op 1
I20250115 10:20:44.799497 25508 maintenance_manager.cc:418] P 12345: Scheduling 1: perf score=10.000000
I20250115 10:20:44.800056 25508 maintenance_manager-test.cc:129] Prepared op 1
I20250115 10:20:44.800225 25508 maintenance_manager.cc:418] P 12345: Scheduling 1: perf score=10.000000
I20250115 10:20:44.800262 25507 maintenance_manager-test.cc:136] Performing op 1
I20250115 10:20:44.800750 25506 maintenance_manager-test.cc:136] Performing op 1
I20250115 10:20:44.800668 25507 maintenance_manager.cc:642] P 12345: 1 complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"spinlock_wait_cycles":360832}
I20250115 10:20:44.800994 25506 maintenance_manager.cc:642] P 12345: 1 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250115 10:20:44.801168 25508 maintenance_manager-test.cc:129] Prepared op 1
I20250115 10:20:44.801347 25508 maintenance_manager.cc:418] P 12345: Scheduling 1: perf score=10.000000
I20250115 10:20:44.801779 25508 maintenance_manager-test.cc:129] Prepared op 1
I20250115 10:20:44.802008 25506 maintenance_manager-test.cc:136] Performing op 1
I20250115 10:20:44.801987 25508 maintenance_manager.cc:418] P 12345: Scheduling 1: perf score=10.000000
I20250115 10:20:44.802376 25506 maintenance_manager.cc:642] P 12345: 1 complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"spinlock_wait_cycles":417664}
I20250115 10:20:44.802577 25507 maintenance_manager-test.cc:136] Performing op 1
I20250115 10:20:44.802841 25507 maintenance_manager.cc:642] P 12345: 1 complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
[ OK ] MaintenanceManagerTest.TestUnregisterWhileScheduling (8 ms)
[ RUN ] MaintenanceManagerTest.ComprehensiveTest
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/maintenance_manager-test.cc:981: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] MaintenanceManagerTest.ComprehensiveTest (3 ms)
[----------] 16 tests from MaintenanceManagerTest (2304 ms total)
[----------] Global test environment tear-down
[==========] 16 tests from 1 test suite ran. (2305 ms total)
[ PASSED ] 12 tests.
[ SKIPPED ] 3 tests, listed below:
[ SKIPPED ] MaintenanceManagerTest.TestNewOpsDontGetScheduledDuringUnregister
[ SKIPPED ] MaintenanceManagerTest.ManyOperationsHeavyUpdateStats
[ SKIPPED ] MaintenanceManagerTest.ComprehensiveTest
[ FAILED ] 1 test, listed below:
[ FAILED ] MaintenanceManagerTest.TestPriorityOpLaunch
1 FAILED TEST