AutoQA 0.5.0 released

AutoQA 0.5.0 has just been released and deployed. Package maintainers will see some notable changes. What are they?

Most important changes include:

• The result reports are package/update specific.
  Previously we tested e.g. 20 proposed updates and put the results into one big log (and you had to search for the one that’s yours). Now we create a separate report for every package/update. You won’t see redundant information in your report. In case you really need to see the whole log there is a link to “full log”.
• Depcheck filtering
  The previous bullet was about tests in general. Things are a little more complex for depcheck test. When dependencies are involved, we can’t separate just your package/update from the others, because it all interacts together – you update may fail depcheck because of some other update. In order to shorten the log in this case we wrote a simple filter that parses the output and selects only those text excerpts that are likely to indicate the broken dependency. Instead of putting full depcheck output into your log, we put there only these excerpts (and you can display the “full log” if desired). Let’s hope this will help you to find the problem more easily. Please bear in mind that this is just a first attempt, we are likely to improve it in the future. Sometimes the depcheck output is just insanely long and this approach simply doesn’t help.
• A new HTML format of result reports.
  That allows us to make pretty formatting in the report, highlight errors and generally make it more readable. For those who are opted-in for package build results, we will send you just a plain-text overview and a link to the HTML report, because we don’t want to send you HTML-formatted emails.
• Substantial email notification reduction.
  Previously we posted test results as update comments in Bodhi  and Bodhi sent an email for every such comment. Thanks to Luke Macken we now have an option to modify this behavior and so we do. When all tests simply pass, Bodhi will send you no email at all. If some test fails, Bodhi will send you an email only after all tests are completed (i.e. when the last result arrives), it won’t spam you for every single test result separately. This should significantly lower the amount of email package maintainers receive from Bodhi. We made important part of this feature configurable, so we can easily adjust in the future according to people’s requests. Tim Flink will probably write a separate blogpost describing this improvement.
• Test documentation.
  Some of the test results are not easy to read through and understand. We aim for having a thorough documentation for every single test case we execute. The two most visible test cases – depcheck and upgradepath – already have this documentation. You can find it under AutoQA tests category on Fedora wiki. The relevant wiki page will always be linked from inside the log. Please consult these docs when unsure why your test failed. We will continuously add more details to those pages. If you feel something is missing in there, let us know.

There are also some remaining problems we know of and want to fix soon:

• Depcheck produces wrong results for (some) packages containing “Conflicts” and “Obsoletes” dependencies.
  That is ticket #325. Usually such packages are considered to have broken dependencies even though in reality they install just fine. It is a flaw in depcheck and needs to be fixed.
• Upgradepath doesn’t take proposed updates for multiple releases into account.
  That is ticket #330. Basically if you propose an update for Fedora 14 and Fedora 15, only Fedora 15 update will pass upgradepath. Only after this update is pushed the Fedora 14 update will be marked as passed. But usually both of these updates are pushed at the same time, and not in the top-bottom approach. Technically our test is correct, but it’s not intuitive and friendly. We plan to improve that soon.

If you have some comments, bug reports or enhancement suggestions, please let us know. Our mailing list autoqa-devel or our Trac instance is at your disposal.