AutoQA 0.8 released

AutoQA 0.8 has just been released and deployed.

The most important changes are listed in the NEWS file. Here’s some selection that could be interesting for an average Fedora package maintainer:

  • Our Fedora 17 support was lagging behind. You should now receive reports for all your proposed updates for Fedora 17.
  • We finally have a database to store all the results. It’s publicly accessible and you can use it to search for results of a specific test executed on a specific package build. You can access it at autoqa.fedoraproject.org. We don’t yet have any notification support (like RSS), but it’s planned. Currently you just search for your favorite NVR and you’ll get all the tests we executed on it and their results. Please note this database has been started up today and it doesn’t contain any past results.
  • We now perform automated kickstarted anaconda installation for every new Fedora 17 compose. The test is called rats_install.

If you have some comments, bug reports or enhancement suggestions, please let us know. Our mailing list autoqa-devel or our Trac instance is at your disposal.

Advertisements
AutoQA 0.8 released

AutoQA 0.7 released

AutoQA 0.7 has just been released and it will be deployed to our production server soon.

Most important changes:

  • Minimum supported Python version is now Python 2.6. RHEL 5 became unsupported.
  • AutoQA library and package dependencies are now automatically copied to the test client prior to the test execution. It means that you no longer need to maintain your test clients except for initial set-up with the autotest server. All RPM installations are now handled out-of-the-box.
  • Initscripts test was removed due to systemd replacing SysV init scripts from Fedora 15 up.
  • AutoQA harness and AutoQA watchers now support standard logging facilities. Their output is now stored at /var/log/autoqa/. Debug log level can be enabled in autoqa.conf file.
  • Test results hyperlinks are now shortened using Yourls tool. A new configuration file yourls.conf is available. Link shortening applies for opt-in emails and Bodhi comments. Using URL shortening is optional, if you don’t set up Yourls server everything will work correctly but no links will be shortened.

If you have some comments, bug reports or enhancement suggestions, please let us know. Our mailing list autoqa-devel or our Trac instance is at your disposal.

AutoQA 0.7 released

AutoQA 0.6 released

AutoQA 0.6 has just been released and deployed.

Most important changes:

  • Upgradepath test checks FXX-updates-pending repositories now and if there is a build that fulfills upgrade path constraint then the test is marked as passed. Note that all such pending builds must be pushed at the same time in order to satisfy the upgrade path requirement. This means that package maintainers should no longer receive errors when pushing the same update simultaneously to several Fedora releases.
  • Depcheck test now operates on the latest versions of packages from the stable repositories. That should fix some of the encountered problems (ABRT guys know best).
  • There is a custom HTTP 404 page when accessing unavailable log file.

If you have some comments, bug reports or enhancement suggestions, please let us know. Our mailing list autoqa-devel or our Trac instance is at your disposal.

AutoQA 0.6 released

AutoQA 0.5.0 released

AutoQA 0.5.0 has just been released and deployed. Package maintainers will see some notable changes. What are they?

Most important changes include:

  • The result reports are package/update specific.
    Previously we tested e.g. 20 proposed updates and put the results into one big log (and you had to search for the one that’s yours). Now we create a separate report for every package/update. You won’t see redundant information in your report. In case you really need to see the whole log there is a link to “full log”.
  • Depcheck filtering
    The previous bullet was about tests in general. Things are a little more complex for depcheck test. When dependencies are involved, we can’t separate just your package/update from the others, because it all interacts together – you update may fail depcheck because of some other update. In order to shorten the log in this case we wrote a simple filter that parses the output and selects only those text excerpts that are likely to indicate the broken dependency. Instead of putting full depcheck output into your log, we put there only these excerpts (and you can display the “full log” if desired). Let’s hope this will help you to find the problem more easily. Please bear in mind that this is just a first attempt, we are likely to improve it in the future. Sometimes the depcheck output is just insanely long and this approach simply doesn’t help.
  • A new HTML format of result reports.
    That allows us to make pretty formatting in the report, highlight errors and generally make it more readable. For those who are opted-in for package build results, we will send you just a plain-text overview and a link to the HTML report, because we don’t want to send you HTML-formatted emails.
  • Substantial email notification reduction.
    Previously we posted test results as update comments in Bodhi  and Bodhi sent an email for every such comment. Thanks to Luke Macken we now have an option to modify this behavior and so we do. When all tests simply pass, Bodhi will send you no email at all. If some test fails, Bodhi will send you an email only after all tests are completed (i.e. when the last result arrives), it won’t spam you for every single test result separately. This should significantly lower the amount of email package maintainers receive from Bodhi. We made important part of this feature configurable, so we can easily adjust in the future according to people’s requests. Tim Flink will probably write a separate blogpost describing this improvement.
  • Test documentation.
    Some of the test results are not easy to read through and understand. We aim for having a thorough documentation for every single test case we execute. The two most visible test cases – depcheck and upgradepath – already have this documentation. You can find it under AutoQA tests category on Fedora wiki. The relevant wiki page will always be linked from inside the log. Please consult these docs when unsure why your test failed. We will continuously add more details to those pages. If you feel something is missing in there, let us know.
There are also some remaining problems we know of and want to fix soon:
  • Depcheck produces wrong results for (some) packages containing “Conflicts” and “Obsoletes” dependencies.
    That is ticket #325. Usually such packages are considered to have broken dependencies even though in reality they install just fine. It is a flaw in depcheck and needs to be fixed.
  • Upgradepath doesn’t take proposed updates for multiple releases into account.
    That is ticket #330. Basically if you propose an update for Fedora 14 and Fedora 15, only Fedora 15 update will pass upgradepath. Only after this update is pushed the Fedora 14 update will be marked as passed. But usually both of these updates are pushed at the same time, and not in the top-bottom approach. Technically our test is correct, but it’s not intuitive and friendly. We plan to improve that soon.
If you have some comments, bug reports or enhancement suggestions, please let us know. Our mailing list autoqa-devel or our Trac instance is at your disposal.
AutoQA 0.5.0 released

AutoQA: Why Upgradepath test fails so often?

As part of the AutoQA test suite we have an Upgradepath test that ensures that package dependencies won’t be broken when upgrading from one Fedora release to a higher one. There is often some confusion among package maintainers why it fails so often when they propose an update for several Fedora release at once. I’d like to explain.

Here’s an example failure log:

========================================
wordpress-3.1.2-1.fc13 into dist-f13-updates
========================================
[ OK ] dist-f13
	Latest package: wordpress-2.8.6-2.fc13
[FAIL] dist-f14 + dist-f14-updates
	Latest package: wordpress-3.1-1.fc14
	Error: Proposed package must be less than or equal to the latest package
[FAIL] dist-f15 + dist-f15-updates
	Latest package: wordpress-3.1-1.fc15
	Error: Proposed package must be less than or equal to the latest package
[ OK ] dist-f16
	Latest package: wordpress-3.1.2-1.fc16
RESULT: FAILED

The proposed update for F13-updates fails upgradepath, because that package exists in F14(+updates) and F15(+updates) only in lower versions. You wouldn’t be able to upgrade from F13 to F14 correctly if the proposed update landed.

But here’s the tricky part – wordpress maintainer proposed this update also for F14-updates and F15-updates, not just F13. And the obvious question now is:

Why does upgradepath fail for F13, when the same update for F14 and F15 is already proposed?

The answer is:

Because we can’t be sure all of them are going to be pushed at the same time.

Various things might happen. Let’s imagine RelEng team will push the update for F13, but will reject the same update for F14. Or the F14 update will get unpushed in the meantime. Or something else.

This calls for some kind of transactional behavior: “All these updates for F13, F14 and F15 must be pushed at the same time. Or just F15 update can be pushed. But no other way.” Unfortunately we can’t do this, yet. Until we can, there is only one way how to be perfectly sure that upgradepath constraint is fulfilled, and that means the updates for higher Fedora releases must land in the repos first.

Nevertheless we strive to be more user-friendly, especially concerning log readability and test understandability. It’s our current top priority. We will improve the logs to contain helpful information as much as we can, to explain this issue better. Very soon every test will have its own documentation wiki page. Upgradepath documentation is already available, please have a look at AutoQA test/Upgradepath. And we will continue to improve that.

If you have some suggestions, please send them to autoqa-devel. Thanks.

AutoQA: Why Upgradepath test fails so often?

Missing AutoQA logs

Maybe you noticed that some AutoQA result links that are posted as Bodhi comments (example) return “Not Found” error when you try to click on them. They can be just a few days old. That’s unfortunate and we have recently worked on fixing that issue.

First of all, some of our tests got stuck in a loop and generated many GBs of log files. Because our disk space is not immense, it quickly depleted our free space. On top of that, a bug in tmpwatch seemed to prune our log files much more frequently than we asked for. That resulted in deletion of many fresh logs.

We have provided quick fixes for those problems and everything should be back in normal. We aim for keeping the logs for at least a month.

If you have anything on your mind, contact as at autoqa-devel. Thanks.

Missing AutoQA logs

AutoQA 0.4.7 released

AutoQA 0.4.7 has just been released and deployed!

What are the major changes?

  • Upgradepath and depcheck tests won’t crash when there is a broken Bodhi update in which some packages are marked with updates-pending Koji tag and some are not.
  • Bodhi comments now emphasize that AutoQA results are purely informative, we don’t block anything.
  • AutoQA now correctly detects modified Bodhi updates and runs tests again for them.
  • Upgradepath no longer reports failures when you want to push to F(N)-updates repository and F(N+1)-main repository contains lower version of your package.

We also heard your complaints about sending you too many emails through Bodhi comments (one for every test result for every arch). Our current framework architecture does not allow miracles, but we plan to work hard in the next several weeks to alleviate at least some of the symptoms. And in the long-time perspective, it will be even better. The purpose of AutoQA is to help you, not to discourage you. Please continue to feed us with your comments, they are very helpful – email your ideas to autoqa-devel. Thanks.

And finally, our fun mascot 🙂

Happy weekend.

AutoQA 0.4.7 released