AutoQA 0.8 released

AutoQA 0.8 has just been released and deployed.

The most important changes are listed in the NEWS file. Here’s some selection that could be interesting for an average Fedora package maintainer:

• Our Fedora 17 support was lagging behind. You should now receive reports for all your proposed updates for Fedora 17.
• We finally have a database to store all the results. It’s publicly accessible and you can use it to search for results of a specific test executed on a specific package build. You can access it at autoqa.fedoraproject.org. We don’t yet have any notification support (like RSS), but it’s planned. Currently you just search for your favorite NVR and you’ll get all the tests we executed on it and their results. Please note this database has been started up today and it doesn’t contain any past results.
• We now perform automated kickstarted anaconda installation for every new Fedora 17 compose. The test is called rats_install.

If you have some comments, bug reports or enhancement suggestions, please let us know. Our mailing list autoqa-devel or our Trac instance is at your disposal.

AutoQA 0.7 released

AutoQA 0.7 has just been released and it will be deployed to our production server soon.

Most important changes:

• Minimum supported Python version is now Python 2.6. RHEL 5 became unsupported.
• AutoQA library and package dependencies are now automatically copied to the test client prior to the test execution. It means that you no longer need to maintain your test clients except for initial set-up with the autotest server. All RPM installations are now handled out-of-the-box.
• Initscripts test was removed due to systemd replacing SysV init scripts from Fedora 15 up.
• AutoQA harness and AutoQA watchers now support standard logging facilities. Their output is now stored at /var/log/autoqa/. Debug log level can be enabled in autoqa.conf file.
• Test results hyperlinks are now shortened using Yourls tool. A new configuration file yourls.conf is available. Link shortening applies for opt-in emails and Bodhi comments. Using URL shortening is optional, if you don’t set up Yourls server everything will work correctly but no links will be shortened.

If you have some comments, bug reports or enhancement suggestions, please let us know. Our mailing list autoqa-devel or our Trac instance is at your disposal.

AutoQA 0.6 released

AutoQA 0.6 has just been released and deployed.

Most important changes:

• Upgradepath test checks FXX-updates-pending repositories now and if there is a build that fulfills upgrade path constraint then the test is marked as passed. Note that all such pending builds must be pushed at the same time in order to satisfy the upgrade path requirement. This means that package maintainers should no longer receive errors when pushing the same update simultaneously to several Fedora releases.
• Depcheck test now operates on the latest versions of packages from the stable repositories. That should fix some of the encountered problems (ABRT guys know best).
• There is a custom HTTP 404 page when accessing unavailable log file.

If you have some comments, bug reports or enhancement suggestions, please let us know. Our mailing list autoqa-devel or our Trac instance is at your disposal.

AutoQA 0.5.0 released

AutoQA 0.5.0 has just been released and deployed. Package maintainers will see some notable changes. What are they?

Most important changes include:

• The result reports are package/update specific.
  Previously we tested e.g. 20 proposed updates and put the results into one big log (and you had to search for the one that’s yours). Now we create a separate report for every package/update. You won’t see redundant information in your report. In case you really need to see the whole log there is a link to “full log”.
• Depcheck filtering
  The previous bullet was about tests in general. Things are a little more complex for depcheck test. When dependencies are involved, we can’t separate just your package/update from the others, because it all interacts together – you update may fail depcheck because of some other update. In order to shorten the log in this case we wrote a simple filter that parses the output and selects only those text excerpts that are likely to indicate the broken dependency. Instead of putting full depcheck output into your log, we put there only these excerpts (and you can display the “full log” if desired). Let’s hope this will help you to find the problem more easily. Please bear in mind that this is just a first attempt, we are likely to improve it in the future. Sometimes the depcheck output is just insanely long and this approach simply doesn’t help.
• A new HTML format of result reports.
  That allows us to make pretty formatting in the report, highlight errors and generally make it more readable. For those who are opted-in for package build results, we will send you just a plain-text overview and a link to the HTML report, because we don’t want to send you HTML-formatted emails.
• Substantial email notification reduction.
  Previously we posted test results as update comments in Bodhi  and Bodhi sent an email for every such comment. Thanks to Luke Macken we now have an option to modify this behavior and so we do. When all tests simply pass, Bodhi will send you no email at all. If some test fails, Bodhi will send you an email only after all tests are completed (i.e. when the last result arrives), it won’t spam you for every single test result separately. This should significantly lower the amount of email package maintainers receive from Bodhi. We made important part of this feature configurable, so we can easily adjust in the future according to people’s requests. Tim Flink will probably write a separate blogpost describing this improvement.
• Test documentation.
  Some of the test results are not easy to read through and understand. We aim for having a thorough documentation for every single test case we execute. The two most visible test cases – depcheck and upgradepath – already have this documentation. You can find it under AutoQA tests category on Fedora wiki. The relevant wiki page will always be linked from inside the log. Please consult these docs when unsure why your test failed. We will continuously add more details to those pages. If you feel something is missing in there, let us know.

There are also some remaining problems we know of and want to fix soon:

• Depcheck produces wrong results for (some) packages containing “Conflicts” and “Obsoletes” dependencies.
  That is ticket #325. Usually such packages are considered to have broken dependencies even though in reality they install just fine. It is a flaw in depcheck and needs to be fixed.
• Upgradepath doesn’t take proposed updates for multiple releases into account.
  That is ticket #330. Basically if you propose an update for Fedora 14 and Fedora 15, only Fedora 15 update will pass upgradepath. Only after this update is pushed the Fedora 14 update will be marked as passed. But usually both of these updates are pushed at the same time, and not in the top-bottom approach. Technically our test is correct, but it’s not intuitive and friendly. We plan to improve that soon.

If you have some comments, bug reports or enhancement suggestions, please let us know. Our mailing list autoqa-devel or our Trac instance is at your disposal.

AutoQA: Why Upgradepath test fails so often?

As part of the AutoQA test suite we have an Upgradepath test that ensures that package dependencies won’t be broken when upgrading from one Fedora release to a higher one. There is often some confusion among package maintainers why it fails so often when they propose an update for several Fedora release at once. I’d like to explain.

Here’s an example failure log:

========================================
wordpress-3.1.2-1.fc13 into dist-f13-updates
========================================
[ OK ] dist-f13
	Latest package: wordpress-2.8.6-2.fc13
[FAIL] dist-f14 + dist-f14-updates
	Latest package: wordpress-3.1-1.fc14
	Error: Proposed package must be less than or equal to the latest package
[FAIL] dist-f15 + dist-f15-updates
	Latest package: wordpress-3.1-1.fc15
	Error: Proposed package must be less than or equal to the latest package
[ OK ] dist-f16
	Latest package: wordpress-3.1.2-1.fc16
RESULT: FAILED

The proposed update for F13-updates fails upgradepath, because that package exists in F14(+updates) and F15(+updates) only in lower versions. You wouldn’t be able to upgrade from F13 to F14 correctly if the proposed update landed.

But here’s the tricky part – wordpress maintainer proposed this update also for F14-updates and F15-updates, not just F13. And the obvious question now is:

Why does upgradepath fail for F13, when the same update for F14 and F15 is already proposed?

The answer is:

Because we can’t be sure all of them are going to be pushed at the same time.

Various things might happen. Let’s imagine RelEng team will push the update for F13, but will reject the same update for F14. Or the F14 update will get unpushed in the meantime. Or something else.

This calls for some kind of transactional behavior: “All these updates for F13, F14 and F15 must be pushed at the same time. Or just F15 update can be pushed. But no other way.” Unfortunately we can’t do this, yet. Until we can, there is only one way how to be perfectly sure that upgradepath constraint is fulfilled, and that means the updates for higher Fedora releases must land in the repos first.

Nevertheless we strive to be more user-friendly, especially concerning log readability and test understandability. It’s our current top priority. We will improve the logs to contain helpful information as much as we can, to explain this issue better. Very soon every test will have its own documentation wiki page. Upgradepath documentation is already available, please have a look at AutoQA test/Upgradepath. And we will continue to improve that.

If you have some suggestions, please send them to autoqa-devel. Thanks.

Missing AutoQA logs

Maybe you noticed that some AutoQA result links that are posted as Bodhi comments (example) return “Not Found” error when you try to click on them. They can be just a few days old. That’s unfortunate and we have recently worked on fixing that issue.

First of all, some of our tests got stuck in a loop and generated many GBs of log files. Because our disk space is not immense, it quickly depleted our free space. On top of that, a bug in tmpwatch seemed to prune our log files much more frequently than we asked for. That resulted in deletion of many fresh logs.

We have provided quick fixes for those problems and everything should be back in normal. We aim for keeping the logs for at least a month.

If you have anything on your mind, contact as at autoqa-devel. Thanks.

AutoQA 0.4.7 released

AutoQA 0.4.7 has just been released and deployed!

What are the major changes?

• Upgradepath and depcheck tests won’t crash when there is a broken Bodhi update in which some packages are marked with updates-pending Koji tag and some are not.
• Bodhi comments now emphasize that AutoQA results are purely informative, we don’t block anything.
• AutoQA now correctly detects modified Bodhi updates and runs tests again for them.
• Upgradepath no longer reports failures when you want to push to F(N)-updates repository and F(N+1)-main repository contains lower version of your package.

We also heard your complaints about sending you too many emails through Bodhi comments (one for every test result for every arch). Our current framework architecture does not allow miracles, but we plan to work hard in the next several weeks to alleviate at least some of the symptoms. And in the long-time perspective, it will be even better. The purpose of AutoQA is to help you, not to discourage you. Please continue to feed us with your comments, they are very helpful – email your ideas to autoqa-devel. Thanks.

And finally, our fun mascot 🙂

Happy weekend.

AutoQA – upgradepath vs updates to multiple Fedora releases

If you are a package maintainer, you probably need sometimes to push the same update to several Fedora releases. But upgradepath may report failures for some of your updates. What’s wrong?

Let’s imagine this situation:

Packages in stable updates:

• foo-1.0-1.fc14
• foo-1.0-1.fc15
• foo-1.0-1.fc16

Now you decide to propose these updates:

• foo-1.1-1.fc14
• foo-1.1-1.fc15
• foo-1.1-1.fc16

The result may be following:

• foo-1.1-1.fc14 – upgradepath will FAIL because Fedora 15 still contains just foo-1.0
• foo-1.1-1.fc15 – upgradepath will FAIL because Fedora 16 still contains just foo-1.0
• foo-1.1-1.fc16 – upgradepath will PASS since there is no Fedora 17

The effective result is that you have to first push the update to the highest Fedora version (Rawhide) and only then you will see upgradepath produce PASS result for the update going to the previous version of Fedora.

You probably guessed that the reason for this behavior is that upgradepath considers every update individually, not as “set of updates”. In the future we would like to improve that and prepare these “set of updates safe to push” for RelEng team. But currently the aforementioned behavior was the one that was quick and easy to do. We plan many improvements in the future.

What are the implications?

1. If you want to have certainty that your update doesn’t break upgradepath constraint, push to the highest Fedora release first and then consequently to lower ones. Or at least inspect upgradepath logs if there is really no other issue with your update than this one.
2. If you want to push update for a single Fedora release, you don’t have to push for the others. Read Minor release bumps for old branches.
3. Upgradepath is re-evaluated for all pending updates regularly (and very often). If your update received a FAILED result, correct the situation and in a short time you should receive PASS result. If you update still fails, we will inform you only once in 3 days to avoid spamming.
4. Don’t forget that AutoQA results are purely informative, they do not block your actions.

AutoQA vs Bodhi – round 2

As you probably noticed not so long ago we announced that AutoQA would be sending comments to Bodhi and inform package maintainers about the results of important test cases (depcheck and ugpradepath). We enabled the functionality and disabled it again very soon due to a number of problems that were discovered. Good news – we’re going for round two!

In the first run we discovered some bugs in our test code, but we also found a number of packages that were incorrectly tagged as pending updates even though they had been in the stable repository for a long time already. The former problem should be solved by AutoQA 0.4.6 (and 0.4.5) which we are releasing today. The latter problem was fixed on Bodhi side by Luke Macken (huge thanks for quick response). Hopefully now the package maintainers won’t receive any alerts about already pushed package updates, only about the newly proposed ones.

AutoQA Bodhi comments should be re-enabled today. As mentioned before we will quickly disable them if something goes wrong, fix the problem, then re-enable them again. If you don’t receive any AutoQA results for your package, don’t be surprised. However if you receive results but they seem to be wrong, please tell us so that we can investigate: autoqa-devel or #fedora-qa. Thanks!

AutoQA will be providing comments for Fedora updates really soon…

If you’re a Fedora package maintainer, you will be surely interested in this topic.

AutoQA is a framework for automatic test execution written for Fedora needs. It is developed by Fedora QA team. We have finally reached the state where we can experimentally share some tests results with the wide public. They were already available in the autoqa-results mailing list, but that’s not really what we mean by public consumption. Let’s improve that.

Now prepare the drums… ready?… we are going to post the results into Fedora Update System (a.k.a. Bodhi)!

After you propose an update in Bodhi (either to -updates or to -updates-testing), you will start receiving AutoQA results as Bodhi comments.

A few very important notes:

• The test results are purely informative. We won’t block/reject your updates if they fail some test. But we hope that you (or RelEng team members) will have a look at the issue and it will help you/them to decide whether it is ok to push this update or not.
• The two tests which will be executed (described below) are designed to check for issues that don’t offer too much speculation. We don’t want new updates to break the dependencies, ever. We want new updates to keep upgradepath constraint, always. If the result is failed, there is either a problem in the update or a problem in our test. If you believe the latter is the case, please, report the problem to us – AutoQA, autoqa-devel. Those tests are very complex and there is at least one bug in every program. We know that it will be a bumpy road until we have them perfect.
• It’s very easy to disable Bodhi comment posting if something goes really wrong. We will probably do that if we discover some serious issue and re-enable it again after we fix it. So don’t be afraid, we won’t flood your inbox (the doomsday is not until 2012, you know), and also don’t be surprised if the comments don’t arrive sometimes.

There are two tests whose results will be posted to Bodhi:

1. depcheck – This test checks the dependencies (provides/requires/conflicts/obsoletes flags) of the proposed packages coming into -updates or -updates-testing. It tries to decide whether your update would break the repository (i.e. some packages would have unresolved dependencies in that case). There’s a lot of very complex logic behind it.
2. upgradepath – This test checks whether the upgradepath requirement is fulfilled. In an example with Fedora 13 and package foo-1.1-1.fc13 it means that every higher Fedora release (14, 15, 16) must contain same or higher NVR of the foo package than is the currently proposed one (and vice versa for lower Fedora releases). This constraint is currently checked only for -updates repository requests.

Next to every result you’ll see an URL pointing to the full results log. You’ll want to inspect that if your package was marked as failing some test. Example URL:

http://autoqa.fedoraproject.org/results/70273-autotest/qa03.c.fedoraproject.org/

The results directory is currently a little bit chaotic, but the important files are:

• <testname>/results/output.log where you’ll see a shortened output of the test with important highlights as the first lines
• debug/client.DEBUG contains full debug log as created by the framework

In those files you should be able to find the reason why your package failed the test (search for your NVR).

The tests are executed fairly often (almost for every update request), which means that after you correct the problem (if there was any), you should quickly receive another comment saying that your package now passes the test. If your package fails again however, you won’t receive another comment – we don’t want to spam you with Bodhi notification emails too often. Currently we wait at least three days before inserting two subsequent fail comments.

Feedback/comments welcome.