Skip to content
July 3, 2013 / Kamil Páral

The heroes of Fedora 19 Final testing

1360095720_PrizeFedora 19 the Schrödinger’s Cat is out of the box, and alive! I have again gathered some interesting statistics about QA contributions, this time during the Final phase (between Beta and Final release).

Fedora 19 Final – wiki matrices

This is the list of people who filled in our release validation wiki matrices (Install, Desktop and Base), which are posted for every test compose (and release candidate) that we create. The purpose is to see which areas have been thoroughly tested and which were not.

Test period: Fedora 19 Beta – Fedora 19 Final
Testers: 29
Reports: 1035
Unique referenced bugs: 58

Name Reports submitted Referenced bugs1
robatino 259 929177 946964 958426 958427 972025 972046 974050 974052 975227 975230 975324 975325 976280 976281 977669 977671 977715 977732 978114 978124 978125 979171 979172 (23)
nonamedotc 127 972225 977879 (2)
adamwill 103 858270 892178 975800 978008 (4)
kparal 103 973068 974032 974038 977715 977816 977962 (6)
Wutao85 55
lbrabec 49 855824 858270 977962 978298 (4)
jpospisi 48 964586 975483 (2)
eischmann 39
lnie 24
pkotvan 23 969684 975813 (2)
jskladan 21
tflink 21 975495 (1)
mkrizek 21 977962 (1)
jsedlak 19 973747 975375 978346 (3)
pschindl 18
lnovy 16 743281 858270 (2)
mmarhefk 16 892178 959796 975521 (3)
Martix 15 976417 (1)
boblfoot 12 971191 (1)
jreiser 12 976582 (1)
kevin 10
satellit 10
werkman 5 971109 971255 976034 (3)
todoleza 3 970988 (1)
konradr 2 977974 977987 978036 (3)
mattdm 1
jreznik 1 705086 (1)
mooninite 1
dgilmore 1

1 This is a list of bug reports linked to the wiki results. They don’t have to be reported by that concrete person.

Who would have thought… Andre Robatino (robatino) is at the top again. It’s also very nice to see how many bugs he referenced during his testing. The more referenced bugs the more publicity, the more likeliness that someone reproduces them, the more likeliness someone fixes them. Thanks, Andre!

The community participation list continues with nonamedotc, who provided an amazing number of test case results as well. Then there is a long list of Red Hatters, the highest score split between Adam Williamson (adamwill) and Kamil Páral (kparal) (a.k.a. me). Since I referenced more bugs, I consider myself a winner :-P “I only believe in statistics that I doctored myself.” :)

The community participation continues with Bob Lightfoot (boblfoot), who had been greatly active during Beta, John Reiser (jreiser), Thomas Gilliard (satellit) and others. Thank you, everyone.

Fedora 19 Final – Bugzilla

This is a trimmed list of people who reported bugs into Bugzilla against Fedora 19 in the specified time period. When compared to Fedora 19 Beta, the numbers went up again, especially the number of reporters. That’s really great.

Test period: Fedora 19 Beta – Fedora 19 Final (2013-05-29 – 2013-06-28)
Reporters: 614
New reports: 1486

Name Reports submitted1 Excess reports2 Accepted blockers3
IBM Bug Proxy 31 2 (6%) 0
Mark Salter 31 2 (6%) 0
Adam Williamson 27 1 (3%) 2
quickbooks.office at gmail.com 27 1 (3%) 1
Mikhail 26 1 (3%) 0
David Woodhouse 23 0 (0%) 0
Kamil Páral 22 2 (9%) 4
Andre Robatino 21 18 (85%) 1
Zbigniew Jędrzejewski-Szmek 17 0 (0%) 0
Dan Mashal 16 4 (25%) 0
Karel Volný 16 0 (0%) 0
Vojtěch Boček 13 0 (0%) 2
Emmanuel Pacaud 13 0 (0%) 0
Marian 12 3 (25%) 0
Jan Sedlák 11 0 (0%) 0
Lukas Brabec 11 1 (9%) 0
Martin Holec 11 2 (18%) 0
Tim Waugh 11 0 (0%) 0
A.J. Werkman 10 0 (0%) 0
Florian Weimer 10 0 (0%) 0
Heiko Adams 10 0 (0%) 0
Joachim Backes 10 3 (30%) 0
Chris Murphy 9 0 (0%) 1
Bojan Smojver 9 0 (0%) 0
Diogo Campos 9 0 (0%) 0
Jonas Thiem 9 0 (0%) 0
David Jaša 8 1 (12%) 0
Dean Hunter 8 1 (12%) 0
Endi Sukma Dewata 8 2 (25%) 0
James 8 2 (25%) 0
Jeff Bastian 7 1 (14%) 1
Jiří Martínek 7 1 (14%) 0
Marcus Moeller 7 3 (42%) 0
Matthew Miller 7 0 (0%) 0
Simon Lewis 7 1 (14%) 0
TKS 7 1 (14%) 0
Mark Hamzy 6 0 (0%) 1
Nix\ 6 2 (33%) 1
arnav 6 2 (33%) 0
Kalev Lember 6 1 (16%) 0
Krzysztof Daniel 6 0 (0%) 0
Rolle 6 1 (16%) 0
Russ Anderson 6 1 (16%) 0
Siddhesh Poyarekar 6 0 (0%) 0
James Heather 5 0 (0%) 1
Jiri Eischmann 5 1 (20%) 1
Michael Scherer 5 0 (0%) 1
Alexey Derlaft 5 2 (40%) 0
Andrea Oliveri 5 1 (20%) 0
Björn Esser 5 0 (0%) 0
D. Charles Pyle 5 0 (0%) 0
Dennis Gilmore 5 0 (0%) 0
Edgar Hoch 5 0 (0%) 0
Flóki Pálsson 5 1 (20%) 0
Francisco de la Peña 5 1 (20%) 0
Joshua Holm 5 0 (0%) 0
Konrad Rzeszutek Wilk 5 0 (0%) 0
Lukas -krtek.net- Novy 5 2 (40%) 0
mark 5 0 (0%) 0
Matthias Clasen 5 0 (0%) 0
nonamedotc at gmail.com 5 2 (40%) 0
Parag 5 2 (40%) 0
Paul Whalen 5 0 (0%) 0
Reartes Guillermo 5 0 (0%) 0
Vít Ondruch 5 0 (0%) 0
Álvaro Castillo 5 0 (0%) 0
Štefan Gurský 5 0 (0%) 0
…and also 547 other reporters who created less than 5 reports each, but 834 reports combined!

1 The total number of new reports (including “excess reports”). Reopened reports or reports with a changed version are not included, because it was not technically easy to retrieve those. This is one of the reasons why you shouldn’t take the numbers too seriously, but just as interesting and fun data.
2 Excess reports are those that were closed as NOTABUG, WONTFIX, WORKSFORME, CANTFIX or INSUFFICIENT_DATA. Excess reports are not necessarily a bad thing, but they make for interesting statistics. Close manual inspection is required to separate valuable excess reports from those which are less valuable.
3 This only includes reports that were created by that particular user and accepted as blockers afterwards. The user might have proposed other people’s reports as blockers, but this is not reflected in this number.

The top reporters are nicely and evenly distributed, there are no extreme performers as in the wiki statistics. The top position share IBM Bug Proxy and Mark Salter. Mark reported a number of build failures for ARM architecture. I assume IBM Bug Proxy is probably a shared account among IBM employees who care about Fedora development – they reported a number of anaconda, kernel and boot issues. It’s always nice to see other companies involved in raising the Fedora quality – welcome, IBM :-)

When it comes to general bug reporting, the top Red Hatter is Adam Williamson. As for the community, quickbooks.office reported the same number of bugs, most of them being SELinux error duplicates. Mikhail closely follows with a much more interesting mix of bug reports, which also applies for David Woodhouse. The community list follows with Andre Robatino (mostly fake bug reports for test purposes), Zbigniew Jędrzejewski-Szmek, Dan Mashal and others.

It’s great to see so many bug reports and so even distribution of the top reporters. Thanks, all of you.

Fedora 19 Beta – updates testing

This is a trimmed list of people who provided feedback in Bodhi for all updates proposed into Fedora 19 in the specified time period. The purpose is to make sure that broken updates do not enter the stable repository and the release continuously stabilizes. The number of testers increased considerably again as compared to Beta.

Test period: Fedora 19 Beta – Fedora 19 Final (2013-05-29 – 2013-06-28)
Testers: 240
Comments1: 1107

Name Updates commented
Igor Gnatenko (ignatenkobrain) 155
Adam Williamson (adamwill) 76
nonamedotc 71
Kalev Lember (kalev) 57
Alexander Kurtakov (akurtakov) 47
bitlord 45
T.C. Hollingsworth (patches) 41
quickbooks 35
Piotr Drąg (raven) 29
Kevin Fenzi (kevin) 28
Ankur Sinha (ankursinha) 20
keramidas 20
misc 20
Michael Schwendt (mschwendt) 16
Peter Robinson (pbrobinson) 15
Wolfgang Ulbrich (raveit65) 13
Dan Mashal (vicodan) 13
Matthias Clasen (mclasen) 12
Matthias Runge (mrunge) 11
bojan 10
Kamil Páral (kparal) 9
leigh123linux 8
D. Charles Pyle (dcharlespyle) 6
Pete Walter (pwalter) 6
Joachim Backes (backes) 6
mharmsen 6
Rex Dieter (rdieter) 5
Dennis Gilmore (ausil) 5
Ed Greshko (egreshko) 5
Dominic Hopf (dmaphy) 5
Endi Sukma Dewata (edewata) 5
Abhishek Koneru (kaskahn) 5
Kai Engert (kengert) 4
Jiri Eischmann (eischmann) 4
pnemade 4
Martin Kho (mkho) 4
Vojtech Bocek (tassadar) 4
Reindl Harald (hreindl) 4
Heiko Adams (heikoada) 4
Ian Malone (imalone) 3
Lukas Brabec (lbrabec) 3
Anton Arapov (aarapov) 3
Dan Horák (sharkcz) 3
Ray Strode (rstrode) 3
Hans Müller (cairo) 3
Greg Schlau (gregschlau) 3
Sérgio Monteiro Basto (sergiomb) 3
Nathan Kinder (nkinder) 3
gil 3
Martin Kosek (mkosek) 3
Artyom (artkun) 3
Roland Grunberg (rgrunber) 3
…and also 188 other reporters who created less than 3 reports each, but 235 reports combined!

1 If a person provides multiple comments to a single update, it is considered as a single comment. Karma value is not taken into account.

A new star has emerged, Igor Gnatenko (ignatenkobrain) completely dominated here. Great job, Igor! The top performing Red Hatter was, yet again, Adam Williamson (adamwill).

The top community list, apart from Igor, also contains nonamedotc, Kalev Lember (kalev), bitlord, T.C. Hollingsworth (patches), quickbooks, Piotr Drąg (raven), and others. It’s great to see so many people involved, because testing proposed updates and providing karma is one of the easiest way to contribute, yet extremely important at the same time. Thanks to anyone who contributed!

Summary

For most parts, the number of reports stays similar to Beta, but the number of people involved rose considerably. Beta release seems to be very attractive to the Fedora community and rightly so, it’s the last step before everything is pronounced stable. We’re glad to see so many people involved in release testing and we hope that we will see you again in Fedora 20 cycle!

In the meantime you can help as well – for example testing the proposed updates for stable Fedora releases is as useful, maybe even more important, than for releases in development. You can also play with Rawhide and help us find the bugs well in advance. There is a lot of possibilities – if it sounds interesting to you, be sure to have a look at QA/Join, follow the announcements and talk to us in #fedora-qa on IRC and test list. Your involvement will help not only Fedora and its users, but the whole Linux ecosystem. Thanks!


When reading the statistics, please take it with a grain of salt. The numbers are not directly comparable. People might see some reports as more valuable than others. Some people tested a lot of components, but haven’t found many problems (but that also helps). Some people used their skills in other areas than those mentioned. This is not meant to be a comparison chart, but a well-meant “thank you” letter.
The statistics were generated by these scripts.

4 Comments

Leave a Comment
  1. Adam Williamson / Jul 3 2013 22:38

    “IBM Bug Proxy” is more a ‘Red Hat’ thing than a Fedora thing, most reports from IBMers go through some kind of internal reporting system over at IBM which then sends them out ‘upstream’, so bugs to do with RH stuff come out to our Bugzilla. You see a lot of such reports if you look at RHEL bugs, IBM is a major RHEL customer. It is great that they’re testing F19, though, indeed.

  2. Peter Smith / Jul 4 2013 21:36

    What’s up with the Red Hat and community separation? You make it sound like the “community” is a second class member. Please don’t make this distinction and don’t post separate lists.

    • a _passerby / Jul 5 2013 01:41

      On the contrary, the point of the blog is to encourage community contributions by highlighting them. People might be happy to see their contributions “recognized”. The distinction being shown here is just to highlight the community contribution, in my opinion.

      Unlike the developers, testers are practically invisible. So, this blog by Kamil, in my opinion, will definitely encourage community participation.

    • Kamil Páral / Jul 8 2013 14:19

      I’m trying to highlight community members by explicitly naming them out in the description. I don’t do that for Red Hatters that much, just naming the first one, usually.

      Since the public doesn’t know who is a Red Hatter and who is not, people could easily say “all of those are Red Hatters anyway”. I’m trying to show that it’s not true (more often that not the most active members are from the community) and also thank these community people publicly. Red Hatters do these tasks often (although not always) as part of their daily job, which puts even more praise onto the community top-performers.

      I think the slight distinction is an interesting information, but if you have some advice how to it better, I’ll love to hear that.

Leave a Reply (Markdown syntax supported)

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: