Command-line access to GNOME keyring

gnome-keyringA few days back I suddenly needed to access GNOME keyring from a command-line, to be used in my backup script. Of course I can save my passwords in plaintext in files like .netrc, but why should I, when all my passwords are already stored securely in the GNOME keyring? I have googled and googled and found nothing, there is no tool for accessing GNOME keyring from shell (until now). But I have found a simple Python module though (kudos to Sebastian Rittau).

I’m a complete beginner with Python, but I said ‘Hey, that’s a good opportunity to learn it a little bit!’.  So I present you my outstanding result, the gkeyring project: 🙂

https://launchpad.net/gkeyring

Retrieving your secrets is as simple as:

$ ./gkeyring.py get -s myserver.com -p ftp -u user

This will look for all keyring items concerning server myserver.com with protocol ftp and username user. Note: Currently the script is only able to access network passwords (which is what I needed).

The first time you access the keyring item you will be prompted whether to allow the script to access it. You can click “Always” and then it won’t ask again. Finally you just cut the output a little and you are done! Ready to be used in your script.

Hope that helps someone (it does help me). If you feel like improving the script, please do!

zsync – transfer large files efficiently

downloadA few days ago I have stumbled upon a zsync tool used for a fast transfer of very large files. The reason I have noticed it is because Ubuntu started to use it for its daily live images. And because I am curious, I have studied it and realized that zsync is great! 🙂 And I have also created some tests to see how well it works.

So what is the zsync?

zsync is a file transfer program. It allows you to download a file from a remote server, where you have a copy of an older version of the file on your computer already. zsync downloads only the new parts of the file. It uses the same algorithm as rsync. However, where rsync is designed for synchronising data from one computer to another within an organisation, zsync is designed for file distribution, with one file on a server to be distributed to thousands of downloaders. zsync requires no special server software — just a web server to host the files — and imposes no extra load on the server, making it ideal for large scale file distribution.

How does it work?

You simply generate a small .zsync file on the server for each big file you offer users to download. This .zsync file contains description of the contents of the big file. The user can then use the “zsync” tool with your .zsync file as an argument and use arbitrary file as a base for the new big file. It can be really any file, a previous version of the big file, a pre-pre-pre-previous version of the big file, or even a newer version of it! But the more it is similar the better. The zsync tool will then compare the files and download only the differences needed to assemble the new big file.

Usages

zsync is a great tool for developers who regularly download updated big files like daily CD/DVD images and similar stuff. It can really save you a lot of time and bandwidth it the files haven’t changed a lot. Especially if you have a slow internet line or the server has a slow line you might appreciate it. So what’re you waiting for?

Pitfalls

Oh, I haven’t told you yet. zsync is not currently available in Fedora because of some license problems 😦 But you can always find RPMs elsewhere.

Testing the efficiency

Here are the tests I have performed to see how well it works.

Ubuntu Karmic development images:

base image resulting image download size saved bandwidth
20090830 (703 MB) 20090831 (687 MB) 57 MB 91.7%
20090831 (687 MB) 20090901 (687 MB) 1 MB 99.9%
20090830 (703 MB) 20090901 (687 MB) 57 MB 91.7%
20090901 (687 MB) 20090830 (703 MB) 73 MB 89.6%

As you can see, zsync works great for Ubuntu images. When updating very recent images, you can save about 90% of bandwidth (and time). Unfortunately I haven’t had any older images to try to update from not-so-recent image. But notice that with zsync you can also go “back in time” from a more recent image to an older one (the last line).

Fedora 12 Alpha images:

base image resulting image download size saved bandwidth
RC1 (701 MB) RC2 (705 MB) 241 MB 65.8%
RC2 (705 MB) Final (705 MB) <1 MB 100.0%
RC1 (701 MB) Final (705 MB) 241 MB 65.8%
Final (705 MB) RC1 (701 MB) 237 MB 66.2%

For Fedora 12 Alpha images you can still save more than half when updating from RC1 to RC2 and you will be extremely pleased when updating from RC2 to Alpha Final. Although those two files are not the same, they are nearly the same (probably just some system label has changed) and you will get the image instantly.

Fedora 12 nightly composes:

base image resulting image download size saved bandwidth
20090809 (703 MB) 20090818 (702 MB) 287 MB 59.1%
20090818 (702 MB) 20090824 (630 MB) 404 MB 35.8%
20090824 (630 MB) 20090825 (636 MB) 361 MB 43.3%
20090825 (636 MB) 20090827 (640 MB) 340 MB 46.9%
20090809 (703 MB) 20090827 (640 MB) 423 MB 33.9%
20090827 (640 MB) 20090809 (703 MB) 486 MB 30.8%

For Fedora 12 nightly composes the savings number is apparently not so great as for Ubuntu images. Quite interestingly some long term updates may be more efficient than some short term updates – it depends how much the contents have changed over time. You still get around 45% of saved bandwidth, which is very nice. Why the Fedora images differ so much as opposed to Ubuntu images? You can read the Oxf13’s explanation on Fedora QA IRC meeting – the whole ISO is a squashfs image, that is not suitable for this kind of difference comparison. Ubuntu is probably using other method, which is more compatible with zsync algorithm.

Conclusion

So what do you think? Should there be .zsync files for ISO images in Fedora? I hope this article will help the infrastructure team to decide 🙂