fedora-review – Package reviews made easier

November 11, 2011 in fedora, json, packaging, projects, python, review, tool

For package to be included in default Fedora repositories, it has to go through process called package review. If you’ve done a few package reviews you know big chunks of this process are repeated ad-nausea in every review.
There have been quite a few tools aimed at automating and simplifying this process. However they all had one major flaw. They have been designed for reviewing specific class of packages, be it Perl, Python or generic C/C++ packages. Few us us decided to change this.
We used Tim Lauridsen’s FedoraReview package as a base for our work and started adding new features and tweaks. Current work has a website on fedorahosted where you’ll find all important information. Full feature list would be quite long, but I’ll list a few major things:
  • Bugzilla integration
  • Mock integration
  • JSON api for external plugins (more info further down)
  • Several automated tests
The tool runs all checks/tests on spec file and rpms and writes output into a text file. Snippet of the output looks like this:

Package Review
==============

Key:
- = N/A
x = Check
! = Problem
? = Not evaluated



==== Generic ====
...
[ ]: MUST License field in the package spec file matches the actual license.
[ ]: MUST License file installed when any subpackage combination is installed.
[!]: MUST Package consistently uses macros (instead of hard-coded directory names).
Using both %{buildroot} and $RPM_BUILD_ROOT
...
[x]: SHOULD Spec use %global instead of %define.
...

==== Java ====
[!]: MUST If package uses "-Dmaven.local.depmap" explain why it was needed in a comment
[!]: MUST If package uses "-Dmaven.test.skip=true" explain why it was needed in a comment

Issues:
[!]: MUST Package consistently uses macros (instead of hard-coded directory names).
Using both %{buildroot} and $RPM_BUILD_ROOT
[!]: MUST If package uses "-Dmaven.local.depmap" explain why it was needed in a comment
[!]: MUST If package uses "-Dmaven.test.skip=true" explain why it was needed in a comment
We display only relevant results. In other words if there are no post/postun scriptlets there is no reason to include sanity output checking in the template. This will make more and more sense as we add more checks.

JSON API

So how is it that different people will be able to write additional plugins for this review tool? We provide a relatively simple JSON api through stdin/stdout.
So to create a new check plugin you create a script or program in your language of choice. There is only one requirement:
  • Programming language has to have JSON format support

When the review tool runs your plugin it will send following message on its stdin:


{
"supported_api": 1,
"pkgname": "package name",
"version": "package version",
"release": "package release",
"srpm":"path/to/srpm",
"spec":{ path: "path/to/spec",
"text": "spec text with expanded macros"},
"rpms":[ "path/to/rpm", ...],
"rpmlint": "rpmlint output",
"build_dir": "/path/to/src/directory/after/build"
}

When your plugin is done with checks it returns following message by writing it to stdout:


{
"command": "results",
"supported_api": 1,
"checks":
[
{
"name": "CheckName",
"url": "URL to guidelines usually",
"group": "Group for this test.(Java, Perl, etc.)",
"text": "Check description that shows on review template",
"deprecates":["DeprecatedTest", ...]
"type": "MUST"|"SHOULD",
"result": "pass"|"fail"|"inconclusive",
"extra_output": "text",
},
...
]
}

If the plugin closes stdout without writing anything there, it means there were no relevant automated tests to run and no non-automated tests to include in template for manual evaluation. This is useful so we don’t include for example Perl-related test output for Java packages and vice-versa.

Roadmap

While the tool is already usable and soon to be packaged in Fedora, there are still quite a few things we want to improve:

  • Add more functions to API (currently there is just get_section)
  • Automate all automatable tests currently available
  • Get rid of redundant tests (don’t duplicate rpmlint)
  • Add more tests of course!
  • Maybe add templating support?

We have currently 3-4 active committers, checks for C/C++, generic, Java, R packages. There is already and example external plugin written in Perl. If you have any improvement ideas, bugreports or just want to tell us we suck because we should have done X…get in touch!

Print expanded SourceX: urls from spec files

July 26, 2011 in fedora, packaging, python, rpm, script

I’ve noticed quite a few times that people add comments to their Source0: urls without macros to seemingly simplify manual downloading. It looks like this:

Name: jsoup
Version: 1.6.1
...
# http://jsoup.org/packages/jsoup-1.6.1-sources.jar
Source0: http://%{name}.org/packages/%{name}-%{version}-sources.jar

This creates burden on maintainers to keep those urls up-to-date as version changes, so I created simple python script for printing out Source urls from spec files:


#!/usr/bin/python

import rpm
import sys

ts=rpm.TransactionSet()
spec_obj = ts.parseSpec(sys.argv[1])

sources = spec_obj.sources

for url, num, flags in sources:
print url

Chmod this +x, put into your PATH and enjoy by giving it path to spec file.
*Edit*: Probably much nicer way to do the same thing already present on your system (courtesy of Alexander Kurtakov):

spectool X.spec

I knew there was something like this, but forgot what it was. Oh well…2 minutes lost.

Accessing WebDAV calendar from commandline

March 11, 2011 in open source, programming, projects, python, software

Have you tried to access Zimbra or Google Calendar from command line? I have. And I couldn’t find any normal command line client that would be able to read and write these calendars, display alerts etc. Well there is a googlecl project, but it’s specific for Google Calendar and is not using standard WebDav iCal access methods.

Thus I set out to create console application that would fulfil my needs. What are my requirements?:

  • Read/write access to Google Calendar and Zimbra (at least)
  • Multiple remote calendars
  • Working alerts
  • Nice ncurses UI (but also ability to just display some info and quit)
  • Correct handling of timezones
  • Integration with mail client (open ics files received by email)
  • I guess a lot more :-)

I had a look at existing python modules that work on iCalendar, WebDAV and combination of both. There are quite a few of them, but I just didn’t like their APIs. They were usually complex and required knowledge of iCal specification. So I decided to create simplified module that would be easy to understand (even if not so powerful).

I named the project pywebcal (yes, unimaginative) and it’s now on github. I would LOVE some input. I know it’s far from perfect (or complete), but let’s see. For now it offers read-only support for Zimbra (Google should work too but I haven’t tested in a while).

You can have a look at the example directory that contains one simple example you can run in-place and see if it works :-) I did my best to create proper test cases covering problems with timezones and whatnot, and this helped me quite a lot with recent refactoring. I am now using vobject library as my backend and it is rather nice to use. Plan is to allow access to vobject components so that my simplified API is not preventing some advanced modifications.

Next step is obviously to start working on ncurses application itself. Anyone wants to help?

PyQTrailer revisited

November 10, 2010 in en, fedora, open source, projects, pyqt, python, software

Some time ago, I wrote about my little project: apple.com trailer downloader. Apple is still not very open-source friendly as far as its trailer website is concerned. So all points I made in my original post still stand. To my surprise this little project is still alive and kicking, with new ideas for improvements coming and coming :-). What is even more important: it seems that so far no breakage happened due to apple changing something on the web.

Since the first version I released almost 6 months ago several new features appeared. Some of them include:

  • Parallel downloading of trailers
  • Ability to run movie player (mplayer, vlc, etc) without downloading file to HDD
  • Lot of customisation/performance options added
  • Working support for trailer search
  • Localisation support
  • Python 3 support

Latest version (0.5.2) is available in Gentoo repositories already, and should hit Fedora updates in next day or so (this will be delayed due to new package acceptance criteria though). Enjoy.

Python3, PyQt4 and missing QString

October 20, 2010 in howto, open source, programming, pyqt, python, qt

As I was recently adding support for Python 3 to my little trailer downloader application that I mentioned before (PyQTrailer) I encountered a strange problem with PyQt4 that only occurred in Python 3.

Let’s take this simple python example:


$ python
>>> from PyQt4.QtCore import QString
>>>

That same code snippet doesn’t work in python3 interpreter though:


$ python3
>>> from PyQt4.QtCore import QString
Traceback (most recent call last):
File "", line 1, in
ImportError: cannot import name QString
>>>

My first instinct was: Bug! Gentoo PyQt4 ebuild was doing something terrible and somehow made PyQt4 unusable in python3 interpreter. Turns out my gut instinct was wrong (once again :-) ).

PyQt4 since version 4.6 changed API of QString and QVariant for python 3.x. For QString this is due to fact that from Python 3.0, string literals are unicode objects (no need for u’unicode’ magic anymore). This means that you can use ordinary Python strings in place for QString. But I wanted my QString for something like this:


...
downloadClicked = pyqtSignal((QString, ))
...

This snippet creates Qt signals that you can then emit. Question is… How can we update this for Python 3.x? We could probably just replace QString with type(“”), but for a change that wouldn’t work with Python 2.x. So? Python dynamic nature to the rescue!
Edit: simplified QString definition (thanks Arfrever)


try:
from PyQt4.QtCore import QString
except ImportError:
# we are using Python3 so QString is not defined
QString = str

If we put previous code sample to the beginning of our Python file we can use QString in our code and it will keep working both in Python 3.x and Python 2.x. Case closed dear Watson.

Downloading trailers on Linux – final solution

May 10, 2010 in en, linux, open source, projects, python, software

I love films. All of them to be exact. I believe you just have to be in the right mood and you would enjoy even few of the worst movies ever made. Even though I am a proponent of open source philosophy, we as a society are obviously not ready to embrace it in entertainment industry just yet.

This is where www.apple.com/trailers comes into play. Apple made great deals with movie studios and you can watch/download newest movie trailers. Well…sort of. Apple employs variety of restrictions which makes this site next to useless on a Linux desktop. It hides links to trailers themselves behind reference files so that when you download with your favorite browser, you will only get small reference file not the trailer itself. And that is after you circumvent user-agent protection. Because apple believes nothing but iTunes/iPad/iOtherAppleStuff should access these trailers. There are scripts around that can make downloading possible for Linux users. I have been using Apple Trailer Download script for Greasemonkey for quite some time, but it always stopped working after some time.

Another opportunity for me I guess. I have been trying to improve my Python-fu for some time so what better way then a small project like this? I started last weekend after I found out Apple actually publishes JSON data of trailers on its site. This made access quite easy from python and is quite error-prone to changes of website itself (as long as Apple doesn’t pull whole JSON thingy…but they are actively using it too). Long story short…there are two outputs from this endeavor:

  • pytrailer – python module to simplify access to movies on apple.com/trailers
  • pyqtrailer – Qt4 interface that displays poster, movie information and enables downloading of trailers

You can report bugs on respective websites (there are quite a few now, but basic downloading for hd trailers works). If you want to try it out just running:

# easy_install pyqtrailer
should work as long as you have PyQt4 installed. You can just run pyqtrailer now and you should see something like this:

That’s it. I will improve/fix it a bit but don’t expect too much :-)

Technical decorating that makes sense

July 26, 2009 in en, gsoc, open source, projects, python

I have been using Python for several small-ish project in past year or two. In that time I have never found reason to really use decorators. You might have seen them in Java source codes in form of
@deprecated
void getVal()
{
//code
}

Python has same syntax, but because it’s scripting language certain things are different (read: more flexible :-) ).

Now to the main thing. Where did I use it? As I was deciding what ORM library/tool to use in my GSoC project, I came to conclusion that I could probably use Django DB backend to work with database. This way I would avoid doing the same thing (ORM) twice. Once for backend and once again for web interface later on. As always, things are not as straightforward as they seem in the beginning. Django is web framework and it’s tied with its database backend. In other words, the backend was not created with standalone usage in mind. Some things get a bit hairy because of that. These things are mostly connection management and exception handling.

Stackoverflow to the rescue (once again). There was already a question regarding use of only db part of django framework. Normally function like this in django:


would have to become this:


def add_package(self, name):
reset_queries()
try:
p = Package.objects.filter(name=name)
if len(p) > 0:
return p[0].id
p = Package(name=name)
p.save()
return p.id
except:
_rollback_on_exception()
finally:
close_connection()

Imagine that this exception handling, rollbacks and connection closing would have to be in every function. A bit ugly isn’t it? We cannot really use inheritance to our advantage, but we could use metaclass(es). I like look of decorators a bit better, so that’s what I used. So final code looks like this:


def dbquery(f):
def newfunc(*args, **kwargs):
reset_queries()
try:
return f(*args, **kwargs)
except Exception, e:
_rollback_on_exception()
raise e
return newfunc

@dbquery
def add_package(self, name):
p = Package.objects.filter(name=name)
if len(p) > 0:
return p[0].id
p = Package(name=name)
p.save()
return p.id
For other functions we could just add simple @dbquery in the beginning and voila, problem solved. Maybe there are even cleaner and/or better ways to do the same thing but at least I finally found non-trivial use for decorators.

Picasa Album Downloader roadmap

August 19, 2008 in en, google, java, projects, python

In my first post about Picasa Album Downloader java applet I promised more in-depth technical information about the project.

Project idea came when few of my less computer savvy friends wanted to download all photos from my Picasa Web Album. So far there have been few different ways to do that:

  • install Picasa and use it,
  • install some other software to download photos,
  • go photo-by-photo and download them one-by-one.

None of those methods is very user friendly. Why isn’t there a “Download album as zip archive” link on Picasa? I have a few theories, but that’s probably for another blog post :)

Question is: How to enable users to download Picasa albums easily? Apparently I was not the only one with the idea of creating web service to create zip file for users to download. Fortunately Google provides APIs for most of its services in few languages. More precisely you can access Picasa easily using:

  • .NET
  • Java
  • PHP
  • Python

Since I started learning Python step-by-step few months ago, I thought about using it for the job. Then I realized that I will need hosting for the web service. There are not too many free python hosting services. Those that are free usually have some restrictions.

Even Google provides hosting services using its own App Engine, with support for Python in particular. I created simple prototype python script that was able to download selected album from given user to the chosen output directory. It ran just fine when I was testing it, but stopped working when run inside dev_appserver.py. Reason? Hotlinking.

Picasa Web Album checks referer header and if it doesn’t match one of Google domains, Picasa blocks access to the images. Since App Engine dosn’t allow clearing of referer header, this effectively blocks using full scale images from Picasa in App Engine. So python is out. What else is there?

I don’t have much experience with .NET, and I also don’t think that it would be suitable for web application that is supposed to be free. I already had some experience with PHP and for project like this one, it would probably do the job just fine. There was a problem though…Google Data APIs needs at least PHP 5.14 to work, but hosting services I had at my disposal had lower versions installed.

Status? Python, .NET, PHP, Java. And here we are. The result is a Java applet that enables users to download full Picasa album without installing any software. There is also a project page at Google Code. First version took about 1 day to code. I released it under GPLv3, so if you want to contribute, you are welcome to do so. If you find any bugs or have ideas how to make the applet better, let me know.

Picasa Album Download

August 14, 2008 in en, google, java, projects, python

Picasa web albums is a great service. As far as I can tell it has very few disadvantages over competing websites. Although I have never used Flickr or similar services so I am not really one to judge.

There is one thing with Picasa web albums that quite a few people have asked me:

Can I download whole album from Picasa at once, without having to click through all the photos one-by-one?

Well I used to tell people to install Picasa to their computer, but less tech-savvy users had problems with this approach. Some companies also have restrictions on installing software in their networks. No wonder with numbers of trojans, malware and similar things on the Internet these days. Getting rid of them can take forever…

I found quite a few projects dealing with downloading from picasa. All of them required installing some application (or at least download one). Perfect solution? Web service.

As an aspiring Software engineer (pun intended) I set up on a quest to solve this problem once and for all. Goal:

  • Download complete Picasa Web Album into computer without having to install anything first
  • Multiplatform (Windows, Linux, MacOS X,…) support. Ideally only browser-requiring solution.

Simple right? Well yes and no. I will publish technical details and solutions I tried in some other post (edit: I already did). Now. without further ado, I present to you: