Belated FOSDEM 2013 trip report

February 12, 2013 in conference, fedora, fosdem, packaging, report

I’ve been to FOSDEM this year (again) and as always it’s been an interesting experience. You meet all kinds of people, you can create new contacts and realize different projects can share some code or at least approaches. This is just going to be a rundown of talks that I’ve seen (or most of them anyway).

(R)evolution of Java Packaging in GNU/Linux

I’ll start with talk Mikolaj Izdebski and myself gave in Java devroom titled (R)evolution of Java Packaging in GNU/Linux, since that resulted in some possible interesting cooperation. The  talk went reasonably well without any hiccups. We could have used more time for discussion and generic talk about Java buildsytems, but oh well. 

After the talk Charles Oliver Nutter (headius @ JRuby/JVM projects) mentioned Sonatype is already running conversion of all jar files to rubygems on their infrastructure. His idea (which seems pretty neat) was to make this for RPMs as well. He got us in touch with Jason van Zyl and it seems if we create conversion tool we might get this to run on their infrastructure. This actually ties in nicely with some work we are doing to improve life for Java developers on Fedora because they’d be able to install unlimited number of different Maven artifacts that could tie in directly to Yum/RPM and companies could limit access to Maven Central repository and just allow these RPMs.

The state of OpenJDK & OpenJDK7u & OpenJDK Q&A

These were talks by Oracle guys giving updates where JVM is going, and how does the schedule look. Nothing too interesting or worrying here, but it seems that Oracle is really trying to put more people working on JVM, plus there’s community participation as well. Mark Reinhold’s prediction was that OpenJDK board will stay the same after reelections in next few months.

Interestingly Q&A at the end of Saturday was waaay more calm than previous years. Maybe that has to do something with community around OpenJDK maturing? :-)

Porting OpenJDK to AArch64

One of the more technical talks about intricacies of simulating hardware that (almost) doesn’t exist. Our Andrews (Andrew Haley and Andrew Dinn) had some hard drive issues but they handled it pretty gracefully. The  trick with jumping between AArch64 and x86 code outside of simulator was quite interesting. Also interesting were notes on register usage by JVM and how number of registers helps or hinders implementations.

Community Management in Meat Space

Talk by our very own Leslie Hawthorn and Lydia Pintscher. Basically it was a quick tutorial for dealing with decision making and problems within communities. A sort of small prelude to Leslie’s keynote. It just had more duct tape :-0

Gentoo Hardened

This talk was done by one of more controversial people (one of co-developers of eudev which got a lot of flaming apparently). Francisco Riera used a volunteer to show various features of hardened kernels (not necessarily SELinux style) in Gentoo, how and why they operate in certain ways. Each change was accompanied with quite nice examples how in non-hardened kernels certain things could be misused. I missed last few minutes due to other talk.

Samba4

Jeremy Allison talked about history and splitting of of Samba4 from Samba3 codebase. Apparently there was almost a fork due to “old” network filesystem guys and new “let’s create an AD” guys. Code still has samba3 & samba4 source directories but they are merging/cleaning them up. They have a lot of code bundled/developed in Samba4 to ease integration and configuration. KerberosLDAP..the list goes on. Makes me wonder if all of this was *really* necessary. I guess that’s why Jeremy added this bit to one of the slides:

We stopped checking for monsters under our beds when we realized they were inside us.

systemd, two years later

Surprisingly no flamewars. During the talk I (perhaps) understood why Lennart is facing so many of them though. I believe he doesn’t clearly state that “Yes, we reimplemented part of X in systemd so that it’s more reliable. BUT! You can still install that old thing, we won’t break it and you can still configure it”. Prime example for me being acpid, which Lennart replaced just to get power button working. That’s probably fine for 99% of use cases, but on a lot of notebooks acpid is primary way to handle interesting buttons that X knows nothing about (and perhaps we need them to work outside of X)

FreedomBox 1.0

FreedomBox is a project by Eben Moglen and Bdale Garbee, now mostly a software solution running on top of Debian systems to make private and secure communication without dependence on governments. And one important feature: it has to be usable by common people. There were already 2 keynotes about it in previous years. Now it’s slowly coming to 1.0. There was an interesting point about replacing CA infrastructure in webservers with p2p/gnupg trust principles and development of apache module that would be able to authenticate these.

LibreOffice: cleaning and refactoring a giant code-base

I really like where LO is going. They’ve been mostly doing huge cleanups, getting rid of old cruft, German comments etc. Java dependency is going away. They have a healthy community, ood code review/unit test processes. All the right pieces. I can’t wait for when they really start adding new things.

Has GNOME community turned crazy?

Talk about various controversial features developed by Gnome is past few years. Mostly Gnome Shell. Vincent Untz made a good point: Gnome 2.0 was nothing like Gnome 2.3x. However when he asked audience if they thought Gnome 3.0 was ready for release even people who like Gnome shell said no. There were a lot of Gnome people in the audience who were co-answering questions.

The Keeper of Secrets

Our Leslie Hawthorn had a closing keynote, which in my mind was a continuation  of her previous talk. It dealt with handling community participants when they have problems they would like to keep secret while still allowing the community to handle their absence. Most valuable part for me was probably resources & references to other literature. Because let’s face it…you can’t really give a silver bullet generic answer, because each situation is different. I guess it could have been articulated more clearly, because some people in the audience never realized this in my opinion.

Now I can start looking forward to DevConf already.

Auto-tag Fedora package git repo from koji builds

January 26, 2012 in automation, fedora, git, packaging, tool

I often browse through various Fedora packages and miss having git tags in package repositories corresponding with builds done in koji. Therefore I created following simple bash script that will create git tags in current git repo (if it’s a Fedora package).

#!/bin/bash

giturl=`fedpkg giturl`
if [ $? -ne 0 ];then
echo "This doesn't look like fedora package directory"
exit 1
fi

pkgname=`echo "${giturl}" |\
sed -e 's|git://pkgs.fedoraproject.org/\(.*\)?.*|\1|'`
# make sure we are up-to-date
git fetch

# go through last 3 releases (incl. rawhide)
for dist in f15 f16 f17;do
builds=`koji list-tagged "${dist}" "${pkgname}" | \
grep "${pkgname}" | awk '{print $1}'`
for build in $builds;do
# task urls sometimes have ".git" suffix
git_sha=`koji buildinfo "${build}" | grep '^Task:' | \
sed -e "s|.*${pkgname}\(\.git\)*:\(.*\))|\2|"`
version=`echo $build | sed -e "s:${pkgname}-::"`
echo BUILD: $pkgname\($version\) = $git_sha
git tag "${version}" "${git_sha}"
done
done

Enjoy!

fedora-review – Package reviews made easier

November 11, 2011 in fedora, json, packaging, projects, python, review, tool

For package to be included in default Fedora repositories, it has to go through process called package review. If you’ve done a few package reviews you know big chunks of this process are repeated ad-nausea in every review.
There have been quite a few tools aimed at automating and simplifying this process. However they all had one major flaw. They have been designed for reviewing specific class of packages, be it Perl, Python or generic C/C++ packages. Few us us decided to change this.
We used Tim Lauridsen’s FedoraReview package as a base for our work and started adding new features and tweaks. Current work has a website on fedorahosted where you’ll find all important information. Full feature list would be quite long, but I’ll list a few major things:
  • Bugzilla integration
  • Mock integration
  • JSON api for external plugins (more info further down)
  • Several automated tests
The tool runs all checks/tests on spec file and rpms and writes output into a text file. Snippet of the output looks like this:

Package Review
==============

Key:
- = N/A
x = Check
! = Problem
? = Not evaluated



==== Generic ====
...
[ ]: MUST License field in the package spec file matches the actual license.
[ ]: MUST License file installed when any subpackage combination is installed.
[!]: MUST Package consistently uses macros (instead of hard-coded directory names).
Using both %{buildroot} and $RPM_BUILD_ROOT
...
[x]: SHOULD Spec use %global instead of %define.
...

==== Java ====
[!]: MUST If package uses "-Dmaven.local.depmap" explain why it was needed in a comment
[!]: MUST If package uses "-Dmaven.test.skip=true" explain why it was needed in a comment

Issues:
[!]: MUST Package consistently uses macros (instead of hard-coded directory names).
Using both %{buildroot} and $RPM_BUILD_ROOT
[!]: MUST If package uses "-Dmaven.local.depmap" explain why it was needed in a comment
[!]: MUST If package uses "-Dmaven.test.skip=true" explain why it was needed in a comment
We display only relevant results. In other words if there are no post/postun scriptlets there is no reason to include sanity output checking in the template. This will make more and more sense as we add more checks.

JSON API

So how is it that different people will be able to write additional plugins for this review tool? We provide a relatively simple JSON api through stdin/stdout.
So to create a new check plugin you create a script or program in your language of choice. There is only one requirement:
  • Programming language has to have JSON format support

When the review tool runs your plugin it will send following message on its stdin:


{
"supported_api": 1,
"pkgname": "package name",
"version": "package version",
"release": "package release",
"srpm":"path/to/srpm",
"spec":{ path: "path/to/spec",
"text": "spec text with expanded macros"},
"rpms":[ "path/to/rpm", ...],
"rpmlint": "rpmlint output",
"build_dir": "/path/to/src/directory/after/build"
}

When your plugin is done with checks it returns following message by writing it to stdout:


{
"command": "results",
"supported_api": 1,
"checks":
[
{
"name": "CheckName",
"url": "URL to guidelines usually",
"group": "Group for this test.(Java, Perl, etc.)",
"text": "Check description that shows on review template",
"deprecates":["DeprecatedTest", ...]
"type": "MUST"|"SHOULD",
"result": "pass"|"fail"|"inconclusive",
"extra_output": "text",
},
...
]
}

If the plugin closes stdout without writing anything there, it means there were no relevant automated tests to run and no non-automated tests to include in template for manual evaluation. This is useful so we don’t include for example Perl-related test output for Java packages and vice-versa.

Roadmap

While the tool is already usable and soon to be packaged in Fedora, there are still quite a few things we want to improve:

  • Add more functions to API (currently there is just get_section)
  • Automate all automatable tests currently available
  • Get rid of redundant tests (don’t duplicate rpmlint)
  • Add more tests of course!
  • Maybe add templating support?

We have currently 3-4 active committers, checks for C/C++, generic, Java, R packages. There is already and example external plugin written in Perl. If you have any improvement ideas, bugreports or just want to tell us we suck because we should have done X…get in touch!

My job: I am a software chef

November 3, 2011 in fedora, packaging, personal, rant

How often are you asked what is your job? Most non-IT people will not be able to understand packaging, dependencies, rpms and whatnot. Hell, I even had trouble explaining what I do to my ex-schoolmates from university working in a traditional corporate environments. And they are software developers.
Was that just my problem? I don’t think so. I had an epiphany while on a vacation few months back. I am almost sure the idea was not mine and it was just my subconsciousness that stole it from someone else. So what is my revelation? As you might have guessed from the title:


I am a software chef. I create recipes and prepare them.

I work in a restaurant, that we call Linux distribution. There are many restaurants, each having their own recipes, rules and so on. Some restaurants form “chains” where they share most of their recipes. In these cases there is usually one restaurant that creates most recipes (Debian is such a restaurant in its Linux ecosystem).
Each restaurant usually has hundreds of chefs, some of them specialize in few recipes (build scripts), some are more flexible. In my case I specialize in a type of recipes dealing with coffee (i.e. Java).
Every recipe starts with customer (user) ordering some meal they have heard about. I look up ingredients (upstream projects) the food is made of and start recreating recipe for our restaurant. Quite often the food is made of more recipes (dependencies) and I have to create those first. Sometimes these recipes are already being prepared by other chefs, so I just use their work for my final meal. However our ingredients can be slightly bit different from the original. For example we have cow milk, but no goat milk that was in original recipe. So I have to find a way to fix the recipe using spices (patches).
Creating recipes is only part of my job though. I also work with our suppliers of ingredients (upstream developers). Sometimes the ingredients are bad, or I have found a way to improve the ingredient so I contact the suppliers and we work together.
Third part of my job is improving cooking process (simplifying packaging). So sometimes I move some furniture around so that other chefs don’t have so much between the fridge and other places. Or I create a new mixer (tools) that speeds up mixing of ingredients.
Final part of my job is to work in a VIP part of the restaurant (RHEL). Only some customers can go there, most meals are usually very similar to normal restaurant, but each meal is tasted (tested) before we give it to customers and if they don’t like it they can complain and we bring them improved recipe.
I find this metaphor kind of works for most things to a surprising degree. For the record:
  • Package maintainers – chefs
  • QE/QA – tasters
  • Security – bouncers
  • Release engineering – waiters (sorry guys)

Do you have an idea where this came from? Or can you think of a better metaphor for packaging? I’ll probably keep updating and expanding this post as I go so I can point people to this when then want to know what I do..

Automatic javadoc subpackage generation

November 3, 2011 in fedora, java, macro, packaging, rpm

Do you hate repeating the same thing over and over again? I know I do…
Java packaging guidelines state that we have to include javadocs with all java packages. This means we have to repeat following code in almost all packages (except pom and resource projects):

...
%package javadoc
Summary: API documentation for %{name}
Group: Documentation
Requires: jpackage-utils

%description javadoc
%{summary}.
...

%install
...
# javadoc
install -d -m 755 %{buildroot}%{_javadocdir}/%{name}
cp -pr target/site/apidocs/* %{buildroot}%{_javadocdir}/%{name}

...

%files javadoc
%doc LICENSE
%doc %{_javadocdir}/%{name}
...
The code is practically the same in all packages so why not automate this? Well there were two main reasons why this wasn’t done before:
  • Copying of files needs to be done during install phase
  • If package contains license, javadoc has to have it too

We solved both of these in a fairly reasonable way. Resulting macro help looks like this:


# %create_javadoc_subpackage can be used to completely create
# javadoc subpackage for java projects.
# !!! Needs to be used at the end of %install section
# There are these variables that change its behaviour:
#
# %__javadoc_license - set if the license is in non-standard place to
# prevent Requires on main package
# %__apidocs_dir - set custom path to directory with javadocs
# (defaults to target/site/apidocs)
# %__javadoc_skip_requires - if defined javadoc subpackage will not
# require main package under any circumstances (useful
# if upstream doesn't provide separate license file)
#

Is it understandable enough? If you need to generate javadocs, just make them build and then add %create_javadoc_subpackage macro call at the end of %install section. Normally you shouldn’t have to change anything. We search in a few standard places for licenses. More specifically we look for LICENSE* COPYING* doc/LICENSE* doc/COPYING* license*. Do you have more ideas where to look? It’s easy to add. If we don’t find license we automatically add requires on main package and assume you put license in there. If upstream doesn’t provide separate license file you can do %global __javadoc_skip_requires t and we will ignore licensing completely.

I’d like this added to our packaging guidelines so we can start using it. My testing shows it works fairly well. I’d love to improve it so you could place it anywhere in the spec, not just %install section, but rpm macros are…complicated.
*Note*: For gory details head over to our git repository. For now it’s in separate feature branch.

Making packaging Maven projects easier

September 12, 2011 in en, fedora, packaging

There are two recent changes to our Java guidelines in Fedora and use of Maven when packaging that I’d like to mention today

Maven dependency mapping macros

Thing I haven’t blogged about yet but it’s pretty important: We have new macros for maven depmaps in Fedora. In the past when you wanted to map certain groupId:artifactId to a file in _javadir, you had to include snippet like this in your spec:


%add_to_maven_depmap com.google.guava guava 05 JPP guava
%add_to_maven_depmap com.google.collections google-collections 05 JPP guava

This tells our maven that com:google:guava:guava and com.google.collections:google-collections can be found in one of the repositories as JPP/guava.jar. It meant you had to know the groupId:artifactId and other information, plus it was extremely easy to make a mistake here causing all sorts of trouble. Current code doing the same thing:


%add_maven_depmap JPP-guava.pom guava.jar -a "com.google.collections:google-collections"

We parse the pom file and get groupId:artifactId from it, plus we do additional sanity checks such as:

  • naming of pom and jar file have to be consistent
  • jar file has to exist if packaging type is not pom

If you need additional mappings you can easily add them. There are few other options for this new macro useful in certain situations.

Maven test deps skipping

Long story short: When you use -Dmaven.test.skip=true in Fedora packages you no longer need to patch those test dependencies out of pom.xml.

We’ve had Apache Maven in Fedora for quite some time and packaging using Maven has been getting easier over time due to small tweaks to our packaging macros and guidelines changes. However there has been one problem that’s been bugging all Java packagers and was especially confusing for those starting to package software built with Maven. The problem is that Maven creates a tree of dependencies before it starts building the project, but it includes test dependencies even when tests are being skipped.

Skipping tests is sometimes necessary due to problems with koji, or dependencies and up until now we had to either patch those tests dependencies from pom.xml or use custom dependency mappings (ugly concept in itself).

Last week I decided it’s about time someone did something about this, so I dug in the Maven code and created a solution (more of a hack really) that is already included in Fedora. If you want the gory details, you can read the patch itself (I advise against it). I’ll try to make the patch work properly so that it can be included in mainstream code.

I can just hope that packagers will find these changes helpful, but general feedback has been positive.

Addition of fedpkg rpmlint

July 27, 2011 in fedora, packaging, rpmlint

*Edit:* Yes, there is fedpkg lint but it somewhat limited so read on. Instead of addition of “rpmlint” command Pingou will improve current lint

Recently I was trying to help OpenSuSE guys with some updates to their Java stack and I was sent link to their build system. I noticed a file called jpackage-utils-rpmlintrc and this got me thinking…

What if we added rpmlint command to fedpkg with per-package rpmlint ignore settings? Turns out Pingou took my idea and implemented it in under an hour :-)

An example run:


$ fedpkg rpmlint
plexus-interpolation.spec: W: invalid-url Source0: plexus-interpolation-1.14.tar.xz
0 packages and 1 specfiles checked; 0 errors, 1 warnings.

plexus-interpolation.spec: W: invalid-url Source0: plexus-interpolation-1.14.tar.xz
plexus-interpolation.src: W: spelling-error %description -l en_US interpolator -> interpolate, interpolation, interrogator
plexus-interpolation.src: W: invalid-url Source0: plexus-interpolation-1.14.tar.xz
1 packages and 1 specfiles checked; 0 errors, 3 warnings.
2 packages run
rpmlint has not been run on rpm files but should

OK, so we can run rpmlint on spec, srpm and binary rpms with single command. But I don’t like to see the same warnings all the time, because that means I will probably miss real problems when they appear. For this fedpkg rpmlint uses .rpmlint file as additional rpmlint config. So after creating:


$ cat > .rpmlint << EOF
# we have scm checkout with comment in spec
addFilter('invalid-url')
# false positive
addFilter('spelling-error.*interpolator')
EOF
$ fedpkg rpmlint
0 packages and 1 specfiles checked; 0 errors, 0 warnings.

1 packages and 1 specfiles checked; 0 errors, 0 warnings.
2 packages run
rpmlint has not been run on rpm files but should

Cool right? Pierre sent patch with this feature to fedpkg developers, so hopefully we’ll see this addition soon. I then plan to add custom .rpmlint configurations to all my packages so that they will be warning-free.

Print expanded SourceX: urls from spec files

July 26, 2011 in fedora, packaging, python, rpm, script

I’ve noticed quite a few times that people add comments to their Source0: urls without macros to seemingly simplify manual downloading. It looks like this:

Name: jsoup
Version: 1.6.1
...
# http://jsoup.org/packages/jsoup-1.6.1-sources.jar
Source0: http://%{name}.org/packages/%{name}-%{version}-sources.jar

This creates burden on maintainers to keep those urls up-to-date as version changes, so I created simple python script for printing out Source urls from spec files:


#!/usr/bin/python

import rpm
import sys

ts=rpm.TransactionSet()
spec_obj = ts.parseSpec(sys.argv[1])

sources = spec_obj.sources

for url, num, flags in sources:
print url

Chmod this +x, put into your PATH and enjoy by giving it path to spec file.
*Edit*: Probably much nicer way to do the same thing already present on your system (courtesy of Alexander Kurtakov):

spectool X.spec

I knew there was something like this, but forgot what it was. Oh well…2 minutes lost.

Getting your Java Application in Linux: Guide for Developers (Part 2)

April 20, 2011 in fedora, howto, java, packaging

Ant and Maven

Last time I have written about general rules of engagement for Java developers if they want to make lives of packagers easier. Today I’ll focus on specifics of two main build systems in use today: Ant and Maven, but more so on Maven for reasons I’ll state in a while.

Ant

Ant is (or at least used to be) most widely deployed build system in Java ecosystem. There are probably multiple reasons for it, but generally it’s because Ant is relatively simple. In *NIX world Ant is equivalent of pure make (and build.xml of Makefile). build.xml is just that: an XML, and it has additional extensions to simplify common tasks (calling javac, javadoc, etc.). So the question is:

I am starting a new java project. How can I use Ant properly to make life easier for you?

The most simple answer? DON’T! It might seem harsh and ignorant of bigger picture and it probably is. But I believe it’s also true that Ant is generally harder to package than Maven. Ant build.xml files are almost always unique pieces of art in themselves and as such can be a pain to package. I am always reminded of following quote when I have to dig through some smart build.xml system:

Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.

  –Brian Kernighan

And I have a feeling some people try to be really clever when writing their build.xml files. That said, I understand there are times when using Ant is just too tempting so I’ll include a few tips for it anyway.

Use apache-ivy extension for dependencies

One of main problem with and is handling of various dependencies. Usually, they are in some subdirectory of main tree, some jars versioned, some not, some patched without any note about it…in other words nightmare in itself. Apache-ivy extension helps here because it works with dependency metadata that packagers can use to figure out real build dependencies including versions. We can also be sure that no dependencies are patched in one way or the other.

Ivy is nice for developers as well. It will make your source tarballs much smaller (You do have source tarballs right?!) and your build.xml nicer. I won’t include any examples here because I believe that Ivy documentation is indeed very good.

One lib/ to rule them all

In case you really don’t want to use Ivy, make sure you place all your dependencies in one directory in top level of your project (don’t scatter your dependencies, even if you are using multiple sub-projects). This directory should ideally be called lib/. It should contain your dependencies named as ${name}-${version}.jar. Most of the time you should include license files for every dependency you bundle, because you are becoming distributors and for most licenses this means you have to provide full text of the license. For licenses use identical name as jar filenames, but use “.license” suffix. All in all, make it easy to figure out your build dependencies and play with them.

Don’t be too clever

I can’t stress this enough. Try to keep your build.xml files to the bare minimum. Understanding ten 30 KiB big build.xml files with multiple-phase build and tests spread through 10 directories is no fun. Please think of poor packager when you write your build.xml files. I don’t mind having grey hair that much, but I’d rather if it came later rather than sooner.

Maven

And now we are coming to my favourite part. Maven is a build and project management tool that has extensive plugin support able to do almost anything developer might ask for. And all that while providing formal project structure, so that once you learn how Maven works in one project you can re-use your knowledge in other projects.

Maven goodies

Maven provides several good things for packagers such as providing clear dependencies and preventing simple patched dependencies from sneaking in. Most important advantage for packagers coming with Maven is the fact that problems are the same in all projects. Once you understand how certain Maven plugin works, you will know what to expect and what to look for. But Maven is nice not just for packagers, but also for developers.

Declarative instead of descriptive

You don’t tell Maven:

Add jar A, jar B to the classpath, then use this properies file to set-up test resources. Then compile tests (Have you compiled sources yet?) and then … and run them with X

Instead you place test files and resources into appropriate directories and Maven will take care of everything. You just need to specify your test dependencies in nice and tidy pom.xml.

Project metadata in one place

With Maven you have all project information in one place:

  • Developer contact information
  • Homepage
  • SCM URLs
  • Mailinglists
  • Issue tracker URL
  • Project reports/site generation
  • Dependencies
  • Ability modify behaviour according to architecture, OS or other property

Need I say more? Fill it out, keep it up-to-date and we will all be happy.

Great integration with other tools

Ecosystem around Maven has been growing in past years and now you will find good support for handling your pom.xml files in any major java IDE. But that is just the tip of the iceberg. There are Maven plugins adding all kinds of additional tool support. Running checkstyle on your code, helping with licensing, integration with gpg, ssh, jflex and making releases. There are plugins for that and more.

Support for Ant

If you are in process of migrating your build system from Ant to Maven, you can do it in phases. For parts of your builds you can easily run Ant with maven-ant-plugin. Good example of such migration in progress is checkstyle. In version 5.2 they introduced Maven build system while preserving their old layout and running Ant for tests.

Maven messier side

A.K.A What you need to be aware of. It’s generally quite hard to do something bad in Maven, because it won’t let you do that easily. That said, there are plugins that can make it hard for us to package your software.

maven-dependency-plugin:copy-dependencies

This specific goal can potentially cause problems because it allows to copy classes from dependencies into resulting jar files. As I wrote last time, this is unacceptable because it creates possible licensing, security and maintenance nightmares. If you need even just one class from another project, rather than copying it, add it as a dependency into pom.xml

maven-shade-plugin

Shade plugin is a very shady plugin (pun intended). It can be used to weave depdencies inside your jars while changing their package names and doing all kinds of modifications in the process. I’ll give you a small test now :-) Let’s say you have jar file with following contents:


META-INF/
META-INF/MANIFEST.MF
META-INF/maven/
META-INF/maven/org.packager/
META-INF/maven/org.packager/Pack/
META-INF/maven/org.packager/Pack/pom.properties
META-INF/maven/org.packager/Pack/pom.xml
org/
org/packager/
org/packager/signature/
org/packager/signature/SignatureReader.class
org/packager/signature/SignatureVisitor.class
org/packager/signature/SignatureWriter.class
org/packager/Pack.class

Can you tell, from looking at jar contents where is org.packager.signature subpackage coming from? Take your time, think about it. Nothing? Well here’s a hint:



org.apache.maven.plugins
maven-shade-plugin



org.objectweb.asm
org.packager




I believe this demonstrates why usage of shade plugin is evil (in 99% of cases at least). This is especially problematic if the shaded packages are part of public API of your project, because we won’t be able to simply fix this in one package, but it will cascade up the dependency chain.

maven-bundle-plugin

Bundle is one of the more controversial plugins, because it can be used both for good and bad :-) One of the most important good use cases for bundle plugin is generating OSGI bundles. Every project can easily make their jar files OSGI compatible by doing something like this:


...
bundle
...



org.apache.felix
maven-bundle-plugin
true



...

Easy right? Now to the darker side of bundle plugin. I have another example to test your skills. This one should be easier than shade plugin:


META-INF/MANIFEST.MF
META-INF/
META-INF/maven/
META-INF/maven/org.packager/
META-INF/maven/org.packager/Pack/
META-INF/maven/org.packager/Pack/pom.properties
META-INF/maven/org.packager/Pack/pom.xml
org/
org/objectweb/
org/objectweb/asm/
org/objectweb/asm/signature/
org/objectweb/asm/signature/SignatureReader.class
org/objectweb/asm/signature/SignatureVisitor.class
org/objectweb/asm/signature/SignatureWriter.class
org/packager/
org/packager/Pack.class

Problem is the same as with shade plugin (bundling of dependencies), but at least here it’s more visible in the contents of the jar and it will not poison API of the jar. Just for the record, this is how it was created:



org.apache.felix
maven-bundle-plugin
true


org.objectweb.asm.signature



Summary

Today I wrote about:

  • Ant and why you shouldn’t use it (that much)
  • Ant and how to use it if you have to
  • Maven and why it rocks for packagers and developers
  • Maven and its plugins and why they suck for packagers sometimes

There are a lot more things that can cause problems, but these are the most obvious and easily fixed. I’ll try to gather more information about things we (packagers) can do to help you (developers) a bit more and perhaps include one final part for this guide.

Getting your Java Application in Linux: Guide for Developers (Part 1)

April 8, 2011 in fedora, howto, java, packaging

Introduction to packaging Java

Packaging Java libraries and applications in Fedora has been my daily bread for almost a year now. I realized now is the time to share some of my thoughts on the matter and perhaps share a few ideas that upstream developers might find useful when dealing with Linux distributions.

This endeavour is going to be split into several posts, because there are more sub-topics I want to write about. Most of this is going to be based on my talk I did @ FOSDEM 2011. Originally I was hoping to just post the video, but it seems to be taking more time than I expected :-)

If you are not entirely familiar with status of Java on Linux systems it would be a good idea to first read a great article by Thierry Carrez called The real problem with Java in Linux distros. A short quote from that blog:

The problem is that Java open source upstream projects do not really release code. Their main artifact is a complete binary distribution, a bundle including their compiled code and a set of third-party libraries they rely on.

There is no simple solution and my suggestions are only mid-term workarounds and ways to make each other’s (upstream ↔ downstream) lives easier. Sometimes I am quite terse in suggestions, but if need be I’ll expand them later on.

Part 1: General rules of engagement

Today I am going to focus on general rules that apply to all Java projects wishing to be packaged in Linux distributions:

  • Making source releases
  • Handling Dependencies
  • Bugfix releases

For full understanding a short summary of general requirements for packages to be added to most Linux distributions:

  • All packages have to be built from source
  • No bundled dependencies used for building/running
  • Have single version of each library that all packages use

There are a lot of reasons for these rules and they have been flogged to death multiple times in various places. It mostly boils down to severe maintenance and security problems when these rules are not followed.

Making source releases

As I mentioned previously most Linux distributions rebuild packages from source even when there is an upstream release that is binary compatible. To do this we need sources obviously :-) Unfortunately quite a few (mostly Maven) projects don’t do source release tarballs. Some projects provide source releases without build scripts (build.xml or pom.xml files). Most notable examples are Apache Maven plugins. For each and every update of one of these plugins we have to checkout the source from upstream repository and generate the tarball ourselves.
All projects using Maven build system can simply make packagers’ lives easier by having following snippet in their pom.xml files:




...

maven-assembly-plugin


project




make-assembly
package

single




...


This will create -project.zip/tar.gz files containing all the files needed to rebuild package from source. I have no real advice for projects using Ant for now, but I’ll summarise them next time.

Handling dependencies

I have a feeling that most Java projects don’t spend too much time thinking about dependencies. This should change so here are a few things to think about when adding new dependencies to your project.

Verify if the dependency isn’t provided by JVM

Often packages contain unnecessary dependencies that are provided by all recent JVMs. Think twice if you really need another XML parser.

Try to pick dependencies from major projects

Major projects (apache-commons libraries, eclipse, etc.) are much more likely to be packaged and supported properly in Linux distributions. If you use some unknown small library packagers will have to package that first and this can sometimes lead to such frustrating dependency chains they will give up before packaging your software.

Do NOT patch your dependencies

Sometimes a project A does almost exactly what you want, but not quite…So you patch it and ship it with your project B as a dependency. This will cause problems for Linux distributions because you basically forked the original project A. What you should do instead is work with the developers of project A to add features you need or fix those pesky bugs.

Bugfix releases

Every software project has bugs, so sooner or later you will have to do a bugfix release. As always there are certain rules you should try to uphold when doing bugfix releases.

Use correct version numbers

This depends on your versioning scheme. I’ll assume you are using standard X.Y.Z versions for your releases. Changes in Z are smallest released changes of your project. They should mostly contain only bugfixes and unobtrusive and simple feature additions if necessary. If you want to add bigger features you should change Y part of the version.

Backward compatible

Bugfix releases have to be backwards compatible at all times. No API changes are allowed.

No changes in dependencies

You should not change dependencies or add new ones in bugfix releases. Even updating dependency to a new version can cause massive recursive need for updates or new dependencies. The only time it’s acceptable to change/add dependency version in bugfix release is when new dependency is required to fix the bug.

An excellent example of how NOT to do things was Apache Maven update from 3.0 to 3.0.1. This update changed requirements from Aether 1.7 to Aether 1.8. Aether 1.8 had new dependency on async-http-client. Async-http-client depends on netty, jetty 7.x and more libraries. So what should have been simple bugfix update turned into need for major update of 1 package and 2 new package additions. If this update contained security fixes it would cause serious problems to resolve in timely manner.

Summary

  • Create source releases containing build scripts
  • Think about your dependencies carefully
  • Handle micro releases gracefully

Next time I’ll look into some Ant and Maven specifics that are causing problems for packagers and how to resolve them in your projects.