Ins and Outs of Inputs

Auto-tag Fedora package git repo from koji builds

noreply@blogger.com (Stanislav Ochotnický) — Thu, 26 Jan 2012 16:58:00 +0000

I often browse through various Fedora packages and miss having git tags in package repositories corresponding with builds done in koji. Therefore I created following simple bash script that will create git tags in current git repo (if it's a Fedora package).

#!/bin/bash

giturl=`fedpkg giturl`
if [ $? -ne 0 ];then
    echo "This doesn't look like fedora package directory"
    exit 1
fi

pkgname=`echo "${giturl}" |\
         sed -e 's|git://pkgs.fedoraproject.org/\(.*\)?.*|\1|'`
# make sure we are up-to-date
git fetch

# go through last 3 releases (incl. rawhide)
for dist in f15 f16 f17;do
    builds=`koji list-tagged "${dist}" "${pkgname}" | \
        grep "${pkgname}" | awk '{print $1}'`
    for build in $builds;do
        # task urls sometimes have ".git" suffix
        git_sha=`koji buildinfo "${build}" | grep '^Task:' | \
            sed -e "s|.*${pkgname}\(\.git\)*:\(.*\))|\2|"`
        version=`echo $build | sed -e "s:${pkgname}-::"`
        echo BUILD: $pkgname\($version\) = $git_sha
        git tag "${version}" "${git_sha}"
    done
done

Enjoy!

fedora-review - Package reviews made easier

noreply@blogger.com (Stanislav Ochotnický) — Fri, 11 Nov 2011 15:15:00 +0000

For package to be included in default Fedora repositories, it has to go through process called package review. If you've done a few package reviews you know big chunks of this process are repeated ad-nausea in every review.

There have been quite a few tools aimed at automating and simplifying this process. However they all had one major flaw. They have been designed for reviewing specific class of packages, be it Perl, Python or generic C/C++ packages. Few us us decided to change this.

We used Tim Lauridsen's FedoraReview package as a base for our work and started adding new features and tweaks. Current work has a website on fedorahosted where you'll find all important information. Full feature list would be quite long, but I'll list a few major things:

Bugzilla integration
Mock integration
JSON api for external plugins (more info further down)
Several automated tests

The tool runs all checks/tests on spec file and rpms and writes output into a text file. Snippet of the output looks like this:

Package Review
==============

Key:
- = N/A
x = Check
! = Problem
? = Not evaluated



==== Generic ====
...
[ ]: MUST License field in the package spec file matches the actual license.
[ ]: MUST License file installed when any subpackage combination is installed.
[!]: MUST Package consistently uses macros (instead of hard-coded directory names).
        Using both %{buildroot} and $RPM_BUILD_ROOT
...
[x]: SHOULD Spec use %global instead of %define.
...

==== Java ====
[!]: MUST If package uses "-Dmaven.local.depmap" explain why it was needed in a comment
[!]: MUST If package uses "-Dmaven.test.skip=true" explain why it was needed in a comment

Issues:
[!]: MUST Package consistently uses macros (instead of hard-coded directory names).
        Using both %{buildroot} and $RPM_BUILD_ROOT
[!]: MUST If package uses "-Dmaven.local.depmap" explain why it was needed in a comment
[!]: MUST If package uses "-Dmaven.test.skip=true" explain why it was needed in a comment

We display only relevant results. In other words if there are no post/postun scriptlets there is no reason to include sanity output checking in the template. This will make more and more sense as we add more checks.

JSON API

So how is it that different people will be able to write additional plugins for this review tool? We provide a relatively simple JSON api through stdin/stdout.

So to create a new check plugin you create a script or program in your language of choice. There is only one requirement:

Programming language has to have JSON format support

When the review tool runs your plugin it will send following message on its stdin:

{
    "supported_api": 1,
    "pkgname": "package name",
    "version": "package version",
    "release": "package release",
    "srpm":"path/to/srpm",
    "spec":{ path: "path/to/spec",
            "text": "spec text with expanded macros"},
    "rpms":[ "path/to/rpm", ...],
    "rpmlint": "rpmlint output",
    "build_dir": "/path/to/src/directory/after/build"
}

When your plugin is done with checks it returns following message by writing it to stdout:

{
    "command": "results",
    "supported_api": 1,
    "checks":
    [
        {
         "name": "CheckName",
         "url": "URL to guidelines usually",
         "group": "Group for this test.(Java, Perl, etc.)",
         "text": "Check description that shows on review template",
         "deprecates":["DeprecatedTest", ...]
         "type": "MUST"|"SHOULD",
         "result": "pass"|"fail"|"inconclusive",
         "extra_output": "text",
        },
        ...
    ]
}

If the plugin closes stdout without writing anything there, it means there were no relevant automated tests to run and no non-automated tests to include in template for manual evaluation. This is useful so we don't include for example Perl-related test output for Java packages and vice-versa.

Roadmap

While the tool is already usable and soon to be packaged in Fedora, there are still quite a few things we want to improve:

Add more functions to API (currently there is just get_section)
Automate all automatable tests currently available
Get rid of redundant tests (don't duplicate rpmlint)
Add more tests of course!
Maybe add templating support?

We have currently 3-4 active committers, checks for C/C++, generic, Java, R packages. There is already and example external plugin written in Perl. If you have any improvement ideas, bugreports or just want to tell us we suck because we should have done X...get in touch!

My job: I am a software chef

noreply@blogger.com (Stanislav Ochotnický) — Thu, 03 Nov 2011 18:14:00 +0000

How often are you asked what is your job? Most non-IT people will not be able to understand packaging, dependencies, rpms and whatnot. Hell, I even had trouble explaining what I do to my ex-schoolmates from university working in a traditional corporate environments. And they are software developers.

Was that just my problem? I don't think so. I had an epiphany while on a vacation few months back. I am almost sure the idea was not mine and it was just my subconsciousness that stole it from someone else. So what is my revelation? As you might have guessed from the title:

I am a software chef. I create recipes and prepare them.

I work in a restaurant, that we call Linux distribution. There are many restaurants, each having their own recipes, rules and so on. Some restaurants form "chains" where they share most of their recipes. In these cases there is usually one restaurant that creates most recipes (Debian is such a restaurant in its Linux ecosystem).

Each restaurant usually has hundreds of chefs, some of them specialize in few recipes (build scripts), some are more flexible. In my case I specialize in a type of recipes dealing with coffee (i.e. Java).

Every recipe starts with customer (user) ordering some meal they have heard about. I look up ingredients (upstream projects) the food is made of and start recreating recipe for our restaurant. Quite often the food is made of more recipes (dependencies) and I have to create those first. Sometimes these recipes are already being prepared by other chefs, so I just use their work for my final meal. However our ingredients can be slightly bit different from the original. For example we have cow milk, but no goat milk that was in original recipe. So I have to find a way to fix the recipe using spices (patches).

Creating recipes is only part of my job though. I also work with our suppliers of ingredients (upstream developers). Sometimes the ingredients are bad, or I have found a way to improve the ingredient so I contact the suppliers and we work together.

Third part of my job is improving cooking process (simplifying packaging). So sometimes I move some furniture around so that other chefs don't have so much between the fridge and other places. Or I create a new mixer (tools) that speeds up mixing of ingredients.

Final part of my job is to work in a VIP part of the restaurant (RHEL). Only some customers can go there, most meals are usually very similar to normal restaurant, but each meal is tasted (tested) before we give it to customers and if they don't like it they can complain and we bring them improved recipe.

I find this metaphor kind of works for most things to a surprising degree. For the record:

Package maintainers - chefs
QE/QA - tasters
Security - bouncers
Release engineering - waiters (sorry guys)

Do you have an idea where this came from? Or can you think of a better metaphor for packaging? I'll probably keep updating and expanding this post as I go so I can point people to this when then want to know what I do..

Automatic javadoc subpackage generation

noreply@blogger.com (Stanislav Ochotnický) — Thu, 03 Nov 2011 15:53:00 +0000

Do you hate repeating the same thing over and over again? I know I do...

Java packaging guidelines state that we have to include javadocs with all java packages. This means we have to repeat following code in almost all packages (except pom and resource projects):

...
%package        javadoc
Summary:        API documentation for %{name}
Group:          Documentation
Requires:       jpackage-utils

%description    javadoc
%{summary}.
...

%install
...
# javadoc
install -d -m 755 %{buildroot}%{_javadocdir}/%{name}
cp -pr target/site/apidocs/* %{buildroot}%{_javadocdir}/%{name}

...

%files javadoc
%doc LICENSE
%doc %{_javadocdir}/%{name}
...

The code is practically the same in all packages so why not automate this? Well there were two main reasons why this wasn't done before:

Copying of files needs to be done during install phase
If package contains license, javadoc has to have it too

We solved both of these in a fairly reasonable way. Resulting macro help looks like this:

# %create_javadoc_subpackage can be used to completely create
# javadoc subpackage for java projects.
# !!! Needs to be used at the end of %install section
# There are these variables that change its behaviour:
#
# %__javadoc_license - set if the license is in non-standard place to
#                prevent Requires on main package
# %__apidocs_dir - set custom path to directory with javadocs
#                (defaults to target/site/apidocs)
# %__javadoc_skip_requires - if defined javadoc subpackage will not
#                require main package under any circumstances (useful
#                if upstream doesn't provide separate license file)
#

Is it understandable enough? If you need to generate javadocs, just make them build and then add %create_javadoc_subpackage macro call at the end of %install section. Normally you shouldn't have to change anything. We search in a few standard places for licenses. More specifically we look for LICENSE* COPYING* doc/LICENSE* doc/COPYING* license*. Do you have more ideas where to look? It's easy to add. If we don't find license we automatically add requires on main package and assume you put license in there. If upstream doesn't provide separate license file you can do %global __javadoc_skip_requires t and we will ignore licensing completely.

I'd like this added to our packaging guidelines so we can start using it. My testing shows it works fairly well. I'd love to improve it so you could place it anywhere in the spec, not just %install section, but rpm macros are...complicated.

*Note*: For gory details head over to our git repository. For now it's in separate feature branch.

Understood and agreed with

noreply@blogger.com (Stanislav Ochotnický) — Sun, 16 Oct 2011 22:25:00 +0000

Dear OSS/Fedora/whatever reader. Stop right here.

Oh nothing's going to change my love for you
I wanna spend my life with you
So we make love on the grass under the moon
No one call tell, damned if I do
Forever journey on golden avenues
I drift in your eyes since I love you
I got that beat in my veins for only rule
Love is to share, mine is for you

Making packaging Maven projects easier

noreply@blogger.com (Stanislav Ochotnický) — Mon, 12 Sep 2011 16:47:00 +0000

There are two recent changes to our Java guidelines in Fedora and use of Maven when packaging that I'd like to mention today

Maven dependency mapping macros

Thing I haven't blogged about yet but it's pretty important: We have new macros for maven depmaps in Fedora. In the past when you wanted to map certain groupId:artifactId to a file in _javadir, you had to include snippet like this in your spec:

%add_to_maven_depmap com.google.guava guava 05 JPP guava
%add_to_maven_depmap com.google.collections google-collections 05 JPP guava

This tells our maven that com:google:guava:guava and com.google.collections:google-collections can be found in one of the repositories as JPP/guava.jar. It meant you had to know the groupId:artifactId and other information, plus it was extremely easy to make a mistake here causing all sorts of trouble. Current code doing the same thing:

%add_maven_depmap JPP-guava.pom guava.jar -a "com.google.collections:google-collections"

We parse the pom file and get groupId:artifactId from it, plus we do additional sanity checks such as:

naming of pom and jar file have to be consistent
jar file has to exist if packaging type is not pom

If you need additional mappings you can easily add them. There are few other options for this new macro useful in certain situations.

Maven test deps skipping

Long story short: When you use -Dmaven.test.skip=true in Fedora packages you no longer need to patch those test dependencies out of pom.xml.

We've had Apache Maven in Fedora for quite some time and packaging using Maven has been getting easier over time due to small tweaks to our packaging macros and guidelines changes. However there has been one problem that's been bugging all Java packagers and was especially confusing for those starting to package software built with Maven. The problem is that Maven creates a tree of dependencies before it starts building the project, but it includes test dependencies even when tests are being skipped.

Skipping tests is sometimes necessary due to problems with koji, or dependencies and up until now we had to either patch those tests dependencies from pom.xml or use custom dependency mappings (ugly concept in itself).

Last week I decided it's about time someone did something about this, so I dug in the Maven code and created a solution (more of a hack really) that is already included in Fedora. If you want the gory details, you can read the patch itself (I advise against it). I'll try to make the patch work properly so that it can be included in mainstream code.

I can just hope that packagers will find these changes helpful, but general feedback has been positive.

Fixing fedpkg clog output to be git-friendly

noreply@blogger.com (Stanislav Ochotnický) — Fri, 26 Aug 2011 10:09:00 +0000

I have been doing proxy-maintainer now for a few people and I found strange problem with fedpkg clog in relation to git format-patch and git am.

If you have a changelog message like this:

* Mon Feb 28 2011 Stanislav Ochotnicky  - 2.1.1-1
- Update to 2.1.1
- Update patch
- Disable guice-eclipse for now

fedpkg commit -c would create git commit message like this:

commit 22b5306036b6ef1022498b63e40324370ff7159b (HEAD, f15)
Author:     Stanislav Ochotnicky 
AuthorDate: Fri Aug 26 11:45:54 2011 +0200

    Update to 2.1.1
    Update patch
    Disable guice-eclipse for now

This works fine and mighty as long as you don't try to produce patch from this commit. Let's see what happens with git format-patch HEAD~1.

$ head 0001-Update-to-2.1.1.patch                                                                                           f15 [22b5306]
From 22b5306036b6ef1022498b63e40324370ff7159b Mon Sep 17 00:00:00 2001
From: Stanislav Ochotnicky 
Date: Fri, 26 Aug 2011 11:45:54 +0200
Subject: [PATCH] Update to 2.1.1 Update patch Disable guice-eclipse for now

After adding this patch to repository using git am the line breaks would disappear. This is because git expects empty line after subject and description of the commit afterwards.

I decided to try and fix fedpkg clog a bit. Given the previous changelog, now it creates git message like this:

commit 768964ce2145ef2b472fc5ef8781fb036d586b0e (HEAD, f15)
Author:     Stanislav Ochotnicky 
AuthorDate: Fri Aug 26 11:57:20 2011 +0200

    Update to 2.1.1

    - Update patch
    - Disable guice-eclipse for now

This means that git format-patch can do the right thing. I filed bug report for fedora-package so hopefully we can have this fixed sometime.

Addition of fedpkg rpmlint

noreply@blogger.com (Stanislav Ochotnický) — Wed, 27 Jul 2011 17:05:00 +0000

*Edit:* Yes, there is fedpkg lint but it somewhat limited so read on. Instead of addition of "rpmlint" command Pingou will improve current lint

Recently I was trying to help OpenSuSE guys with some updates to their Java stack and I was sent link to their build system. I noticed a file called jpackage-utils-rpmlintrc and this got me thinking...

What if we added rpmlint command to fedpkg with per-package rpmlint ignore settings? Turns out Pingou took my idea and implemented it in under an hour :-)

An example run:

$ fedpkg rpmlint
plexus-interpolation.spec: W: invalid-url Source0: plexus-interpolation-1.14.tar.xz
0 packages and 1 specfiles checked; 0 errors, 1 warnings.

plexus-interpolation.spec: W: invalid-url Source0: plexus-interpolation-1.14.tar.xz
plexus-interpolation.src: W: spelling-error %description -l en_US interpolator -> interpolate, interpolation, interrogator
plexus-interpolation.src: W: invalid-url Source0: plexus-interpolation-1.14.tar.xz
1 packages and 1 specfiles checked; 0 errors, 3 warnings.
2 packages run
rpmlint has not been run on rpm files but should

OK, so we can run rpmlint on spec, srpm and binary rpms with single command. But I don't like to see the same warnings all the time, because that means I will probably miss real problems when they appear. For this fedpkg rpmlint uses .rpmlint file as additional rpmlint config. So after creating:

$ cat > .rpmlint << EOF
# we have scm checkout with comment in spec
addFilter('invalid-url')
# false positive
addFilter('spelling-error.*interpolator')
EOF
$ fedpkg rpmlint
0 packages and 1 specfiles checked; 0 errors, 0 warnings.

1 packages and 1 specfiles checked; 0 errors, 0 warnings.
2 packages run
rpmlint has not been run on rpm files but should

Cool right? Pierre sent patch with this feature to fedpkg developers, so hopefully we'll see this addition soon. I then plan to add custom .rpmlint configurations to all my packages so that they will be warning-free.

Print expanded SourceX: urls from spec files

noreply@blogger.com (Stanislav Ochotnický) — Tue, 26 Jul 2011 11:41:00 +0000

I've noticed quite a few times that people add comments to their Source0: urls without macros to seemingly simplify manual downloading. It looks like this:

Name:      jsoup
Version:   1.6.1
...
# http://jsoup.org/packages/jsoup-1.6.1-sources.jar
Source0:        http://%{name}.org/packages/%{name}-%{version}-sources.jar

This creates burden on maintainers to keep those urls up-to-date as version changes, so I created simple python script for printing out Source urls from spec files:

#!/usr/bin/python

import rpm
import sys

ts=rpm.TransactionSet()
spec_obj = ts.parseSpec(sys.argv[1])

sources = spec_obj.sources

for url, num, flags in sources:
    print url

Chmod this +x, put into your PATH and enjoy by giving it path to spec file.
*Edit*: Probably much nicer way to do the same thing already present on your system (courtesy of Alexander Kurtakov):

spectool X.spec

I knew there was something like this, but forgot what it was. Oh well...2 minutes lost.

FOSDEM 2011: Java Packaging for Developers - Video

noreply@blogger.com (Stanislav Ochotnický) — Tue, 17 May 2011 13:15:00 +0000

I've mentioned before that I attended FOSDEM this year. It was more than 3 months ago and I finally got my hands on video from my presentation. Courtesy of Andrew John Hughes, licensed under CC-BY-ND. As a refresher, slides are available here. your browser does not support the video tag. You can still download video from my fedorapeople place.
You can play or download rest of the videos by going to Andrew's page.

Getting your Java Application in Linux: Guide for Developers (Part 2)

noreply@blogger.com (Stanislav Ochotnický) — Wed, 20 Apr 2011 13:02:00 +0000

Ant and Maven

Last time I have written about general rules of engagement for Java developers if they want to make lives of packagers easier. Today I'll focus on specifics of two main build systems in use today: Ant and Maven, but more so on Maven for reasons I'll state in a while.

Ant

Ant is (or at least used to be) most widely deployed build system in Java ecosystem. There are probably multiple reasons for it, but generally it's because Ant is relatively simple. In *NIX world Ant is equivalent of pure make (and build.xml of Makefile). build.xml is just that: an XML, and it has additional extensions to simplify common tasks (calling javac, javadoc, etc.). So the question is:

I am starting a new java project. How can I use Ant properly to make life easier for you?

The most simple answer? DON'T! It might seem harsh and ignorant of bigger picture and it probably is. But I believe it's also true that Ant is generally harder to package than Maven. Ant build.xml files are almost always unique pieces of art in themselves and as such can be a pain to package. I am always reminded of following quote when I have to dig through some smart build.xml system:

Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.

--Brian Kernighan

And I have a feeling some people try to be really clever when writing their build.xml files. That said, I understand there are times when using Ant is just too tempting so I'll include a few tips for it anyway.

Use apache-ivy extension for dependencies

One of main problem with and is handling of various dependencies. Usually, they are in some subdirectory of main tree, some jars versioned, some not, some patched without any note about it...in other words nightmare in itself. Apache-ivy extension helps here because it works with dependency metadata that packagers can use to figure out real build dependencies including versions. We can also be sure that no dependencies are patched in one way or the other.

Ivy is nice for developers as well. It will make your source tarballs much smaller (You do have source tarballs right?!) and your build.xml nicer. I won't include any examples here because I believe that Ivy documentation is indeed very good.

One lib/ to rule them all

In case you really don't want to use Ivy, make sure you place all your dependencies in one directory in top level of your project (don't scatter your dependencies, even if you are using multiple sub-projects). This directory should ideally be called lib/. It should contain your dependencies named as ${name}-${version}.jar. Most of the time you should include license files for every dependency you bundle, because you are becoming distributors and for most licenses this means you have to provide full text of the license. For licenses use identical name as jar filenames, but use ".license" suffix. All in all, make it easy to figure out your build dependencies and play with them.

Don't be too clever

I can't stress this enough. Try to keep your build.xml files to the bare minimum. Understanding ten 30 KiB big build.xml files with multiple-phase build and tests spread through 10 directories is no fun. Please think of poor packager when you write your build.xml files. I don't mind having grey hair that much, but I'd rather if it came later rather than sooner.

Maven

And now we are coming to my favourite part. Maven is a build and project management tool that has extensive plugin support able to do almost anything developer might ask for. And all that while providing formal project structure, so that once you learn how Maven works in one project you can re-use your knowledge in other projects.

Maven goodies

Maven provides several good things for packagers such as providing clear dependencies and preventing simple patched dependencies from sneaking in. Most important advantage for packagers coming with Maven is the fact that problems are the same in all projects. Once you understand how certain Maven plugin works, you will know what to expect and what to look for. But Maven is nice not just for packagers, but also for developers.

Declarative instead of descriptive

You don't tell Maven:

Add jar A, jar B to the classpath, then use this properies file to set-up test resources. Then compile tests (Have you compiled sources yet?) and then ... and run them with X

Instead you place test files and resources into appropriate directories and Maven will take care of everything. You just need to specify your test dependencies in nice and tidy pom.xml.

Project metadata in one place

With Maven you have all project information in one place:

Developer contact information
Homepage
SCM URLs
Mailinglists
Issue tracker URL
Project reports/site generation
Dependencies
Ability modify behaviour according to architecture, OS or other property

Need I say more? Fill it out, keep it up-to-date and we will all be happy.

Great integration with other tools

Ecosystem around Maven has been growing in past years and now you will find good support for handling your pom.xml files in any major java IDE. But that is just the tip of the iceberg. There are Maven plugins adding all kinds of additional tool support. Running checkstyle on your code, helping with licensing, integration with gpg, ssh, jflex and making releases. There are plugins for that and more.

Support for Ant

If you are in process of migrating your build system from Ant to Maven, you can do it in phases. For parts of your builds you can easily run Ant with maven-ant-plugin. Good example of such migration in progress is checkstyle. In version 5.2 they introduced Maven build system while preserving their old layout and running Ant for tests.

Maven messier side

A.K.A What you need to be aware of. It's generally quite hard to do something bad in Maven, because it won't let you do that easily. That said, there are plugins that can make it hard for us to package your software.

maven-dependency-plugin:copy-dependencies

This specific goal can potentially cause problems because it allows to copy classes from dependencies into resulting jar files. As I wrote last time, this is unacceptable because it creates possible licensing, security and maintenance nightmares. If you need even just one class from another project, rather than copying it, add it as a dependency into pom.xml

maven-shade-plugin

Shade plugin is a very shady plugin (pun intended). It can be used to weave depdencies inside your jars while changing their package names and doing all kinds of modifications in the process. I'll give you a small test now :-) Let's say you have jar file with following contents:

META-INF/
META-INF/MANIFEST.MF
META-INF/maven/
META-INF/maven/org.packager/
META-INF/maven/org.packager/Pack/
META-INF/maven/org.packager/Pack/pom.properties
META-INF/maven/org.packager/Pack/pom.xml
org/
org/packager/
org/packager/signature/
org/packager/signature/SignatureReader.class
org/packager/signature/SignatureVisitor.class
org/packager/signature/SignatureWriter.class
org/packager/Pack.class

Can you tell, from looking at jar contents where is org.packager.signature subpackage coming from? Take your time, think about it. Nothing? Well here's a hint:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-shade-plugin</artifactId>
  <configuration>
    <relocations>
      <relocation>
        <pattern>org.objectweb.asm</pattern>
        <shadedPattern>org.packager</shadedPattern>
      </relocation>
    </relocations>
  </configuration>
</plugin>

I believe this demonstrates why usage of shade plugin is evil (in 99% of cases at least). This is especially problematic if the shaded packages are part of public API of your project, because we won't be able to simply fix this in one package, but it will cascade up the dependency chain.

maven-bundle-plugin

Bundle is one of the more controversial plugins, because it can be used both for good and bad :-) One of the most important good use cases for bundle plugin is generating OSGI bundles. Every project can easily make their jar files OSGI compatible by doing something like this:

  ...
  <packaging>bundle</packaging>
  ...
  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.felix</groupId>
        <artifactId>maven-bundle-plugin</artifactId>
        <extensions>true</extensions>
      </plugin>
    </plugins>
  </build>
  ...

Easy right? Now to the darker side of bundle plugin. I have another example to test your skills. This one should be easier than shade plugin:

META-INF/MANIFEST.MF
META-INF/
META-INF/maven/
META-INF/maven/org.packager/
META-INF/maven/org.packager/Pack/
META-INF/maven/org.packager/Pack/pom.properties
META-INF/maven/org.packager/Pack/pom.xml
org/
org/objectweb/
org/objectweb/asm/
org/objectweb/asm/signature/
org/objectweb/asm/signature/SignatureReader.class
org/objectweb/asm/signature/SignatureVisitor.class
org/objectweb/asm/signature/SignatureWriter.class
org/packager/
org/packager/Pack.class

Problem is the same as with shade plugin (bundling of dependencies), but at least here it's more visible in the contents of the jar and it will not poison API of the jar. Just for the record, this is how it was created:

<plugin>
  <groupId>org.apache.felix</groupId>
  <artifactId>maven-bundle-plugin</artifactId>
  <extensions>true</extensions>
  <configuration>
    <instructions>
      <Export-Package>org.objectweb.asm.signature</Export-Package>
    </instructions>
  </configuration>
</plugin>

Summary

Today I wrote about:

Ant and why you shouldn't use it (that much)
Ant and how to use it if you have to
Maven and why it rocks for packagers and developers
Maven and its plugins and why they suck for packagers sometimes

There are a lot more things that can cause problems, but these are the most obvious and easily fixed. I'll try to gather more information about things we (packagers) can do to help you (developers) a bit more and perhaps include one final part for this guide.

Getting your Java Application in Linux: Guide for Developers (Part 1)

noreply@blogger.com (Stanislav Ochotnický) — Fri, 08 Apr 2011 15:26:00 +0000

Introduction to packaging Java

Packaging Java libraries and applications in Fedora has been my daily bread for almost a year now. I realized now is the time to share some of my thoughts on the matter and perhaps share a few ideas that upstream developers might find useful when dealing with Linux distributions.

This endeavour is going to be split into several posts, because there are more sub-topics I want to write about. Most of this is going to be based on my talk I did @ FOSDEM 2011. Originally I was hoping to just post the video, but it seems to be taking more time than I expected :-)

If you are not entirely familiar with status of Java on Linux systems it would be a good idea to first read a great article by Thierry Carrez called The real problem with Java in Linux distros. A short quote from that blog:

The problem is that Java open source upstream projects do not really release code. Their main artifact is a complete binary distribution, a bundle including their compiled code and a set of third-party libraries they rely on.

There is no simple solution and my suggestions are only mid-term workarounds and ways to make each other's (upstream ↔ downstream) lives easier. Sometimes I am quite terse in suggestions, but if need be I'll expand them later on.

Part 1: General rules of engagement

Today I am going to focus on general rules that apply to all Java projects wishing to be packaged in Linux distributions:

Making source releases
Handling Dependencies
Bugfix releases

For full understanding a short summary of general requirements for packages to be added to most Linux distributions:

All packages have to be built from source
No bundled dependencies used for building/running
Have single version of each library that all packages use

There are a lot of reasons for these rules and they have been flogged to death multiple times in various places. It mostly boils down to severe maintenance and security problems when these rules are not followed.

Making source releases

As I mentioned previously most Linux distributions rebuild packages from source even when there is an upstream release that is binary compatible. To do this we need sources obviously :-) Unfortunately quite a few (mostly Maven) projects don't do source release tarballs. Some projects provide source releases without build scripts (build.xml or pom.xml files). Most notable examples are Apache Maven plugins. For each and every update of one of these plugins we have to checkout the source from upstream repository and generate the tarball ourselves.
All projects using Maven build system can simply make packagers' lives easier by having following snippet in their pom.xml files:

    <build>
      <plugins>
 ...
 <plugin>
   <artifactId>maven-assembly-plugin</artifactId>
   <configuration>
     <descriptorRefs>
       <descriptorRef>project</descriptorRef>
     </descriptorRefs>
   </configuration>
   <executions>
     <execution>
       <id>make-assembly</id>
       <phase>package</phase>
       <goals>
         <goal>single</goal>
       </goals>
     </execution>
   </executions>
 </plugin>
 ...
      </plugins>
    </build>

This will create -project.zip/tar.gz files containing all the files needed to rebuild package from source. I have no real advice for projects using Ant for now, but I'll summarise them next time.

Handling dependencies

I have a feeling that most Java projects don't spend too much time thinking about dependencies. This should change so here are a few things to think about when adding new dependencies to your project.

Verify if the dependency isn't provided by JVM

Often packages contain unnecessary dependencies that are provided by all recent JVMs. Think twice if you really need another XML parser.

Try to pick dependencies from major projects

Major projects (apache-commons libraries, eclipse, etc.) are much more likely to be packaged and supported properly in Linux distributions. If you use some unknown small library packagers will have to package that first and this can sometimes lead to such frustrating dependency chains they will give up before packaging your software.

Do NOT patch your dependencies

Sometimes a project A does almost exactly what you want, but not quite...So you patch it and ship it with your project B as a dependency. This will cause problems for Linux distributions because you basically forked the original project A. What you should do instead is work with the developers of project A to add features you need or fix those pesky bugs.

Bugfix releases

Every software project has bugs, so sooner or later you will have to do a bugfix release. As always there are certain rules you should try to uphold when doing bugfix releases.

Use correct version numbers

This depends on your versioning scheme. I'll assume you are using standard X.Y.Z versions for your releases. Changes in Z are smallest released changes of your project. They should mostly contain only bugfixes and unobtrusive and simple feature additions if necessary. If you want to add bigger features you should change Y part of the version.

Backward compatible

Bugfix releases have to be backwards compatible at all times. No API changes are allowed.

No changes in dependencies

You should not change dependencies or add new ones in bugfix releases. Even updating dependency to a new version can cause massive recursive need for updates or new dependencies. The only time it's acceptable to change/add dependency version in bugfix release is when new dependency is required to fix the bug.

An excellent example of how NOT to do things was Apache Maven update from 3.0 to 3.0.1. This update changed requirements from Aether 1.7 to Aether 1.8. Aether 1.8 had new dependency on async-http-client. Async-http-client depends on netty, jetty 7.x and more libraries. So what should have been simple bugfix update turned into need for major update of 1 package and 2 new package additions. If this update contained security fixes it would cause serious problems to resolve in timely manner.

Summary

Create source releases containing build scripts
Think about your dependencies carefully
Handle micro releases gracefully

Next time I'll look into some Ant and Maven specifics that are causing problems for packagers and how to resolve them in your projects.

Accessing WebDAV calendar from commandline

noreply@blogger.com (Stanislav Ochotnický) — Fri, 11 Mar 2011 11:00:00 +0000

Have you tried to access Zimbra or Google Calendar from command line? I have. And I couldn't find any normal command line client that would be able to read and write these calendars, display alerts etc. Well there is a googlecl project , but it's specific for Google Calendar and is not using standard WebDav iCal access methods.

Thus I set out to create console application that would fulfil my needs. What are my requirements?:

Read/write access to Google Calendar and Zimbra (at least)
Multiple remote calendars
Working alerts
Nice ncurses UI (but also ability to just display some info and quit)
Correct handling of timezones
Integration with mail client (open ics files received by email)
I guess a lot more :-)

I had a look at existing python modules that work on iCalendar, WebDAV and combination of both. There are quite a few of them, but I just didn't like their APIs. They were usually complex and required knowledge of iCal specification. So I decided to create simplified module that would be easy to understand (even if not so powerful).

I named the project pywebcal (yes, unimaginative) and it's now on github. I would LOVE some input. I know it's far from perfect (or complete), but let's see. For now it offers read-only support for Zimbra (Google should work too but I haven't tested in a while).

You can have a look at the example directory that contains one simple example you can run in-place and see if it works :-) I did my best to create proper test cases covering problems with timezones and whatnot, and this helped me quite a lot with recent refactoring. I am now using vobject library as my backend and it is rather nice to use. Plan is to allow access to vobject components so that my simplified API is not preventing some advanced modifications.

Next step is obviously to start working on ncurses application itself. Anyone wants to help?

Problems with running gpg-agent as root

noreply@blogger.com (Stanislav Ochotnický) — Mon, 14 Feb 2011 17:21:00 +0000

This is gonna be short post for people experiencing various issues with pinentry and gpg-agent. This is mostly happening on systems with only gpgv2.

I have been asked to look at bug 676034 in Red Hat Enterprise Linux. There we actually two issues there:

Running pinentry with DISPLAY variable set but no available GUI pinenty helpers
Using gpg on console after doing "su -"

First problem was relatively easy to figure out. Pinentry finds DISPLAY variable and looks for pinentry-gtk, pinentry-qt or pinentry-qt4 helpers to ask for passphrase. Unfortunately if none of these GUI helpers can be found, pinentry doesn't try their console counterpart. Workaround is simple: unset DISPLAY variable if you are working over ssh connection (or don't use X forwarding when you don't need it). More recent pinentry features proper failover to pinentry-curses

Second problem was a bit more tricky to figure out, although in the end it was a facepalm situation. When trying to use GNUPG as root on console, hoping for pinentry-curses to ask for passphrase, users were instead introduced to this message: ERR 83886179 Operation cancelled. To make things more confusing, everything seemed to work when logging in as root directly from ssh.

At first I thought that this must be caused by environment variables, but this seemed to be incorrect assumption. Instead the reason was that current tty was owned by original owner and not root. This seemed to cause problem with gpg-agent and/or ncurses pinentry. I will investigate who was the real culprit here, but this bug seems to be fixed at least in recent Fedoras

So what should you do if you have weird problems with gpg and pinentry as root? Here's what:

    $ su -
    [enter password]
    # chown root `tty`
    [use gpg, pinentry as you want]

Easy right? As a final note...I've been to FOSDEM and I plan to blog about it, but I guess I am waiting for the videos to show online. It's quite possible I'll blog about it before that however, since it's taking a while.

Automatic squashing of last git commits

noreply@blogger.com (Stanislav Ochotnický) — Fri, 10 Dec 2010 13:16:00 +0000

I have written before about my workflow in Fedora. This workflow includes a relatively high number of rebasing where I squash last two commits into one. I use it to quickly refine and test patches. My history then usually looks something like this:

$ git log --format=oneline --decorate=no | head -3
2db8eacd7f7c20be88824caae5f5af16b9520d34 temp
443bb0f019c87f0090bb9da295a019c0eee23729 Add conditional BRs to enable ff merge between f14
b4c602f9f044598544cff3d68710f68b9447ea0f Fix installation of pom files for artifact jars

Where 443bb0f is my first attempt at the fix and 2db8eac is a fixed fix :-). Before pushing this into upstream repository I usually squash last two commits to look like this:

$ git log --format=oneline --decorate=no | head -2
f852c3e260d21bbc642f861e6fa6ea62caa7b69b Add conditional BRs to enable ff merge between f14
b4c602f9f044598544cff3d68710f68b9447ea0f Fix installation of pom files for artifact jars

I used to do it manually, but I do it often enough it made sense to automate this. So without further ado:

#!/bin/sh

export EDITOR="sed -i '2s/pick/squash/;/# This is the 2nd commit message:/,$ {d}'"

git rebase -i HEAD~2

Save this somewhere inside your $PATH and chmod +x (or set up shell alias). Then just running this will automatically squash last two commits, using HEAD~1 commit message and discarding last commit message. No warranties though :-)

Edit: Benjamin suggested to use "fixup" instead of "squash". This is new thing in git 1.7+. For more information on this see this blogpost.

Fedora has Freemind

noreply@blogger.com (Stanislav Ochotnický) — Tue, 30 Nov 2010 18:33:00 +0000

During past few months I have been reviewing packages for Fedora with approximate speed of 1.53 package per week :-)

Most of packages I reviewed were libraries and other things that are not so interesting to users. But one of last packages I reviewed is different. It is one of the best mind-mapping software in existence :-) Yes, it's Freemind. If you've never tried mind-mapping before, let me quote first paragraph from wikipedia:

A mind map is a diagram used to represent words, ideas, tasks, or other items linked to and arranged around a central key word or idea. Mind maps are used to generate, visualize, structure, and classify ideas, and as an aid to studying and organizing information, solving problems, making decisions, and writing.

So what are you waiting for? Try it out!

There were attempts to package Freemind before, but all would-be maintainers gave up when they noticed how many dependencies are needed and how Java packaging actually works (I don't really blame them). So everyone please. Let's. Have. A. Staaaaanding. Ovaaation. Fooor. Johannes Lips. He made it! If you live close by, buy him a beer will you? :-)

Local DNS caching - just do it already!

noreply@blogger.com (Stanislav Ochotnický) — Thu, 11 Nov 2010 13:00:00 +0000

I recently encountered weird problems with my network connection at home. Everything worked, but was unbelievably slooooow. Ping showed times of ~30 ms, but I could easily see it took more time for those packets to go there and back.

I took me some time to figure out what was happening. Looking back, checking DNS server should have been one of the first things to do. Seems like first DNS server provided by my provider has been down. That meant that every DNS query timed out and then went to the second DNS which got me my response. For some reason ping did DNS query before every new package being sent. That explains its weird behaviour.

This problem got me to finally install local caching dns. I was thinking about doing it before, but I never got around to do it until now. I always thought it's gonna be a few-hour nightmare. Now I blame my previous experience with bind :-D For simple local caching bind would be overkill, so I chose dnsmasq. Using it was as simple as installing, running dnsmasq and executing

$ echo 'nameserver 127.0.0.1' > /etc/resolv.conf.head

From that point on every resolv.conf file generated by dhcpcd will have my local DNS as first DNS server to try. For this time you can add it there manually. Then you can verify your setup works by running following command twice in a row:

$ dig randomserver.com

First execution should have Query time: XX msec with XX being few tens of miliseconds. Query time for second run should be zero or very close to zero.

Congratulations. You have your very own caching server. Who knows...maybe you'll even notice some improvements in your network connection :-)

PyQTrailer revisited

noreply@blogger.com (Stanislav Ochotnický) — Wed, 10 Nov 2010 20:22:00 +0000

Some time ago, I wrote about my little project: apple.com trailer downloader. Apple is still not very open-source friendly as far as its trailer website is concerned. So all points I made in my original post still stand. To my surprise this little project is still alive and kicking, with new ideas for improvements coming and coming :-). What is even more important: it seems that so far no breakage happened due to apple changing something on the web.

Since the first version I released almost 6 months ago several new features appeared. Some of them include:

Parallel downloading of trailers
Ability to run movie player (mplayer, vlc, etc) without downloading file to HDD
Lot of customisation/performance options added
Working support for trailer search
Localisation support
Python 3 support

Latest version (0.5.2) is available in Gentoo repositories already, and should hit Fedora updates in next day or so (this will be delayed due to new package acceptance criteria though). Enjoy.

Python3, PyQt4 and missing QString

noreply@blogger.com (Stanislav Ochotnický) — Wed, 20 Oct 2010 21:55:00 +0000

As I was recently adding support for Python 3 to my little trailer downloader application that I mentioned before (PyQTrailer) I encountered a strange problem with PyQt4 that only occurred in Python 3.

Let's take this simple python example:

$ python
>>> from PyQt4.QtCore import QString
>>>

That same code snippet doesn't work in python3 interpreter though:

$ python3
>>> from PyQt4.QtCore import QString
Traceback (most recent call last):
  File "", line 1, in 
ImportError: cannot import name QString
>>>

My first instinct was: Bug! Gentoo PyQt4 ebuild was doing something terrible and somehow made PyQt4 unusable in python3 interpreter. Turns out my gut instinct was wrong (once again :-) ).

PyQt4 since version 4.6 changed API of QString and QVariant for python 3.x. For QString this is due to fact that from Python 3.0, string literals are unicode objects (no need for u'unicode' magic anymore). This means that you can use ordinary Python strings in place for QString. But I wanted my QString for something like this:

  ...
  downloadClicked = pyqtSignal((QString, ))
  ...

This snippet creates Qt signals that you can then emit. Question is... How can we update this for Python 3.x? We could probably just replace QString with type(""), but for a change that wouldn't work with Python 2.x. So? Python dynamic nature to the rescue!
Edit: simplified QString definition (thanks Arfrever)

try:
    from PyQt4.QtCore import QString
except ImportError:
    # we are using Python3 so QString is not defined
    QString = str

If we put previous code sample to the beginning of our Python file we can use QString in our code and it will keep working both in Python 3.x and Python 2.x. Case closed dear Watson.

Fedora RPG - Three level badge system?

noreply@blogger.com (Stanislav Ochotnický) — Wed, 20 Oct 2010 08:12:00 +0000

I stumbled upon one great idea on Fedora Planet. It is nothing other than Fedora RPG!

In short, it's a system to create characters similar as they are in Role-playing games (RPGs) with levels, skill points and more.

You might think it doesn't make sense to give contributors "points" for non-gaming activities but you would be wrong. Most communities have created ways to reward their members this way. Look no further than my favourite stackoverflow.com. It also uses badge and skill point system for various actions on the website. In one of earliest blog posts about how stackoverflow will work, Jeff Atwood shared his vision: three levels of badges (bronze, silver, gold). Each level with unique badges tailored to the purpose of stackoverflow.

I guess Fedora RPG will go a bit further in this regard. I would love to know how it will all turn out and how the levels will work. Let the games begin!

Packaging workflow, patch management and git magic in Fedoraland

noreply@blogger.com (Stanislav Ochotnický) — Fri, 15 Oct 2010 10:51:00 +0000

Big part of my job is packaging for Fedora Linux (I am pretty sure I haven't mentioned this before :-) ). I have spent last 6 months working on various Java packages, adding new packages to Fedora, updating dependencies etc. I have developed certain workflow which I believe might be of interest to other packagers. So here goes. Most of these hints are about managing patches for your packages. I'll also try to work on concrete package so it won't be completely theoretical.

Let's assume your project already has some history and patches. Let's fix velocity bug 640660 for example. I'll start with steps I took and what they meant, and I'll summarize in the end with rationale what I have gained by using my workflow (and what could be improved).

After modifying BuildRequires and Requires to tomcat6 servlet api I tried to build velocity:

$ fedpkg mock

This is what I got:

---snip----
compile-test:
    [javac] Compiling 125 source files to /builddir/build/BUILD/velocity-1.6.3/bin/test-classes
    [javac] /builddir/build/BUILD/velocity-1.6.3/bin/test-src/org/apache/velocity/test/VelocityServletTestCase.java:135: org.apache.velocity.test.VelocityServletTestCase.MockServletContext is not abstract and does not override abstract method getContextPath() in javax.servlet.ServletContext
    [javac]     static class MockServletContext implements ServletContext
    [javac]            ^
    [javac] Note: /builddir/build/BUILD/velocity-1.6.3/bin/test-src/org/apache/velocity/test/VelocityServletTestCase.java uses or overrides a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
    [javac] Note: Some input files use unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.
    [javac] 1 error
BUILD FAILED
/builddir/build/BUILD/velocity-1.6.3/build/build.xml:251: Compile failed; see the compiler error output for details.
Total time: 47 seconds
---snip---

The issue seems simple to fix, just missing stub function in a test case, right? So what now?

$ fedpkg prep
$ mv velocity-1.6.3 velocity-1.6.3.git
$ cd velocity-1.6.3.git
$ git init && git add . && git commit -m 'init'

This effectively created my small git repository for sources and populated it with all files. Using fedpkg prep step we extracted the tarball and applied already existing patches to unpacked sources. I suggest you create shell alias for last three commands as you'll be using it a lot. We moved directory to velocity-1.6.3.git so that next (accidental?) fedpkg prep won't erase our complicated changes (yes it happened to me once. I've had better days). Note that velocity-1.6.3.git is not a temporary directory. I will keep it around after fixing this bug so that I can use git history, diffs and other features in the future. It is especially nice when you have packages with lot of patches on top.

Now we can easily work in our new git repository, edit source file in question and do:

$ git add src/test/org/apache/velocity/test/VelocityServletTestCase.java
$ git commit -m 'Fix test for servlet api 2.5'
$ git format-patch HEAD~1

This created commit with descriptive message and generated a patch file 0001-Fix-test-for-servlet-api-2.5.patch in our current directory. This is how the patch looks like:

From 8758e3c83411ffadc084d241217fc25f1fd31f42 Mon Sep 17 00:00:00 2001
From: Stanislav Ochotnicky <sochotnicky@redhat.com>
Date: Thu, 14 Oct 2010 10:20:52 +0200
Subject: [PATCH] Fix test for servlet api 2.5

---
 .../velocity/test/VelocityServletTestCase.java     |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)

diff --git a/src/test/org/apache/velocity/test/VelocityServletTestCase.java b/src/test/org/apache/velocity/test/VelocityServletTestCase.java
index 824583e..ac0ab5c 100644
--- a/src/test/org/apache/velocity/test/VelocityServletTestCase.java
+++ b/src/test/org/apache/velocity/test/VelocityServletTestCase.java
@@ -16,7 +16,7 @@ package org.apache.velocity.test;
  * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  * KIND, either express or implied.  See the License for the
  * specific language governing permissions and limitations
- * under the License.    
+ * under the License.
  */
 
 import java.io.IOException;
@@ -149,6 +149,11 @@ public class VelocityServletTestCase extends TestCase
             return this;
         }
 
+        public String getContextPath()
+        {
+            return "";
+        }
+
         public String getServletContextName()
         {
             return "VelocityTestContext";
-- 
1.7.2.3

Now that we have patch prepared for velocity we need to use it in the spec file and we're done.

Let's say our first attempted patch wouldn't work as expected and build (or test) still failed. We modify the sources again and do another commit. What we have now is:

$ git log --format=oneline
c15f7e02eaae93b755cc0bfde6def3d6e67d2b0f (HEAD, master) Fix previous commit
3e3d654c142c7028c9c7895579fba204c4c4bf08 Fix test for servlet api 2.5
2f32554ddf892f4cca3f78b1f82a7c3ab169c357 init

We don't want two patches in the spec file for one fix so: time for git magic. You've probably heard of git rebase if you've been using git for a while. What we want to do now is merge last two commits into one, or in git-speak "squash" them. To do this you have to do:

$ git rebase -i HEAD~2

Now your editor should pop-up with this text:

pick 3e3d654 Fix test for servlet api 2.5
pick c15f7e0 Fix previous commit

# Rebase 2f32554..c15f7e0 onto 2f32554
#
# Commands:
#  p, pick = use commit
#  r, reword = use commit, but edit the commit message
#  e, edit = use commit, but stop for amending
#  s, squash = use commit, but meld into previous commit
#  f, fixup = like "squash", but discard this commit's log message
#
# If you remove a line here THAT COMMIT WILL BE LOST.
# However, if you remove everything, the rebase will be aborted.
#

So we just need to change "pick c15f7e0 Fix previous commit" into "squash c15f7e0 Fix previous commit" (you can also use just 's' instead of 'squash'). Save. Close. Another editor window will open with something like this:

# This is a combination of 2 commits.
# The first commit's message is:

Fix test for servlet api 2.5

# This is the 2nd commit message:

Fix previous commit

# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.

For this case we will delete second message because we just want to pretend our first attempt was perfect :-). Save. Close. Now we have:

$ git log --format=oneline
cbabb6ac43f7bdb8e52ccd09c25cfd0a032b553c (HEAD, master) Fix test for servlet api 2.5
2f32554ddf892f4cca3f78b1f82a7c3ab169c357 init

Repeat as many times as you want. You can also re-order commits and change commit messages with rebase (note that if you just want to change last commit message you can do "git commit --amend"). I generally don't create commits until I have working patch though.

So why do I think all this mumbo-jumbo improves my workflow? Let's see:

I can have long comments for every patch I create (instead a line or two in spec file)
I can use the same patches to send directly to upstream
I don't have to juggle around with diff and remember what files I changed where
Probably several other things I haven't even realized

I have a few things that bother me of course. git format-patch generates filenames that are different from standard practice of %{name}-%{version}-message.patch. This is not a git problem. For packages where only my patches exist I stick with git naming, but when there are different patches I stick with naming they started. Another thing that is bothering me is that creating initial repository by using "fedpkg prep" hides patches that were applied to sources. That's why I am thinking about re-working my packages so that all patches will be in my git repositories as commits with descriptive messages. No need for comments in the spec file anymore. Perhaps someone can suggest other improvements to my approach.

Uploading original photos from Digikam to Flickr

noreply@blogger.com (Stanislav Ochotnický) — Sat, 28 Aug 2010 20:13:00 +0000

I have been using Digikam for managing my photos for some time. It's pretty neat software and development is progressing quite fast. You can think of it as Lightroom, just without the neat non-destructive editing. But that is coming too, thanks to Google and another round of Summer of Code participants. But I wouldn't be writing this blog post if Digikam was flawless would I? :-)

First I'll describe my work-flow in few short bullet-points :-)

Shoot a LOT of photos
Delete almost as many photos
Geotag, tag/keyword and rate remaining photos
If it was party or something similar upload to Facebook right away
Pick few photos and improve them a bit with Gimp (nothing fancy, just crop/levels)
Upload all photos to Flickr as a backup/sharing place

Digikam enables me to work like this, except the last point. Why? Because apparently Digikam developers don't think anyone would update their photos to Flickr without first resizing/recompressing them. This how flickr export dialog looks like this:

See the problem? Even if I don't chose "Resize photos before uploading" Digikam will still re-compress them which is a Really-Bad-Thing(tm) to do with jpeg files. I had some previous experience with Qt3 and even Qt4. so I though it might be a good idea to look into fixing this small annoyance. I will not bore you with the details how I checked out svn repository with git and rest of the stuff. Here is the result:

If you un-check "Send original file (no resizing)" original settings will appear and you can resize/recompress as much as you want (Blasphemy! Madness!). The patch is not flawless, because it won't prevent you from trying to upload RAW files to Flickr, but it's good enough for me :-) It's not even that big, stats are like this:

flickrexport/flickrtalker.cpp |   18 ++++++++++++------
flickrexport/flickrtalker.h   |    2 +-
flickrexport/flickrwidget.cpp |   34 ++++++++++++++++++++++++++--------
flickrexport/flickrwidget.h   |    5 +++++
flickrexport/flickrwindow.cpp |    4 ++++
flickrexport/flickrwindow.h   |    1 +
6 files changed, 49 insertions(+), 15 deletions(-)

You can download the patch from my Dropbox for now until the bugreport I created some time ago will get sorted out (don't hold your breath too much though). The patch applies cleanly across all versions of kipi-plugins I tried (from 0.8.something to 1.4.0). Happy uploading.

Downloading trailers on Linux - final solution

noreply@blogger.com (Stanislav Ochotnický) — Mon, 10 May 2010 21:19:00 +0000

I love films. All of them to be exact. I believe you just have to be in the right mood and you would enjoy even few of the worst movies ever made. Even though I am a proponent of open source philosophy, we as a society are obviously not ready to embrace it in entertainment industry just yet.

This is where www.apple.com/trailers comes into play. Apple made great deals with movie studios and you can watch/download newest movie trailers. Well...sort of. Apple employs variety of restrictions which makes this site next to useless on a Linux desktop. It hides links to trailers themselves behind reference files so that when you download with your favorite browser, you will only get small reference file not the trailer itself. And that is after you circumvent user-agent protection. Because apple believes nothing but iTunes/iPad/iOtherAppleStuff should access these trailers. There are scripts around that can make downloading possible for Linux users. I have been using Apple Trailer Download script for Greasemonkey for quite some time, but it always stopped working after some time.

Another opportunity for me I guess. I have been trying to improve my Python-fu for some time so what better way then a small project like this? I started last weekend after I found out Apple actually publishes JSON data of trailers on its site. This made access quite easy from python and is quite error-prone to changes of website itself (as long as Apple doesn't pull whole JSON thingy...but they are actively using it too). Long story short...there are two outputs from this endeavor:

pytrailer - python module to simplify access to movies on apple.com/trailers
pyqtrailer - Qt4 interface that displays poster, movie information and enables downloading of trailers

You can report bugs on respective websites (there are quite a few now, but basic downloading for hd trailers works). If you want to try it out just running:

# easy_install pyqtrailer
should work as long as you have PyQt4 installed. You can just run pyqtrailer now and you should see something like this:

That's it. I will improve/fix it a bit but don't expect too much :-)

And he's back! (from hibernating)

noreply@blogger.com (Stanislav Ochotnický) — Sat, 27 Mar 2010 22:41:00 +0000

What better way to celebrate summer solstice, than by making my computer able to hibernate? Since my last post a lot has happened with me. I got a new phone (HTC Hero FTW!), I finished university, went traveling a bit and I also got a new notebook (because the old one died on me). R.I.P. Thinkpad R51, welcome Thinkpad T500. There are several things I could start writing about now. Starting with how great Hero and Android is to use all the way to today's blog post: How to make my computer hibernate?

Linux has had support for hibernating for quite a few years now and although it's not perfect, it usually works out of the box. What it needs however is swap device big enough so that it can store image of memory for hibernating. Now I hit a problem. When I got my new Thinkpad I thought "Hey, I have 4GB of RAM...why would I need a swap?". And even if I REALLY needed more than 4GB RAM I can still create temporary swap by using swapfile. Unfortunately I couldn't make swapfile on LVM work with TuxOnIce. TuxOnIce also has another alternative to swap or swapfile for hibernating: Using filewriter, which is quite similar to swapfile support, I managed to get it to work (after some work, kernel debugging and one small patch to TuxOnIce).

I set FilewriterLocation in hibernate.conf to point to a place where I wanted to store hibernation file and I set the size to 4GB. As instructed in TuxOnIce HOWTO, I then ran

hibernate --no-suspend

to create this image. It created the file as expected, but when it was supposed to tell me settings for bootloader (resume argument) it silently failed. When I tried again, whole computer froze. I was puzzled. How could this happen? I am using Linux so things like this don't happen! But hey, I should be able to figure out what's wrong with it right? I set up my kernel to include netconsole, and ran hibernate again. This time I caught where the bug happened. The output was something like this:

TuxOnIce: No image found.
BUG: unable to handle kernel paging request at 6539207a
IP: [<c10763a6>] toi_attr_store+0x186/0x2a0
*pdpt = 0000000032732001 *pde = 0000000000000000
Oops: 0000 [#1] PREEMPT SMP
last sysfs file: /sys/power/tuxonice/file/target
Modules linked in: netconsole aes_i586 aes_generic radeon ttm drm_kms_helper drm
 i2c_algo_bit sco bnep ipt_MASQUERADE iptable_nat nf_nat ipt_LOG nf_conntrack_ip
v4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_t
ables x_tables rfcomm l2cap vboxnetadp vboxnetflt vboxdrv arc4 iwlagn iwlcore ma
c80211 sdhci_pci snd_hda_codec_conexant sdhci pcmcia e1000e uvcvideo mmc_core cf
g80211 snd_hda_intel yenta_socket btusb rsrc_nonstatic tpm_tis pcspkr pcmcia_cor
e videodev v4l1_compat intel_agp wmi agpgart tpm snd_hda_codec tpm_bios video fu
se xfs raid10 raid1 raid0 md_mod scsi_wait_scan sbp2 ohci1394 ieee1394 usbhid uh
ci_hcd usb_storage ehci_hcd usbcore sr_mod sg uvesafb cfbfillrect cfbimgblt cn c
fbcopyarea [last unloaded: microcode]
 
Pid: 12870, comm: hibernate Not tainted 2.6.33.1-w0rm #16 2082BRG/2082BRG
EIP: 0060:[<c10763a6>] EFLAGS: 00010202 CPU: 0
EIP is at toi_attr_store+0x186/0x2a0
EAX: 00000000 EBX: 36203430 ECX: 00000000 EDX: f231f200
ESI: 65392066 EDI: 00f60062 EBP: f6006331 ESP: f62a7f14
 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Process hibernate (pid: 12870, ti=f62a6000 task=f20a0270 task.ti=f62a6000)
Stack:
 00000000 fffffff4 00000001 c1790ca0 00000000 f6e8ab64 c16c75a4 f6d1c380
<0> f62a7f64 c114298d 00000015 00000015 b7709000 f21385c0 f6d1c394 c16c75a4
<0> f6ec7ac0 f21385c0 b7709000 00000015 f62a7f8c c10f207c f62a7f98 00000000
Call Trace:
 [<c114298d>] ? sysfs_write_file+0x9d/0x100
 [<c10f207c>] ? vfs_write+0x9c/0x180
 [<c11428f0>] ? sysfs_write_file+0x0/0x100
 [<c10f221d>] ? sys_write+0x3d/0x70
 [<c1002ccc>] ? sysenter_do_call+0x12/0x22
Code: c7 45 e0 00 00 00 00 3b 5d 08 0f 85 e9 fe ff ff 8b 46 20 85 c0 0f 84 de fe
 ff ff ff d0 8b 7d e0 85 ff 8d 76 00 0f 84 d9 fe ff ff <8b> 46 14 31 d2 e8 60 03
 05 00 8b 46 10 c7 46 14 00 00 00 00 a8
EIP: [<c10763a6>] toi_attr_store+0x186/0x2a0 SS:ESP 0068:f62a7f14
CR2: 000000006539207a
---[ end trace 124a5ee29ef71277 ]---

So what can we deduce from this bug output? Let's go from the top. Bug name (unable to handle kernel paging request) means that it is likely a memory corruption issue. Someone accessed memory that he was not supposed to. IP tells us that function where the error occurred was toi_attr_store in unknown file, unknown line (I don't have debug information included in kernel). There are other information we can get from that output, but I didn't really need them. Quick search through kernel sources told me that toi_attr_store is a function inside kernel/power/tuxonice_sysfs.c. I scanned the code, learning what approximately it did. Then I placed printk statements thorough the function so that I could approximate where inside the function the code fails. After some time I narrowed it down to following snippet:


	if (!result)
    		result = count;

        /* Side effect routine? */
        if (result == count && sysfs_data->write_side_effect)
                sysfs_data->write_side_effect();

        /* Free temporary buffers */
        if (assigned_temp_buffer) {
                toi_free_page(31,
                        (unsigned long) sysfs_data->data.string.variable);
                sysfs_data->data.string.variable = NULL;
        }

Kernel crashed when it tried to call toi_free_page. After a few reboots and printks later I found out that this was just a coincidence, and sysfs_data variable itself became corrupt even before the call to the toi_free_page. Good candidate? Of course: write_side_effect. But what exactly was write_side_effect? This function was passed as an argument, and therefore I wasn't able to easily find out what was real code executed at this point. Time to find out! From my previous debugging attempts I knew code failed while it tried to write location of my resume file into /sys/power/tuxonince/file/target. TuxOnIce code defined handling for string sysfs arguments as such:


#define SYSFS_STRING(_name, _mode, _string, _max_len, _flags, _wse) { \
        .attr = {.name  = _name , .mode   = _mode }, \
        .type = TOI_SYSFS_DATA_STRING, \
        .flags = _flags, \
        .data = { .string = { .variable = _string, .max_length = _max_len } }, \
        .write_side_effect = _wse }

I found this macro used inside tuxonice_file.c source code like this:

 
SYSFS_STRING("target", SYSFS_RW, toi_file_target, 256,
                SYSFS_NEEDS_SM_FOR_WRITE, test_toi_file_target)

So we found our write_side_effect code inside test_toi_file_target function. In one part this function was calling hex_dump_to_buffer to convert device UUID into hexadecimal string. The call looked like this:

 
hex_dump_to_buffer(fs_info->uuid, 16, 32, 1, buf, 50, 0);

This should convert input (fs_info->uuid) into hexadecimal string and store it inside buf. Author of the original code correctly thought about function adding spaces between bytes and therefore need to have more space in the buffer (argument 50 is telling hex_dump_to_buffer how big is output buffer). Unfortunately that same author declared buf as 33 char array. hex_dump_to_buffer therefore stepped outside the buffer and corrupted memory, causing all the problems. I fixed this bug, and sent a patch to the tuxonice-devel mailing list. As of now, it is already in the git repository ready to be released with next bugfix release of TuxOnIce.

That is everything for today, but as I already noted I am using LVM on my system (except root partition) and also use fbsplash for nice animations while rebooting. I am using initrd for this, and I will have another post on that topic.

Mobile (not so) open standards

noreply@blogger.com (Stanislav Ochotnický) — Tue, 25 Aug 2009 22:16:00 +0000

Yesterday I promised I'll talk about why I hate mobile phones. Of course I didn't mean all of them. Just the ones I have to deal with. Why? Well my mobile phone kind of died few days ago. I have a Nokia N73 and it's really quite good phone even if it's a bit old by today's standards. You control the phone by using "joystick" kind of thing in the upper part of keyboard. I decided to include image so you don't have to look for it :-)

So this joystick stopped working (even slightest touch would be evaluated as pushing it, therefore it was unusable). I didn't have my backup phone with me, but one friend gave me her battered Siemens S55. So what was the problem? Well I have the same sim card for almost 10 years now. Back then only 100 contacts would fit on it. I have almost 300 contacts in my N73. So how do I get all contacts from one phone to the other? Normally I could just send them through bluetooth, but since I couldn't really control my N73 this was out of question. I was barely able to turn on the bluetooth. I thought that I'll use SyncML interface to get vCards from N73 to my computer and then sync them again to the S55. In the end I kind of did, but boy was that an unpleasant experience!

So what exactly happened? I installed OpenSync libraries and tools and using multisyncgui I created sync group with one side being file-sync plugin and other was syncml-obex-client plugin. Configuraion of file-sync plugin was mostly just changing path to directory where I wanted to sync. Final version looked like this:


<?xml version="1.0"?>
<config>
  <directory>
    <path>/tmp/sync</path>
    <objtype>contact</objtype>
    <objformat>vcard30</objformat>
  </directory>
</config>

Configuration for syncml-obex-client appeared to be much more challenging. It appears that Nokia N73 has two quirks:

It only talks to SyncML client if it says its name is "PC Suite"
It contains a bug that causes it to freeze after certain amount of data if configuration is not correct

First of these quirks is mentioned in almost every tutorial on data synchronization in Linux. However the second one caused me to lose quite some time. My Nokia N73 would freeze after synchronizing approximately 220-240 contacts. To continue working I had to restart the whole phone.In the end I found out that I need to set parameter recvLimit to 10000 in order to synchronize everything. Final setting for syncml-obex-client looks like this:


<?xml version="1.0"?>
<config>
  <type>2</type>
  <bluetooth_address>00:1B:33:3A:D1:37</bluetooth_address>
  <!-- Nokia Obex server channel -->
  <bluetooth_channel>13</bluetooth_channel>
  <interface>0</interface>
  <identifier>PC Suite</identifier>
  <version>1</version>
  <wbxml>1</wbxml>
  <username></username>
  <password></password>
  <usestringtable>1</usestringtable>
  <onlyreplace>0</onlyreplace>
  <onlyLocaltime>0</onlyLocaltime>
  <recvLimit>10000</recvLimit>
  <maxObjSize>0</maxObjSize>
  <database>
     <name>Contacts</name>
     <objtype>contact</objtype>
     <objformat>vcard21</objformat>
  </database>  
</config>

So after all that I was able to get vCards from my N73 to my notebook. For every vCard OpenSync created file in directory /tmp/sync. Now came the interesting part. How to get these vCards to Siemens S55?

Simple Google search on Siemens S55 and synchronization in Linux seemed to suggest that tool most suited to do the job was scmxx. This little app is specialized on certain Siemens phones. According to some manuals it was supposed to be able to upload vCards themselves, however I couldn't get it to work as scmxx was complaining about invalid command line arguments.After some testing I found out that it could access and change sim card phone numbers.

Unfortunately for me, my sim card has limit of 100 phone numbers, each with 14 character identifier (name). This meant I needed to convert vCards from N73 to special format that scmxx used. Mentioned format looked something like this:


1,"09116532168","Jones Rob"
2,"09223344567","Moore John"
...

First column being number of slot that will be overwritten by new information, second column is number and third one name of contact (less than 15 characters).

So I fired up vim and started coding conversion script. It didn't take long and I had my contact in the old-new phone. There are a lot of hard-coded things in that script since I don't plan to ever use it again but you can download it from my dropbox. Consider it public domain, and if anyone asks I didn't have anything to do with it :-)


import os
import re

MAX_CONTACTS=100

class PbEntry(object):

    def __init__(self, name, tel, year, month, day):
        self.name = name
        self.tel = tel
        self.year = year
        self.month = month
        self.day = day

def cmp_pb(e1, e2):
    if e1.year > e2.year:
        return -1
    elif e1.year < e2.year:
        return 1
    else:
        if e1.month > e2.month:
            return -1
        elif e1.month < e2.month:
            return 1
        return 0


telRe = re.compile('TEL(;TYPE=\w+)*:([*#+0-9]+)', re.M)
revRe = re.compile('REV:(\d{4})(\d{2})(\d{2}).*', re.M)
nameRe = re.compile('^N:(.*);(.*);;;', re.M)
def get_entry_from_text(text):
    ret = nameRe.search(text)
    surname = None
    name = None
    tel = None
    rev = None
    if ret:
        surname =  ret.group(1)
        name = ret.group(2)

    ret = telRe.search(text)
    if ret:
        tel =  ret.group(len(ret.groups()))
    
    if surname and name:
        fn = "%s %s" % (surname,name)
    elif surname:
        fn = surname
    else:
       fn = name
    
    if fn:
        ret = re.search('(.{0,14}).*', fn)
        fn = ret.group(1)

    
    ret = revRe.search(text)
    year = ret.group(1)
    month = ret.group(2)
    day = ret.group(3)

    return PbEntry(fn, tel, year, month, day)


entries = []

files = os.listdir('/tmp/sync')
for file in files:
    fh = open('/tmp/sync/%s' % file, 'r')
    content = fh.read()
    entry = get_entry_from_text(content)
    entries.append(entry)

entries = sorted(entries, cmp=cmp_pb)

i = 1
for entry in entries:
    print '%d,"%s","%s"' % (i, entry.tel, entry.name)
    i = i + 1
    if i > MAX_CONTACTS:
        break

I had my share of incompatibilities between mobile phones, computers and other devices. Fortunately most of devices being sold today use open communication protocols for sharing of data (and other stuff). Too bad people had to put so much energy into reverse engineering proprietary solutions in the past. Just ranting about this vendor lock-in could be spread on quite a few pages. Imagine having 300+ contacts and calendar information in your phone of brand X. When you are buying your new phone, you would be able to synchronize your data only if you bought new phone also from brand X. Would that affect your decision? It sure would affect mine.

Now I have a choice. After fixing my old N73 I will start looking into new phone. So far HTC Hero looks pretty cool and reviews are not half bad.