Work Reloaded

Java Memory Optimization & Server Resources

2012-02-06T12:43:28Z

I’ve learned a bit about Java and also about Linux this past week-end. I finally found out, what it was that kept me from going forward with the deployment of my web application. I kept getting OOM errors when starting up a second java process on my virtual server. Even stackoverflow users, thorough and diligent as they are, could not provide me with a solution. (In the end it turned out however that one had provided me with the right information, he just could not know that I was unable to apply it, due to the configuration of the server.)

To begin here are a few quick notes on java memory usage and two nice articles: common errors and an overview of JVM Options:

Typically – with past projects – we had the problem, that java did not reserve enough memory for the application. So we would try to give it a larger heap. One does this with the Xmx and Xms options. These two are the most commonly used VM options:

-Xmx sets the maximum heap size, but it does not limit the amount of memory the VM may use
-Xms sets the initial heap size

So you might want to start your vm with these java options:

export JAVA_OPTS="-Xmx64M -Xms64M"

This time however I had a new problem: I seemed to not have enough memory on my virtual server. I had been sure that 2GB was sufficient to run a small java web application with a few users for my first alpha test (and it is, but not as smooth as I want it to).

With Xmx and Xms you can not limit the VM. When starting up tomcat the Sun VM would immediately grab 300MB of physical memory. After some searching I found another helpful parameter, some of you might remember this one from trying to keep Eclipse from devouring your development system:

-XX:MaxPermSize=128m

Because the other options only limit the heap size (what your programs use). This one actually limits the VM itself. After adding this to my JAVA_OPTS I found that the VM would finally grab only about 190M (sum of heap size and perm size).

I was however still having trouble running a second java process. Not even java -version would run without throwing my favorite error:

java.lang.OutOfMemoryError: Cannot create GC thread. Out of system resources.

This should have told me. I am writing this article because with that search phrase I did not find a solution. It took me many hours of debugging to arrive at the point where I understood what was wrong and could finally approve the correct answer on stackoverflow.

I was confused because top was telling me there was at least 1GB of memory available (this being a virtual server that would not necessarily be true I guess but since the problem appeared consistently I am quite sure the physical memory is actually available). Even when java was down to <200MB and there was more than 1GB of memory available I kept getting OOM errors (btw the open Jdk used a lot less memory than the sunJdk in my tests). Finally I managed to produce OOM errors even from simple shell commands and that is what led me to the right solution. OOM for me meant “out of (physical) memory”. I never looked at the actual error message: “could not create thread”. Because there are other resources that may not be available that are also counted among the memory resources.

On most systems there is a neat little command that shows you a lot about your systems resource configuration:

ulimit -a

On my system ulimit told me that the number of user processes aka threads was unlimited. Well and that is a lie. Because it is not. The virtualization software enforces a hard limit from outside the container. You cannot necessarily see that inside the container. In my case I had access to the “Parallels Power Panel” and finally saw under Resources -> Memory -> Primary System Parameters that the value numproc had a hard limit of 96. If you have a real server the answers on stackoverflow should provide you with helpful hints how to fix this, if you have a virtual server you can call your hosting provider and ask if 96 was a mistake by chance because it is a ridiculously low value. (I also should have seen the Ressource Alerts for numproc, but I didn’t … as I said I learned a lot this past week-end)

Run top and toggle to thread view with shift-h and it shows you which of your applications uses up how many of your threads. Here’s a quick overview what was going on on my system:

named ~10 threads
mysql ~10 threads
apache ~10 threads
tomcat/java ~30 threads
misc resource eaters: cron, sshd, init, syslogd
each login with ssh and subsequent sudo su eats up your processes

So the more I was debugging the worse my problems got. When I am not logged in and I just started my services my system is using about 50-60 threads just doing nothing. There is no traffic to speak of, as I am not yet hosting anything.

It cannot even run jenkins because java seems to need an awful lot of threads when starting up and jenkins of course needs to run mvn or ant to build and so starts up a second java vm which is really expensive as I now know.

Most of the programs above could possibly be made to use fewer threads for a small test server like this. Which is what I will do for now and then I’ll be thinking about different hosting options.

End of story: This morning I called the support and asked if by chance the value for numproc was an error, since I could not find the small print that might have told me so before ordering such a crippled (but indeed rather cheap) server. But alas I was informed that this was as it should be and the server was meant for testing, I should probably go for an upgrade. I totally understand how companies have to make money. You need a low barrier of entry to get people to sign-up for something cheap and then make them upgrade to more expensive servers quickly. Well done, in that aspect! Most people will never find out what it is that is not working. However with this ridiculously low limit I have lost my trust in that the other virtual servers would be adequate. I am not well-versed enough in server hosting to know in advance all possible limits that I would need to check before ordering one of the more powerful servers and when running a few calculations in the end AWS might be cheaper (ulimit -a tells me they allow 1024 threads on the 1-year-free micro instance). Another alternative might be a root server over which I have full control.

Automate your Android builds comfortably

2011-10-06T10:01:26Z

Or: How to set up your Android app’s Multi Module Maven build as Eclipse project.

This week I spend a few hours searching for a solution for automated Android builds with Maven. I want to launch my first paid app with minimum fuss. There is a free version of the app on the market already and now I want to use a multi module Maven project to manage both a free and a paid version of the same app comfortably. I am a lazy programmer after all.

Since it was not quite the trivial endeavour I had hoped – but of course not expected – it to be, I decided to write this up as an article for myself and of course for all the other Android developers out there who don’t want to waste time setting up projects when they could be gold-plating their apps

You are probably only reading this after you have already created your app and have everything running from Eclipse. And it works, too, up to a certain point. I only started on automated builds when it started hurting me to not have them. So I won’t cover setting up a fresh project. You should have some working knowledge of Maven and Android. I will assume you have both setup Android and Maven projects in the past.

This article assumes that you have an Android project without an automated build somewhere. If you don’t have one just create a new Android project in Eclipse, this will give you a HelloWorld app and everything that is necessary to follow the instructions in this article. Please don’t recycle your project, you can easily copy over your classes and resources later. Start from scratch with the setup until you have understood what is going on – there’ll be less crying if you don’t break your working project along the way. If you really want to do this on an existing project, tag now!

Step 0: Install all necessary Eclipse plugins.

These being: m2eclipse, m2e-android plugin and of course android. You may have to modify your .m2/settings.xml. I added the following lines, so I can call mvn android:deploy on the command line:

<pluginGroups>
  <pluginGroup>com.jayway.maven.plugins.android.generation2</pluginGroup>
</pluginGroups>

What didn’t work: I also experimented with using the existing android archetypes by adding them to .m2/archetype-catalog.xml. Those didn’t quite work out the way I wanted to.
What worked for me: So I reverted to a mostly manual set-up, that I will describe in the following steps.

Step 1: Setup the root project & the build

Create a fresh Maven project in Eclipse.
Select pom-root as archetype.
Set a GroupId and an ArtifactId.
You will get an empty project containing only the most basic of poms.
Add properties to pom.xml
Add (Android) dependencies to pom.xml
Configure build plugins
Add modules

First you are going to modify the root pom.xml to include most things you will need for your build.

You will need a few properties but not all those I have listed (for example the encoding properties). If you have questions regarding these, please don’t hesitate to ask, I’ll respond in the comments.

  <properties>
        <android.sdk.path>/opt/android/</android.sdk.path>
        <rt.jar.path>${java.home}/jre/lib/rt.jar</rt.jar.path>
        <jsse.jar.path>${java.home}/jre/lib/jsse.jar</jsse.jar.path>
        <android.version>2.2.1</android.version>
        <android.platform>7</android.platform>
        <android.emulator.name>ExampleEmulatorName</android.emulator.name>
        <maven.android.plugin.version>2.9.0-beta-5</maven.android.plugin.version>
        <!-- not necessary but nice -->
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
    </properties>

Dependencies are the easy part, these are the ones you will need. Of course you may need additional ones, it should still be easy

	<dependencies>
		<dependency>
			<groupId>com.google.android</groupId>
			<artifactId>android</artifactId>
			<version>${android.version}</version>
			<scope>provided</scope>
		</dependency>
		<dependency>
			<groupId>com.google.android</groupId>
			<artifactId>android-test</artifactId>
			<version>${android.version}</version>
			<scope>provided</scope>
		</dependency>
	</dependencies>

The next part is more complicated, it contains the build. My build contains two important plugins:

the maven-android-plugin
the maven-jarsigner-plugin

I broke the build in three parts to explain what it does (that’s what I was missing for most of the existing articles). The first part is easy: I am configuring the maven-compiler-plugin to use my favoured version of java.

<build>
  <pluginManagement>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
          <configuration>
          <source>1.6</source>
          <target>1.6</target>
        </configuration>
      </plugin>
      ...

Please note that I am placing the plugins inside pluginManagement tags, this means I am only configuring the plugins here. If my modules need any of them they will have to be added to the build of the respective module. I am doing this because I may have different builds for my modules.

I need to rant about this for a moment: it took me a while to figure out the difference between using the pluginManagement vs putting plugins directly in build. Nobody ever tells you about it and it seems to be universal knowledge except most people I know copy without knowing what this all means, which leads to weird builds where a lot of default values are re-configured for example. People just keep fiddling around until it “works somehow”. This leads to unstable code. So because we are all lazy copy-pasters, I am trying to explain some the stuff that everybody seems to think is self-evident – most often it is not. This also helps me learn more about Maven while writing this If you have any questions regarding this article please ask. It can only be improved by the questions people ask!

So that needed to be said.

The second part hopefully is more interesting. This is the android maven plugin configuration:

      ...
      <plugin>
        <groupId>com.jayway.maven.plugins.android.generation2</groupId>
        <artifactId>maven-android-plugin</artifactId>
        <version>${maven.android.plugin.version}</version>
        <configuration>
          <sdk><platform>${android.platform}</platform></sdk>
          <emulator>
            <avd>${android.emulator.name}</avd>
          </emulator>
        </configuration>
	<executions>
	  <execution>
	    <id>alignApk</id>
	    <phase>package</phase>
	    <goals>
	      <goal>zipalign</goal>
	    </goals>
	  </execution>
          <execution>
            <goals>
              <goal>deploy</deploy>
            </goals>
          </execution>
	</executions>
        <extensions>true</extensions>
      </plugin>
      ...

I added the zipalign because either the market or the emulator complains, if you don’t optimize the apk this way. I am also configuring the name of the emulator I use. This is something that will differ for your system because whoever setup the android environment will have created an avd and given it a name. Maybe it is possible to create an avd with the maven plugin to have a standardized environment. If not this should probably go inside an environment specific (or personal) properties file (I am not covering this in this article).

Maven extensions (see last line of configuration) are explained here. It seems this is needed so the plugin will actually execute as part of the build. If you delete this line the modules will complain they don’t know how to build apks. (As you can see I don’t quite get it. So please enlighten me if you do).

There are many more configuration options for this plugin. I am trying to create a minimal version that works. The rest of the setup is complicated enough.

The final plugin is needed, because I want to be able to automate uploading to the market: the jar signer. This is not android specific, nevertheless here’s a sample configuration. You should probably add the signing properties to a property file, you can also add them to the build like my lazy self did (not in this example however, you can do this as an easy exercise at home).

      ...
      <plugin>
        <artifactId>maven-jarsigner-plugin</artifactId>
        <version>1.2</version>
	<configuration>
	  <removeExistingSignatures>true</removeExistingSignatures>
	  <archiveDirectory />
	  <includes><include>target/*.apk</include></includes>
          <archive>${project.build.directory}/${project.build.finalName}.${project.packaging}</archive>
          <certs>true</certs>
          <keystore>${sign.keystore}</keystore>
          <alias>${sign.alias}</alias>
          <storepass>${sign.storepass}</storepass>
          <keypass>${sign.keypass}</keypass>
	</configuration>
	<executions>
	  <execution>
	    <id>signing</id>
	    <goals>
	      <goal>sign</goal>
  	    </goals>
	    <phase>package</phase>
	  </execution>
	</executions>
      </plugin>
    </plugins>
  </pluginManagement>
</build>

This is what’s inside the root project. I also placed the keystore there and all the graphics that have accumulated for the project: screenshots, inkscape files for icons etc.

Step 2: Creating the basis with a library project

My goal is to have two apps that can be deployed: a free version and a paid version. The common stuff will be placed in a so-called Android Library project. What else I do with these, how they differ and how I manage activating the paid functions is not covered here, this is purely the setup. Library Projects are special Android projects that will be imported into other Android projects. They differ from a simple Java module or jar-file in that they are imported as source: both the Java code and all the resources! Also you can override resources from a Library Project in your app project but the resources have to have the same name and need to be placed in the same filename – this is due to the way resources are packed into your apk (you might remember R.java?).

To create the library project simply create a Maven module by right-clicking on your parent project and selecting “New Maven Module Project” from the “Maven” sub menu. Skip archetype selection, we’ll do this manually. Packaging is ‘apklib’. This will create a very basic module that starts off with an error message “Unknown packaging: apklib”. So the first thing you need to do is to add the maven-android-plugin to the build of the library project:

<build>
  <plugins>
    <plugin>
      <groupId>com.jayway.maven.plugins.android.generation2</groupId>
      <artifactId>maven-android-plugin</artifactId>
        <configuration>
          <attachSources>true</attachSources>
        </configuration>
    </plugin>
   </plugins>
</build>

The error should be gone. Before you can add (by copying or developing) your code and resources and of course AndroidManifest.xml you should edit two files .project and .classpath. Because right now this can be a maven build as much as you want to, but it is – at least according to Eclipse – not an Android project, actually it is not even a Java project because we skipped archetype selection.

So here’s a sample .project:

<?xml version="1.0" encoding="UTF-8"?>
<projectDescription>
	<name>example-lib</name>
	<comment></comment>
	<projects>
	</projects>
	<buildSpec>
                <buildCommand>
		<name>com.android.ide.eclipse.adt.ResourceManagerBuilder</name>
			<arguments>
			</arguments>
		</buildCommand>
		<buildCommand>
			<name>com.android.ide.eclipse.adt.PreCompilerBuilder</name>
			<arguments>
			</arguments>
		</buildCommand>
                <buildCommand>
			<name>org.eclipse.jdt.core.javabuilder</name>
			<arguments>
			</arguments>
		</buildCommand>
		<buildCommand>
			<name>org.maven.ide.eclipse.maven2Builder</name>
			<arguments>
			</arguments>
		</buildCommand>
	</buildSpec>
	<natures>
		<nature>com.android.ide.eclipse.adt.AndroidNature</nature>
		<nature>org.eclipse.jdt.core.javanature</nature>
		<nature>org.maven.ide.eclipse.maven2Nature</nature>
	</natures>
</projectDescription>

You can probably configure all this somewhere in Eclipse. It is much faster and less error prone to edit the file. You need the natures to get all the options for the respective project type. I’ve left in the Java builder but I took out the Android APK builder but left in the ResourceManagerBuilder and the PreCompilerBuilder. This file is the same for all your Android modules – except of course for the name. This also should be checked into your version control and be carefully monitored for unplanned changes. The same goes for .classpath, which looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<classpath>
	<classpathentry kind="src" path="src/main/java"/>
	<classpathentry kind="con" path="org.eclipse.jdt.launching.JRE_CONTAINER"/>
	<classpathentry kind="con" path="org.maven.ide.eclipse.MAVEN2_CLASSPATH_CONTAINER"/>
	<classpathentry kind="con" path="com.android.ide.eclipse.adt.ANDROID_FRAMEWORK"/>
	<classpathentry kind="src" path="gen">
		<attributes>
			<attribute name="optional" value="true"/>
		</attributes>
	</classpathentry>
	<classpathentry kind="output" path="target/classes"/>
</classpath>

This is a compromise: the Eclipse Android plugin really wants to use the ‘gen/’ folder while maven really wants to generate everything into target. You cannot configure either the Eclipse Android plugin nor the Maven Android plugin to use different folders (at least not that I found out). So just keep both. To enable you to redeploy changes you made in the library project when running the app modules you need to add the library project to all app modules with eclipse, this modifies your .project file.

Once you have added the Android nature, new errors will probably crop up: you have to create or copy a default.properties file. Now you also need your Android code. If you don’t have a finished project or work-in-progress, you can simply create a new Android project in your workspace and copy everything (java code, res/ directory and default.properties, AndroidManifest.xml) from there. This is a dumb work-around but I did not get any of the Android Maven archetypes to work like I wanted them to (also these are covered elsewhere and I wanted to understand what was really needed for once) so I couldn’t create a Maven Android module in one go. Luckily we don’t need to set up projects quite that often.

Copy over your code and check if the build works. My example project builds now, both in Eclipse and on the command line. The first being for the developer’s comfort, the latter being the whole point of build automation.

You can’t run library projects on the emulator. Don’t try. First we need an application module.

Step 3: Creating the application module

Either follow the first few steps (everything but copying over your existing code and resources!) as when creating the module for the library project or copy the skeleton of the library project and add the module manually to the parent project. If you copy everything don’t forget to remove the code or resources, you don’t need them in the app project.

Add the new module to the parent project:

<modules>
  <module>example-lib</module>
  <module>example-app</module>
</modules>

Then right-click your project, select “Import” and then “Import Existing Maven Projects” your module should show up. The biggest difference is that you will need to change the packaging to ‘apk’. Here’s the complete pom.xml anyway:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<parent>
		<artifactId>example</artifactId>
		<groupId>de.delusions.example</groupId>
		<version>0.0.1-SNAPSHOT</version>
	</parent>
	<groupId>de.delusions.example</groupId>
	<artifactId>example-app</artifactId>
	<version>0.0.1-SNAPSHOT</version>
	<packaging>apk</packaging>
	<name>Example App Project</name>
	<description>the actual app</description>

    <dependencies>
        <dependency>
            <groupId>de.delusions.example</groupId>
            <artifactId>example-lib</artifactId>
            <version>${project.version}</version>
            <type>apklib</type>
        </dependency>
    </dependencies>
    <properties>
       <parent.dir>${project.basedir}/../</parent.dir>
    </properties>

	<build>
		<plugins>
			<plugin>
				<groupId>com.jayway.maven.plugins.android.generation2</groupId>
				<artifactId>maven-android-plugin</artifactId>
			</plugin>
			<plugin>
                <artifactId>maven-jarsigner-plugin</artifactId>
                <version>1.2</version>
            </plugin>
		</plugins>

	</build>

</project>

I highlighted a few lines here.

Line 12: the different packaging, because that’s one of the apps you will want to deploy.
Line 19: the dependency on the library project
Line 25 and 35: the property configures the directory of the parent project where we placed the key store that the added jarsigner plugin needs.

Step 4: Cleaning up common errors

So there might be some more cleanup to do before you can actually run your new multi module maven app project on the emulator. Here’s a few mistakes I made when setting up the example you can download below:

Missing dependency: If you created your project like me, you will now spend about an hour trying to figure out the next error that happens, because Maven doesn’t find your library project as dependency for your app project. If you made the same mistake as I did, you messed up package names in your poms because in the middle of everything you decided on a new package name. Be consistent with package names, is all I can say.

Library projects cannot be run on emulator: because of copying the module somehow the app module was also marked as library project. This can easily be fixed by right-clicking and editing the project properties, the Android menu opens a dialog that will let you uncheck “isLibrary” easily.

ClassNotFoundException on Emulator: Another typical error. You need to fully qualify all the activities and services etc. in the AndroidManifest.xml of your app project now. Because you will probably want to change the package of your project to something different from the library. For example I have two app modules. One has the package name “de.delusions.example.free” and the other “de.delusions.example.paid”. This way I can deploy both apps at the same time with mostly the same code and resources from the library project.

“No need to reinstall”: I was able to run the apk module on the emulator from eclipse however it didn’t re-install the apk when I changed the library project. The solution here is to reference the library module as library project in Eclipse (go to Project > Properties > Android)

“R.java was modified manually! Reverting to generated version!”. This came up after I re-added the Android builders to my .project file (see above). A command line cleanup of the project and an Eclipse restart fixed this. This seems to happen on a variety of occasions (or so the net suggests), that solution may not work for you. Of course you shouldn’t be modifiying R.java manually!

Building workspace hangs I experienced some kind of build cycle that would run forever while I had both the Android apk builder and the Maven builder enabled. Removing the apk builder fixed this.

Hopefully you are now able to select “Run As Android Application” and you can see your app running on the emulator. Modify your library and see if the changes are redeployed on a second run.

Warning: The example is not quite up to date currently. I updated the article with my latest findings but not yet the download. I created a download with the code for the example I used in this article: Download Example.

Step 5: Loose ends

Removing the android builders from the .project file of Eclipse causes a lot of havoc with the way I used to test my app. Right now I can’t quite do it from Eclipse. I have to trigger a maven build to install the apk and then I need to start the app on the emulator manually (there’s supposedly a feature request for starting the app automatically but I haven’t found if it was added to the maven plugin). I’ve experimented a bit with linking projects as libraries (didn’t work like I wanted it). I’ve experimented with re-adding the builders. I’ve not quite solved running everything smoothly from Eclipse. I will be working on that and add my findings to this article.

With the re-added ResourceManagerBuilder and PreCompilerBuilder in .project and by adding the library module as library project in Eclipse, for now I seem to be able to run the app modules from Eclipse and Eclipse redeploys the app when I have change the library project. No need to manually run Maven and no more weird hangups.

BUT .. there is always a but. Working with it I am noticing that only changes in the Java code are used for redeployment. If I only change layouts it just doesn’t deploy the apk to the emulator until I also change a bit of Java code.

I have not yet developed any code exclusively for one of my apk modules. So I am waiting how that turns out. What I did so far was create a custom icon for the paid apk module and changing a few of the string resources.

For a bit of help and an example on a working Android Maven project see the Android-Maven-Eclipse Integration plugins website.

Step 6: Feedback

I hope this helps other developers. Like all writers I love feedback. So if you find this article useful leave me a comment or a question and I will gladly try to answer questions regarding this topic – if I can. If I made mistakes somewhere please point them out, so I can correct them.

Thanks for reading!

A fairy tale of the Wicket and the Maven

2011-07-11T09:09:02Z

Lately I’ve been working on several Apache Wicket projects. Wicket is quite an intuitive and well-rounded web-application framework. Well intuitive for someone who has been programming Java for a long time. Since for the moment I am back to using Real Java™ (as opposed to Android), I am also happily back to using Maven and that’s what this article is mostly about.

When starting out with a new technology I think it’s best to jump in and get started on a project. Because nothing teaches as fast as doing and nothing leaves the same lasting impressions as the mistakes you make.

The fastest way to introduce yourself to wicket is the Wicket Quickstart project. Here’s the commandline (you could also do the same by creating a project in Eclipse but I would have to provide screenshots and much more explanation, when the original page – see link above – does the same quite well).

Say hello to Wicket

mvn archetype:generate -DarchetypeGroupId=org.apache.wicket -DarchetypeArtifactId=wicket-archetype-quickstart -DarchetypeVersion=1.5-RC5.1 -DgroupId=com.workreloaded.wicket -DartifactId=profiles -DarchetypeRepository=https://repository.apache.org/ -DinteractiveMode=false

So now you have this fresh new wicket project. You programmed a few counters, a form and integrated a bit of Hibernate. You used some of the shiny Ajax features. Maybe you even downloaded the ebook Version of Wicket in Action and started on your own version of the Cheesr example. You have probably encountered Models and found some documentation and examples. You have fiddled around with CSS layouts for a while. You have run your application countless times with the integrated Jetty server by just running the provided Start class (now isn’t that neat?). You have decided it’s time to tackle something bigger: a real project. One of those ideas you always wanted to …

The Demonstration Paradox

It’s a bit later. You have done it. You have finished a first version of your idea, a prototype. You want to show the world your greatness and what you have programmed these last few days (or weeks). You want to deploy! You install Tomcat or Jetty on your server and just like that everything works out right from the start – it’s like a fairy tale (or this would be a different article on “how to configure an application server”).

So you copy your nice new mygreatidea-0.0.1-SNAPSHOT.war to the webapps folder of your application server. You even gathered a crowd of soon-to-be-impressed friends, followers and circles. Excitedly you click on the link to the app to present it: http://mydomain.com:8080/mygreatidea/. “Lo and behold my …” but wait, didn’t you forget something? You forgot the magic Wand of Making All Installation Locations The Same and now it’s too late: the demonstration paradox is staring back at you and whoever is looking and beholding.

I wonder which of the classics hit first: no styles, no database or is it only the debug mode or is your application just slow because of the huge amount of logging your writing out? (Note: there’s a place in the persistence.xml that calms down Hibernate – a lot!)

Since this is a fairy tale we can easily travel back in time and correct those mistakes. But what were the mistakes? There was basically just one mistake that every developer makes more than once (this particular mistake seems to have some difficulty leaving enough of an impression for us to learn from even if the results can by quite catastrophic): It is the “But it works on my computer!” effect. Corollary: “But it works on my computer and the development system!”

Not all installation locations are created equal, some are for example more restrictive than others. Because this problem exists everywhere, every self-respecting build tool offers a solution. The solution that Maven provides is called “profiles”. You may have used them but have you introduced them from scratch to a project? It’s quite simple actually, most of the work is done in the pom.xml.

Prolific Profiles

First you get to add this little snippet to your pom.xml or your parent-pom or even your .m2/settings.xml.

<profiles>
  <profile>
    <id>dev</id>
    <activation>
      <activeByDefault>true</activeByDefault>
    </activation>
    <properties>
      <build.profile.id>dev</build.profile.id>
    </properties>
  </profile>
  <profile>
    <id>prod</id>
    <properties>
      <build.profile.id>prod</build.profile.id>
    </properties>
    <activation>
      <property>
        <name>profile</name>
        <value>prod</value>
      </property>
    </activation>
  </profile>
</profiles>

This creates 2 profiles, a development profile and a production profile. You can have as many different profiles as you want. For now each profile has a single property: an id that can be used elsewhere in the build. Also important is the activation tag: it determines when this profile is going to be used. In this snippet we have two variants: automatic activation for the default profile and activation by a property: mvn clean install -Dprofile=prod. There are even more types of activation for example you could automatically select the production profile when you want to create a release (I’ll leave this to you to find out as a small exercise).

Next you need to collect the profile specific information. Typical examples for profile specific information can include but are not necessarily limited to:

database configuration
logging configuration
debug modes
path information
different implementations in your code

I am sure you can think of some more. So to allow you to control all these different configurations it would be wise to have a central property file that collects all those differences between your different deployments. Let’s see what the file config.properties could look like for the development profile:

database.name=mygreatidea
database.user=me
database.password=mypassword
logging.level=DEBUG
logging.path=mygreatidea/log
wicket.mode=development

For other profiles you would have the same properties but of course different values. You need to place the different properties in files in such a way that maven can find them by the profile id that we defined in the first xml-snippet. Most commonly this is done in profile specific directories:
mygreatidea\profiles\dev\config.properties mygreatidea\profiles\prod\config.properties

Now you must make this information known to the build. This is again done in the pom.xml by simply adding a so-called filter to the build.

<build>
  <filters>
    <filter>profiles/${build.profile.id}/config.properties</filter>
  </filters>
  <resources>
    <resource>
      <filtering>true</filtering>
      <directory>src/main/resources</directory>
    </resource>
    <resource>
      <filtering>true</filtering>
      <directory>src/main/webapp</directory>
    </resource>
  </resources>
</build>

What this does is quite simple: it will apply this filter to all files for which filtering is true. Applying the filter means quite simply substituting any properties (don’t worry I’ll get to that) found in the files in src/main/resources (and src/main/webapp) with the profile specific properties from the config.properties file. In a typical Wicket project you will find both the log4j.properties and the persistence.xml in that resources folder hierarchy. Adding something as resource also causes Maven to copy those files to the target folder.

In Wicket projects the files in src/main/webapp are not normally filtered because the war plugin is by (Wicket) default configured to look for them in the src folder. However the easiest way to deactivate development mode in wicket is by adding the information to the web.xml config file. So I have added webapps as resources to be copied to the target folder. (You will also have to modify the Start class, that you use for local testing. It also needs the filtered files, just point it to target instead of src/ and you’ll be fine.)

In the final step add this to your web.xml

  <context-param>
    <param-name>configuration</param-name>
    <param-value>${wicket.mode}</param-value>
  </context-param>

There are two important things to note in this snippet: first it shows how to turn on or off development mode in Wicket (allowed values are development or deployment by the way). Secondly and most importantly it demonstrates how the properties in your profile can be used in the filtered files. It should now be easy to add your properties to the logging configuration and your database configuration.

You can even use this mechanism to generate profile specific java code (of course that is a bit more complicated unless you only substitute string literals).

After you finished replacing all the profile specific values with properties during your timetravel in the past, you go back to the future just to the point before the deployment and you dazzle the crowd with your great idea™

I think profiles are a very important addition to any project because you only have to think about each difference once, implement it and forget about it. And if you do encounter the demonstration paradox it’s quite easy to not repeat the same mistake because there’s simple way to fix such problems!

Thanks for reading. I hope you enjoyed this little tale. It’s the week-end and I am entitled to have some fun

How to use the Android Camera

2011-06-07T14:23:36Z

I have now spent quite a few hours programming for my next app which needs to take pictures. I did not want to use the camera app that’s already there for several reasons. So I need to programm my own. The API is not all that difficult to understand. Here’s the basic version of how to talk to the camera:

    //creating the camera object
    Camera camera = Camera.open();
    camera.setPreviewDisplay(getSurfaceHolder());
    final Camera.Parameters parameters = findBestParameters(w, h);
    camera.setParameters(parameters);
    camera.startPreview();
    final PictureCallback callback = new PictureCallback() {
        @Override
        public void onPictureTaken(byte[] data, Camera camera) {
            try {
                //async task for storing the photo
                new SavePhotoTask(CameraView.this.ctx, data).execute();
            } catch (final SavePhotoException e) {
                //some exceptionhandling
            }
        }
    };
    //no need for shutter callback or raw callback, jpg callback is enough
    camera.takePicture(null, null, callback);

Doesn’t sound so hard? It only took me hours to get it right. Because I never got to the onPictureTaken(...) method. The callback never happend! I made the mistake wanting to do something else after taking the picture – who wouldn’t?

There are several issues with that. It takes the phone quite a while (~5 seconds) to actually take that picture once the camera.takePicture(...) method is called. Since the method is an asynchronous callback you can do all kinds of fun things(tm), while waiting for the result. For example I managed to execute some very important piece of code:

public void clearCamera() {
        if (this.camera != null) {
            try {
                this.camera.stopPreview();
                this.camera.release();
                this.camera = null;
            } catch (final Exception e) {
                //handle some exceptions
            }
        }
    }

This should be called by onPause() of your activity (and onResume() you should of course re-open the camera!). So if for some reason while you do the fun things(tm) I mentionned above your activity decides to take a break, the callback never happens! Ouch.

So how did I get my activity to wait for the callback and then do something?

In the first code sample you can see that the actual picture storing takes place in a class called SavePhotoTask. This class is an asynchronous task that offers the functionality I needed. Please notice that since the first code sample some things changed and the task now also implements the callback interface:

public class SavePhotoTask
    extends AsyncTask<Boolean, Void, File>
    implements PictureCallback {

    //fill in the rest as a light exercise at home
    private final ProgressDialog dialog;

    @Override
    protected void onPreExecute() {
        this.dialog.setMessage(String.format("Saving Picture"));
        this.dialog.show();
    }

    @Override
    protected File doInBackground(Boolean... params) {
        this.camera.takePicture(null, null, this);
        return this.spot.getPhoto();
    }

    @Override
    protected void onPostExecute(final File success) {
        if (this.dialog.isShowing()) {
            this.dialog.dismiss();
        }
        if (success != null && success.exists()) {
            DialogFactory.keepPhotoDialog(this.ctx, this.photo).show();
        } else {
            Toast.makeText(this.ctx, "You failed!", Toast.LENGTH_SHORT).show();
        }
    }

    @Override
    public void onPictureTaken(byte[] data, Camera camera) {
        try {
            //I guess you know how to save a file
            saveFile(data);
        } catch (final Exception e) {
            //capture flying shit
        }
    }
}

So now I can execute my picture taking as an asynchronous task, that gets a callback to onPostExecute(...) when it is finished. I can then open another dialog prompting the user to do something. I introduced that additional dialog to get a break before leaving the original activity (remember how leaving will close the camera and we should not do it before the picture is stored safely on the sdcard).

So to keep it complete, here’s the final piece of code: the dialog.

    public static AlertDialog keepPhotoDialog(final Activity a, final MyPhoto photo) {
        final AlertDialog.Builder builder = new AlertDialog.Builder(a);
        builder.setMessage("Keep photo?");
        builder.setCancelable(false);
        builder.setPositiveButton("yes", new DialogInterface.OnClickListener() {
            @Override
            public void onClick(DialogInterface dialog, int id) {
                final Intent i = new Intent(a, DoFunThingsActivity.class);
                i.putExtra(DoFunThingsActivity.PHOTO, photo);
                a.startActivity(i);
            }
        });
        builder.setNegativeButton("no", new DialogInterface.OnClickListener() {
            @Override
            public void onClick(DialogInterface dialog, int id) {
                final File file = new File(photo.getFilename());
                if (file.exists()) {
                    file.delete();
                }
                dialog.dismiss();
                //you might want to call startPreview() again at this point
            }
        });
        return builder.create();
    }

So here’s it, I chained two dialogs to take a picture and do fun things(tm) with it later without having to start handle threads and all that stuff.

Any questions?

Open Source Android: DroidWeight

2011-06-07T07:48:50Z

A few minutes ago I made my first commit on the new svn repository for DroidWeight on google code: http://code.google.com/p/droidweight/.

I had a couple of requests to open source it. Since I never planned on making money from the app anyway – after all it is my learning app – I thought I could just as well share. Maybe I’ll find a couple of contributors.

So if you would like access to the repository and the google group just drop me a note via mail or the contact form and I’ll see to it.

Right now the documentation is practically non-existent, so you should be brave enough to find your own way through the code.

I’ll still be publishing releases as before only now everyone can see the code and take it to make their own stuff with it, laugh about dumb beginner’s errors I made or contribute patches.

This is the first time I am open sourcing a project of mine and I am curious how it will turn out

PS: the app is close to 10.000 downloads on the market!

Call for links

2011-05-16T09:45:01Z

I am looking for pages, forums, communities and blogs on all things scrum or agile. I will also happily take a look at general software development processes and methodology sources. I’d be grateful for some good places to start reading.

What are your favorite places to go read up on the latest agile news?

3 Approaches to Change

2011-05-13T12:50:08Z

In Can you change your team? I asked you to think about what you want to change the most about your work. What you want to change dictates which approaches to change is most likely to succeed. The best approach depends largely on how much influence you have on that aspect of your work.

Analyse your problem

Now that you know what you want to change, it is time to start thinking about a solution. Most of the common problems can be sorted into three categories according to the most promising approach to get them fixed:

improve communication
improve transparency
improve team organization

1. Talk to each other

If your problem is within your direct sphere of influence say within your team, it is very likely that improving communication is the way to go. Such problems can be hard to spot, they are subtle because they are so close to home. Communication is all about talking with each other. Many teams are not doing that. There may be regular team meetings in your team. But are you really talking to each other? During your breaks when you talk to each other, are you talking about work?

It is always possible and also quite simple to change the way a team communicates. One of the easiest ways to get massive improvements is a daily stand-up meeting. This is taken from the agile repertoire and those 10 minutes cost no more than a coffee break yet they can provide much insight for a team.

2. Enable others to see your team

Many problems originate outside of your team. Those can be quite obvious because people tend to complain out these much more openly: it is much easier to blame people who are at least one step removed from you in the organization. Usually you do not have the power to tell people from other teams or those higher up in the hierarchy what to do. So this issue is a favorite of those who prefer complaining to doing anything.

For this you another type of communication: Transparency. Transparency is about communicating to people outside your team. It is about showing others the status quo. It is less about fixing a problem than about making it visible to others. If you do not have direct influence on what or who you want to change this indirect method can come in helpful. Showing others facts and numbers how something influences the productivity of a team will have much more impact than any other argument.

3. Get creative and “deal with it”

The last type of issue is hardest to spot and hardest to fix. Sometimes the cause of trouble is so far removed from your sphere of influence that neither improved communication nor transparency are going to be of any help: you simply cannot change the way things are.

“Changing Requirements” can often be such an issue. Showing others, that this is causing trouble for your team is not going to have any effect. The problem may come from so far higher up the corporate food chain, that there is simply noone you can influence. This is where the right mindset plays an important role: accept the problem as fact and get creative in dealing with it. Take it as a challenge to you and your team to organize yourself in a way that you are able to work with the problem rather than against it.

For this last category I have an example: The team I was working in had trouble with estimates and project finishing dates. The official process had a rule that in the beginning of a project the team leader should give a rough estimate how long a project would take. Later then the estimate was supposed to be refined. The problem was that the refined estimate was never taken into account for the project plan because the rough estimates were entered into the company roadmap and were never changed later. Once this was recognized by the team we skipped the rough estimate part and instead handed in the refined estimate very early in the project lifecycle. Those estimates could still be wrong but at least one of the biggest sources of wrong estimates was eliminated in that way.

Compromise

It is not easy to find such solutions. You need to have an eye for it which only comes through practice. Also it has proven to be one of the most convincing arguments to put forward: “I know I cannot change the facts of how the estimates are taken into account, I only want to change how we deal with it!” Offering to change your side in order to solve a problem clearly shows your commitment to improve the work process and may help finding a compromise that everybody profits from.

Can you change your team?

2011-05-15T06:25:51Z

Yes you can!

The words may have gotten a bit old by now. I still say: you won’t know until you have tried and succeeded.

Are you working in one of those teams, that I described in my last article? It can be hard to imagine, that you are able to facilitate any change regardless of your role in the team. Are you the team leader and everybody thinks you should be the one to change everything? Or are you a team member and you think you do not have the power to change anything? You all have your excuses for not even trying:

Team Leader: “I have these great ideas but I have no idea how to get the team to accept them, they are too stubborn and they hate having to change.“

Team Member: “I see all the possible improvements that could be made and I really want to change the way we work, but I cannot tell anyone what to do. I just don’t have the authority!”

Even when every person on the team knows that something should be done, it does not mean that anything is going to improve by itself! Toss out the excuses and imagine that you can change the way things are: because you can!

The question is how?

People are loath to change. We all know that. Who has not watched at least one if not several attempts to improve the way a team works and see them fail miserably? Initially people may give new ideas a try. And yet after a few weeks somehow everything is back to the “good” old way.

I think the reason for this happening is, that the majority of people is settled in their routine. Any disturbances are likely going to be viewed as exactly that: a disturbance.

Start small

I propose starting with small changes on the biggest problem. My question at the end of the article was about this: “What is your biggest issue currently?” That should be your starting point. With the most obvious problem you will have the best chances to convince your colleagues or team members to do something about it.

I have asked a few people and got some of the following answers:

“I do not like how people are treated and how they treat each other”
“I hate that we are asked for estimates for the projects, which are then ignored”
“I wish I knew what my colleagues were working on, I hate being a 1-woman-show”
“There’s just too much work.”
“There’s too little work! I don’t have anything to do and I am bored.”
“I don’t know, what my job is, half of the time.”
“I have no idea how far my team got with their work this week.”
“Our projects always take longer than planned for.”
“The requirements are written badly and change all the time”
“I cannot get any work done because there are too many meetings, telephone calls and people coming into my office.”

Take your pick of 1 thing.

What is the smallest step you can take towards solving this problem?

Tomorrow I’ll post the second part that talks about different approaches to achieve change and improve your life at work.

Why every team needs a development process

2011-05-06T17:45:54Z

Imagine a typical software development team. There are 2 types of such teams: those that are based around a single project or those around a single system. People tend to want to differentiate between those types only to find out in the end that they are remarkably similar if you zoom out far enough. Since my experience is with the latter I’ll talk about that. If something does not apply to your team on first glance it just might on second glance. The similarities are enough that I think I can generalize and make my point.

What then do I want to talk about? Three attributes most IT projects share are

the complexity is always greater than anticipated
projects tend to grow over time
they are difficult to manage

Nothing new here probably. There are solutions that help getting a grasp on the complexity and make some parts of IT projects not easy but easier. Those things are called ” development processes” and I have a favorite in that department which you will find out soon if you don’t know already.

So imagine that team I was talking about and all the people and the “stuff” that is involved in it. I’ll put some arbitrary numbers to it to clarify the aspects I am talking about.

the people

1 team leader (if you are unlucky you have a structure that gives a team different leaders with different roles)
5-7 employees spread over a few offices (hopefully in the same building, sometimes however not even in the same country)
3-4 freelancers or “consultants” (everybody is a consultant today)
3 project managers for the different projects the team is currently working on

the “stuff”

1-2 main systems / project
5-6 subsystems / associated projects
3-8 on-going development projects for the system – usually at least 1 more than there are developers
6-12 months of roadmap planned out in mostly less detail, just enough to build up pressure
80% legacy code written by former team members
5 major bugs keeping developers from the work
42 minor bugs keeping developers from resting on their laurels if they ever manage to fix the major ones
operating issues for the running systems
at least 1 big performance problem that keeps nagging at the minds of the developers day and night

and last but definitely not least:

development processes < ε

Sound familiar?

The effects on different aspects of the development work are easily guessed:

4 Effects on Projects

project priorities are unclear & the priorities typically change often
projects rarely get finished on time and sometimes never at all
requirements have to be dropped
more and more unfinished tasks & projects pool into the murky backwater of “later”

4 Effects on Communication

meetings are taking up a lot of time and yet communication is still never enough
it is hard to figure out at any one time what everyone is working on
finding out the state of a project/working package is hard
some tasks are done more than once while others are forgotten

4 Effects on Quality

non-functional requirements (typically those are the ones that are supposed to increase quality) get dropped
technical debt accumulates and the interest to pay for that increases faster than the national debt of Greece as you just do the “quick and easy” solution – “for now”
there is no time to modernize the legacy code
time allocated to new projects is misappropriated to finish up development of old projects

4 Effects on Team Members

the number of context switches for team members increase as projects fight for the attention of the developers
time spent fixing bugs (old & new) increases
everybody is more or less working on “their” stuff, so when someone is sick or on vacation (or quits) their work stalls
more and more conflicts between people pop up as stress increases

I won’t even get started on technical issues like version control, release management, deployment and operating.

Things can work out for years without a process. It is much easier and doesn’t require any knowledge at all to “just get started with the project” instead of defining rules and structures how the team is supposed to work.

I know a project manager who doesn’t think much of development processes. He has about 30 years of experience in the job and has his own methods to get projects finished. His projects work out great because he is simply very good at what he is doing.

However not every project has such a project manager all the time. What development processes – agile or not – give us is a structure beyond each single project. Rules to go by, when we’re not sure how to proceed. A great project manager does the same thing: he gives you rules, structures and helps with decisions. Wouldn’t it be nice to be able to work in the same structured way without having to have a nanny hold your hand all the time?

When the next crisis hits your team (or company) all of a sudden you may have to be more (cost-)efficient. Then the precarious balance on which an unstructured team may have built past successes can quickly turn into a messy “situation” that takes all the fun out of work.

A structured development process does not solve all problems for you. No process is a magic cure-all. A process is about providing you with techniques and tools that sometimes provides you with a solution. At other times it may simply work like an aspirin that contains the pain so you can continue working. Sometimes it is enough to realize the pain is there to be able to do something about it. So even the analysis of your situation may help provide part of the solution.

To adapt itself to a given situation is the most important attribute of any process. A process that fits a team will be accepted much more readily by the team members. That is the reason why I am a fan of agile processes and especially Scrum. You may have a different impression but let me tell you Scrum is great at adapting! As a sidenote: I do believe that it is better to have any process than none so if you have an agile process: great for you! If you have any other process: still great for you!

“What is your biggest issue currently? Where is your team hurting right now?” Usually this is my first question when talking to anyone about processes. If you don’t have a process and are thinking about introducing one then the reason for doing so probably is, that something is not working out. There is usually a very specific need to be adressed and a process should deliver some tool to help tackle that problem.

So what is your biggest issue currently? What aspect of your work is in need of a reload?

Why haven’t you done anything about it, yet?

The Android Dropdown: Spinner

2011-05-03T10:29:57Z

I am currently working on the next release of RpgTracker and I thought I might do another tutorial since I haven’t written one in quite a while.

In Android the dropdown selection widget is called Spinner. Dropdowns are used to select one of a fixed set of values. They are quite helpful to improve usability of apps to make sure only the right values can be chosen.

In this short tutorial you will find two examples with different applications for a spinner:

a spinner used as part of an activity
a spinner in a dialogue

I have taken the code samples from my own applications. If you have any questions regarding the examples please don’t hesitate

First there is an example from my RpgTracker. In this example I use an Enum to represent the selected values in the activity in which the user can edit one of the objects used in the app:

public Enum ValueType {
        DefaultType, NotSoDefaultType, AnotherType
}

public class EditValue extends Activity implements OnItemSelectedListener {

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        //other initialization code
        createTypeDropDown(currentType);
    }

    private void createTypeDropdown(ValueType currentType) {
        final Spinner spinner = (Spinner) findViewById(R.id.type);
        final ArrayAdapter&lt;CharSequence&gt; adapter = ArrayAdapter.createFromResource(this, R.array.type_array,
                android.R.layout.simple_spinner_item);
        adapter.setDropDownViewResource(android.R.layout.simple_spinner_dropdown_item);
        spinner.setAdapter(adapter);
        valueType.setSelection(currentType.ordinal());
        //do not forget the listener, otherwise nothing happens:
        spinner.setOnItemSelectedListener(this);

    }

    @Override
    public void onItemSelected(AdapterView&lt;?&gt; parent, View view, int pos, long id) {
        this.type = ValueType.values()[pos];
    }

    @Override
    public void onNothingSelected(AdapterView parent) {
        this.type = ValueType.Default;
    }

}

So this is the javacode you need to map the spinner to the Enums values. Of course you also need to specify the dropdown / spinner in your layout files and define the values that are shown.

Layout:

<TextView android:text="@string/type_prompt"/>
<Spinner android:id="@+id/type" android:prompt="@string/type_prompt"/>

Values:

<string-array name="type_array">
    <item>Default</item>
    <item>Currency</item>
    <item>Experience</item>
</string-array>

Don’t forget to also add these two snippets to any landscape layout files or to the resources for other languages.

So that’s how you can create a dropdown field and map it to an Enum easily.

You can also use a spinner in a dialogue. The second example is from DroidWeight. I am checking the preferences to see which fields are currently tracked by the user and then offer the user a choice of one of these fields:

    private static AlertDialog getChoice(Context context) {
        final List<String> tracked = UserPreferences.getTrackedTypes(context);
        final String[] items = tracked.toArray(new String[0]);
        final AlertDialog.Builder builder = new AlertDialog.Builder(context);
        builder.setTitle(R.string.menu_showtype);
        builder.setSingleChoiceItems(items, -1, new DialogInterface.OnClickListener() {
            @Override
            public void onClick(DialogInterface dialog, int item) {
                UserPreferences.setDisplayField(context, tracked.get(item));
                dialog.dismiss();
                }
            });
        final AlertDialog alert = builder.create();
        return alert;
    }