Umpteen Choices

Tomcat 7 Service : start in debug mode

2012-01-26T12:02:00.001+00:00

Hit a problem today, doing a really simple tutorial app, so decided to debug Tomcat 7 remotely from Eclipse. I started looking on the web, and almost everything refers to earlier versions of Tomcat.
The ones which did mention running Tomcat 7 didn't appear to be using it as a service, so I had a quick look at the Tomcat site to see if there were any how-to's.

From there, the answer was simple.

From the command line :
H:\Tomcat 7.0\bin\tomcat7 //TS//Tomcat7 -agentlib:jdwp=transport=dt_socket,address=8000,server=y,suspend=n

When started like this, Eclipse was able to connect using it's standard remote debug config.

Multi module build in Maven

2011-09-12T00:26:00.067+01:00

I've been at this one for a couple of days, so decided to post this here for future reference. My aim is to create a jar file, with a sub directory called lib, and have all the 3rd party dependencies in there, e.g

    project.jar
      /lib/spring-beans-3.0.5.RELEASE.jar
      /lib/logback.jar
      /lib/commons-cli-1.1.jar

The project looks like :
adapter
        core
        ftp
        web-service

At this stage, I'm putting demo code together for different sprints, so only wanted some jar files to run from the command line. Maven Assembly is the package designed to do this, so I read the Apache Maven Assembly information, and the Sonatype manual on assemblies, but the examples on both were mixed, and it wasn't clear to me how to do get this working.. All of the material here was all written using Maven Assembly 2.2.1.

I got some basic examples going quickly, but then got bogged down adding a custom assembly descriptor (see below).
The recommendation in both references is to add another module to the project, so it looks like :

adapter
        core
        ftp
        web-service
distribution

Below is the example code I'm currently using to get this : it will be refined, and as it get's better, I'll edit this post.

Parent pom :
Once you move to a multimodule build, make sure you move the plug in information out of the parent pom.xml, and ensure it exists only in the new distribution module's pom.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <name>proj</name>
    <groupId>com.proj.adapter</groupId>
    <artifactId>project</artifactId>
    <packaging>pom</packaging>
    <version>1.0-SNAPSHOT</version>
    <description>
        ..
    </description>
    <modules>
        <module>proj-core</module>
        <module>proj-ftp</module>
        <module>proj-ws</module>
        <module>distribution</module>
    </modules>
    <properties>
        ....
        <maven.assembly.version>2.2.1</maven.assembly.version>
        <plexus.utils.version>1.1</plexus.utils.version>
    </properties>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>${project.groupId}</groupId>
                <artifactId>proj-core</artifactId>
                <version>${project.version}</version>
            </dependency>
        </dependencies>
    </dependencyManagement>
    <dependencies>
        ..
    </dependencies>
    <build>
        <pluginManagement>
        ..
        </pluginManagement>
    </build>

</project>

Distribution module pom.xml
The main items here are :
1. Identify the location of the custom assembly descriptor (shown in bold),
2. Bind the lifecycle in the executions tag. Some of the examples I saw had the configuration within the execution tag, but I never got this to work,


 <?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
 <modelVersion>4.0.0</modelVersion>
 <parent>
  <groupId>com.proj.adapter</groupId>
  <artifactId>adapterProject</artifactId>
  <version>1.0-SNAPSHOT</version>
 </parent>
 <artifactId>dist</artifactId>
 <packaging>pom</packaging>
 <name>Distribution</name>
 <dependencies>
  <dependency>
   <groupId>com.proj.adapter</groupId>
   <artifactId>adapterProject-core</artifactId>
   <version>1.0-SNAPSHOT</version>
  </dependency>
 </dependencies>
 <build>
  <finalName>adapterProject</finalName>
  <plugins>
   <plugin>
    <artifactId>maven-assembly-plugin</artifactId>
    <version>${maven.assembly.version}</version>
    <configuration>
     <descriptors>
      <descriptor>src/main/assembly/bin.xml</descriptor>
     </descriptors>
    </configuration>
    <executions>
     <execution>
      <id>distro-assembly</id>
      <phase>package</phase>
      <goals>
       <goal>single</goal>
      </goals>
     </execution>
    </executions>
   </plugin>
  </plugins>
 </build>
</project>

Custom assembly descriptor
This is just an xml file, referenced from the distribution modules pom.
For a good few hours this was the bit that got me, every attempt to build returned an error message that one file had to be included, and I couldn't see where I'd gone wrong on my dependencies. Eventually I cracked it, and have highlighted it here.
The examples I saw used

<includes>
    <include>*-ws</include>
   </includes>

and other similar examples. Some didn't have the useAllReactorProjects included, but, eventually I got the distribution module to get the correct dependencies and find the file using the snippet below.

<assembly
 xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2 http://maven.apache.org/xsd/assembly-1.1.2.xsd">
 <id>bin</id>
 <formats>
  <format>dir</format>
 </formats>
 <moduleSets>
  <moduleSet>
   <useAllReactorProjects>true</useAllReactorProjects>
   <includes>
    <include>${project.groupId}:adapterProject-core</include>
   </includes>
   <binaries>
    <outputDirectory>/</outputDirectory>
    <unpack>false</unpack>
    <includeDependencies>true</includeDependencies>
    <dependencySets>
     <!--  create the location of the core jar-->
     <dependencySet>
      <includes>
       <include>${project.groupId}:adapterProject-core</include>
      </includes>
      <useTransitiveDependencies>false</useTransitiveDependencies>
      <outputDirectory>/</outputDirectory>
      <unpack>true</unpack>

     </dependencySet>
     <!--  create the lib directory for the supporting jars -->
     <dependencySet>
      <excludes>
       <exclude>${project.groupId}:adapterProject-core</exclude>
      </excludes>
      <useProjectArtifact>false</useProjectArtifact>
      <useTransitiveDependencies>false</useTransitiveDependencies>
      <outputDirectory>lib</outputDirectory>
     </dependencySet>
    </dependencySets>
   </binaries>
  </moduleSet>
 </moduleSets>
</assembly>

Hope this helps someone else struggling with this.

UMLet - UML Tool for Fast UML Diagrams | Eclipse Plugins, Bundles and Products - Eclipse Marketplace

2011-08-17T16:59:00.001+01:00

UMLet - UML Tool for Fast UML Diagrams | Eclipse Plugins, Bundles and Products - Eclipse Marketplace

Very nice tool for simple UML diagramming, that I could then export to Word. I ran it as an Eclipse plug in, and would definitely use this again. There's almost no learning curve, the interface is simple. I thought of BoUML as a good tool, but this is even easier to use.

Home Computing 101

2011-07-14T21:56:00.000+01:00

After a recent IT disaster, the need to back up to more than one location has been driven home pretty brutally. After some work, and some help, I did manage to get back most of what I had before. The biggest loss has been the source code from previous jobs : some of that was quite neat, and always a useful reference.
So backups are back on a weekly schedule to alternating locations : this time one to an external hard drive on Sunday, and one from the primary computer to the server on a Saturday. I will need to organize the backups for the server's repository as well. Backups are through Genie 9's Genie Backup Manager Home 8.0. It's easy enough to use, but haven't tried restoring from it yet.

As part of the general effort to restore the PC, I decided to partition the C drive on the primary PC into 3 areas : C: for Windows and general family use, H: for development work and data, and I: which will eventually host Ubuntu. I use Ubuntu a bit at work, but it'd be useful to get more time with it.
This proved to a useful guide to partitioning a hard drive. From there I looked at SwissKnife, but in the end went for Easeus Partition Master Home Edition. I can't recommend it enough : easy to use, pretty quick, nice UI. If anyone knows any drawbacks on this tool, let me know, because at the moment, I'm sold.

Tomcat 7 Windows Service memory management - a correction

2011-05-05T21:58:00.061+01:00

In an earlier post, I made a serious mistake, so this should go some way to correcting it.

First Hudson Run

My first run of Hudson on Tomcat 7 led to the infamous error :

FATAL: PermGen space java.lang.OutOfMemoryError: PermGen space

Not surprising, given the spec of the machine I'm using, but still a pain.
Using VisualVM, it shows the PermGen size to be still at the default 64MB size. Attempts to monitor the build process weren't successful as the entire machine ground to a halt.

Previously on Tomcat 5/6, I'd've set the catalina.bat or catalina.sh file to contain the memory management options, but this file doesn't exist in Tomcat 7. This was new to me.

Tomcat itself isn't helpful with Permgen, as although it's possible to find the inital and maximum heap memory size on the Manager App, Server Status tab, it doesn't show the permgen size.

There are some really good sites listing the mechanics of setting Tomcat's memory.
WEBMAP is for a Tomcat running on jdk 1.3, but the step by step process to figuring out which parameters are useful is useful.

CATALINA_OPTS vs JAVA_OPTs
I found some sites that suggested setting the OPTS as a Windows environment variable. Several blogs and sites suggest using JAVA_OPTS. I'd caution on this, as although others got it to work, JAVA_OPTS are used by other applications (JBoss in particular was mentioned), not just Tomcat. I decided to stick with CATALINA_OPTS.
But, when I tried setting the environment variable, and restarting Tomcat, I didn't see an increase in the Heap size. I'm not saying it doesn't work, but I couldn't get it to work. Combined with the fact that I didn't like the idea of doing it this way as it seems to be scattering the configuration across a number of places, I didn't really put any further effort into making this work.

Setting memory options on Tomcat

Remember -Xms and -Xmx only set the heap size. These are easy enough to set through the Monitor Tomcat GUI, on the Java tab as the Initial Memory Pool (equivalent to -Xms) and the Maximum Memory Pool (-Xmx).

To set the Permgen size, you need either -XX:PermSize=XXXm or -XX:MaxPermSize=XXXm, and see WEBMAP for a more detailed review of each of these. Again, this can be set on the Java tab of the Monitor Tomcat GUI, this time as a Java Option, I went for -XX:PermSize=128m.

The difference can be seen on the next screenshot, again of the MonitorVM application. Heap size has gone from 67MB to 134MB.

Effect of all of this ?

I now have a Hudson build that doesn't crash Tomcat either during or immediately after each run. Heap size during a build is about the 75MB mark. Maybe now I can start doing some coding ...

Useful references

In addition to those mentioned above, I found these two references from http://www.skill-guru.com to be useful :
Setting Heap Size
Setting Permgen Size

Solving Tomcat 7 memory problems.

2011-04-19T22:18:00.006+01:00

Hudson builds
I started running Hudson builds again, looking for the cause of these out of memory errors.
The project builds, but then fails after each Maven project has built :

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] expiryTracker ..................................... SUCCESS [5.516s]
[INFO] model ............................................. SUCCESS [29.813s]
[INFO] API for expiry tracker ............................ SUCCESS [4.797s]
[INFO] maven-importer .................................... SUCCESS [3.500s]
[INFO] expirytracker-hib ................................. SUCCESS [3.343s]
[INFO] webapp Maven Webapp ............................... SUCCESS [4.453s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 58.266s
[INFO] Finished at: Tue Apr 19 21:21:22 BST 2011
[INFO] Final Memory: 20M/49M
[INFO] ------------------------------------------------------------------------
maven builder waiting
channel stopped
Archiving artifacts
Deploying artifacts to http://server:8082/archiva/repository/snapshots/
uniqueVersion == false is not anymore supported in maven 3
Deploying the main artifact pom.xml
FATAL: PermGen space
java.lang.OutOfMemoryError: PermGen space
 at sun.net.www.protocol.http.HttpURLConnection.getServerAuthentication(Unknown Source)
 at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
 at java.net.HttpURLConnection.getResponseCode(Unknown Source)
 at org.apache.maven.wagon.providers.http.LightweightHttpWagon.finishPutTransfer(LightweightHttpWagon.java:185)
 at org.apache.maven.wagon.AbstractWagon.putTransfer(AbstractWagon.java:413)
 at org.apache.maven.wagon.AbstractWagon.transfer(AbstractWagon.java:392)
 at org.apache.maven.wagon.AbstractWagon.putTransfer(AbstractWagon.java:365)
 at org.apache.maven.wagon.StreamWagon.put(StreamWagon.java:163)
 at org.apache.maven.repository.legacy.DefaultWagonManager.putRemoteFile(DefaultWagonManager.java:543)
 at org.apache.maven.repository.legacy.DefaultWagonManager.putArtifact(DefaultWagonManager.java:494)
 at hudson.maven.artifact.deployer.DefaultArtifactDeployer.deploy(DefaultArtifactDeployer.java:160)
 at hudson.maven.reporters.MavenArtifactRecord.deployMavenArtifact(MavenArtifactRecord.java:139)
 at hudson.maven.reporters.MavenArtifactRecord.deploy(MavenArtifactRecord.java:126)
 at hudson.maven.RedeployPublisher.perform(RedeployPublisher.java:128)
 at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
 at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:622)
 at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:601)
 at hudson.maven.MavenModuleSetBuild$RunnerImpl.post2(MavenModuleSetBuild.java:762)
 at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:548)
 at hudson.model.Run.run(Run.java:1386)
 at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:405)
 at hudson.model.ResourceController.execute(ResourceController.java:88)
 at hudson.model.Executor.run(Executor.java:145)

VisualVM
At this point, I'm not sure whats going on.
In an effort to diagnose, I installed Java's memory viewer, VisualVM. I've used it precursor before, when it was first bundled with early versions of Netbeans 6.
Installing is straightforward, but found that Tomcat didn't show up as a process running under the VM. When Tomcat runs as a service, even if it's running under the same user account that you used to launch VisualVM, Tomcat won't appear by default in the VisualVM process list. I found the answer here, at Matt Woodward's blog , which included the vital information about including

-Dcom.sun.management.jmxremote.port=8086
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false

in the Java Options box on the Java options tab.
With the steps from that and a restart of Tomcat, I now have Tomcat 7 being monitored by VisualVM.

Rush just about over

2011-04-01T00:08:00.001+01:00

In the last month, I've now racked up about 60+ hours overtime. Combined with time in lieu from last year, that makes it about 10 days time in lieu I'm owed. Sweet.
And next week, can get back to my own project.

Late working

2011-02-22T21:26:00.000+00:00

Work keeps getting in the way
We started the final sprint before pilot last week. The team reviewed the estimates, and found there was an excess of work over available hours - no surprises there.
So at least there's been a sensible solution. A lot of people on the team have young kids (including me), so rather than work weekends and panic in the last week, as a team we voted to stay late Tues and Thurs for the remainder of the sprint. Not great, but better than the alternatives. And at least there's pizza.

But it does mean there's almost no time to work on my own project.

Open Source Licences

2011-02-04T21:37:00.000+00:00

I'm going to build this whole app by stitching together open source apps and plug-ins, but as this is mine, I better start making sure all licenses are correct. I did get a link to this site today, which I'll keep as a handy useful reference.

Hudson

2011-02-03T21:29:00.004+00:00

Continuous Integration needed
The advantages of a CI server are well documented elsewhere, and life is easier with them, so I decided to use Hudson. Again, I haven't used this one before, so was keen to see what it was like.

Hudson
The Hudson documentation seems good at first pass, so installation onto the Tomcat service on the server box should be straightforward.
First problem was the installation of the UnxUtils utilities required : no reference was made on how to install these. A quick Google search found these instructions. Problem solved.

The best reference I found to set up Hudson on Windows was this one.

As at the end of tonight, there are still problems getting a build to work. I've made two mistakes :
1. Set the maven home to be the <maven>/bin directory, instead of <maven>

2. Now getting permgen errors running the build - sigh.

Archiva revisited

2011-02-02T22:49:00.001+00:00

Why revisit ?

Last time around I set up Archiva as a stand alone service, on port 8081. This proved easy enough, and I had my first couple of builds, so I started to think about a continuous integration service.
I've set up Cruise Control in a previous job, and had been happy with it. Having looked around, this time round I'm going to use Hudson, and see if it's as easy to use as the documentation claims.

Web server needed
Having looked at Hudson, it can run stand alone, or be deployed to a server. For ease of admin, I decided to put both Archiva and Hudson on the same Tomcat server, and run it as a service. I've been using Tomcat 5.0.x for years, none of the companies I've worked for have felt the need to move past it, so I may as well jump in and see what Tomcat 7 is like.
First impressions : bit disappointing really. The familiarity is good, but there isn't much new for the user there. Compared to the information available through Glassfish or Websphere consoles, it's poor. There are workarounds, I'll see if lambda probe works with T7.

Tomcat 7 problems
Downloading and installing Tomcat 7 as a service was no problem, and I got Archiva installed as a web app - easy enough when following the site documentation

One thing to remember when running Archiva on Tomcat as a service : set the parameters in the Tomcat 7 properties dialog box:
-Dappserver.base=D:\Tomcat 7.0
-Dappserver.home=D:\Tomcat 7.0

But
When I came back a couple of days later, I found the client couldn't connect to the server, the dreaded "Server is taking too long to respond" message came back.
I dread these kid of sys admin problems, no matter how trivial. Anyway, I could ping the Tomcat box, on the right port, but could not get a browser request to work. I uninstalled Tomcat 7. As a comparison I downloaded Tomcat 6, and ran it on a different port : this time no problems. But then to double check, I re installed Tomcat 7, but this time, no issues connecting to the manager app from the client machine.
So at the end of a night's work, I'm not really any further forward than I was. I now have Archiva running on a Tomcat 7 service. I need to update my Maven settings to point to the new port, and run a Maven build.

Build process

2011-01-27T22:32:00.000+00:00

Why Maven ?

I had to decide which build system to use, Ant or Maven. We use Ant at work, and it's a fairly sophisticated script, using a lot of inheritance and overriding to build the project. I've written Ant scripts before, and to be honest, it's dull and a bit of a ballache. So I decided to try Maven as :
1. It's new
2. It'll save time on the build dependencies
3. It integrates with Eclipse through the m2Eclipse plug in.

I figured once it was up and running, it'd save time, and it's a useful tool to have.

Maven documentation

The Maven documentation isn't good, it reads like one of those guides that makes more sense after you've been using it a while. I've hunted through a few blogs, some of which are good, but that's inevitably fragmented. I'm using Apache Maven 2 Effective Implementation (by Ching and Porter) as my main reference, and it's proving reasonably useful : better than the initial review on Amazon suggests anyway.

What to use as a Maven repository ?

Maven generally recommends using a proxy repository, and after checking a good article on theserverside.com and the Maven book, decided to check out Archiva . It's turned out to be a good find, being very easy to set up and use, with decent documentation. Again, follow the wizards, there isn't much more to it.

Multi module Maven

So I started to generate a multi module Maven project. It turned out to be simple using the mvn:generate task, the only thing I'd have liked would be some guide to the archetypes available. For a newbie, the list is large and it's not clear what's going to be the best to use.
I hit one massive problem during the build, where the first module would build, but the second module couldn't find the first build's arrtifact, stating the jar could not be found in the local repository. Close review of the logs showed the first module was publishing to the local repository, and I could see it there.
The problem ? The dependency tag in each module's pom.xml :

    <dependencies>
        <dependency>
            <groupId>${project.groupId}</groupId>
            <artifactId>maven-importer</artifactId>
            <version>${project.groupId}</version>
        </dependency>
    </dependencies>

It should have read :

    <dependencies>
        <dependency>
            <groupId>${project.groupId}</groupId>
            <artifactId>maven-importer</artifactId>
            <version>${project.version}</version>
        </dependency>
    </dependencies>

Which drives home the lesson I find most days : it's almost always something simple.

Still, I now have a working Maven build system, and a local archive.

Dev environment

2011-01-18T21:55:00.000+00:00

Version Control Choice

Even with something basic, version control is too useful to just omit. I decided to go with the latest version of Subversion (http://subversion.apache.org/) because

I've administered small scale repositories before, and learning version control admin isn't an objective or something I want to spend any time on.
With TortoiseSVN (http://tortoisesvn.tigris.org/) it integrates nicely into Windows Explorer, and it's easy to use
I haven't tried the Subversion Eclipse plug in, but once it's up and running, I'll add this to the IDE.

Installation

I've installed Subversion on the Server machine, and was pretty impressed with the ease of use of Subversion 1.6.13. Previously at work I used a file repository and IIRC it was version 1.4, but setting this new release up as a web server repository proved stunningly easy, simply following the wizards. The browser based front end is pretty good. So now I've got a basic installation up and running, only two users (Admin and Dev users), and my first few files booked in, but with the ability to create multiple repositories as I need them.

Repository per project ?
I did have an internal debate, whether to go for 1 repository for all projects, or 1 repository per project. I checked the first few links from a Google Search, and, at this point, the 1 repository per project model seems easier. I don't have any requirements for hook scripts or other cross project concerns at this point, so the admin overhead isn't a consideration (yet)

So from now on, even dev files, Eclipse settings etc can be tracked and rolled back - useful when I'll be doing so much experimenting over the next few weeks. Nice.

Starting points

2011-01-13T21:40:00.000+00:00

Hardware
I've currently got a Windows home network, run through a Zoom Modem / Router, with two increasingly old PC's linked. The oldest is an old Dell Pentium 2.4Ghz, 2 GB of RAM. Not the best, but it'll do as a host for the database, build system and application servers.
The other is slightly better HP xw4300 Workstation, 3.2Ghz Pentium D with 3GB of RAM. This is the development machine, but also remains the family PC.
Right now, I don't really want to spend any more money on hardware, so options like more RAM, SSD's etc are out, never mind getting new PC's. Besides, what I've got might not be shiny, but it'll meet my needs for now. Maybe later or if a windfall comes along, but right now I can't justify that to the other half.

OS : Linux or Windows ?
I've done some Unix before, not much in the last few years, and we're using Linux at work at the moment. I know I could do Linux VM's, but, improving my Linux isn't really an immediate concern, and it'll just slow me down for a while. I'll take the easy option and stick with Windows on both. I'll come back to this later, maybe near the project end, and migrate the server to a Linux distro - no idea which one at this stage.

Why am I doing this ?

2011-01-12T21:40:00.000+00:00

New Year, New Project
I've started a new software project at home. It'll be a new web app, a fairly standard view onto a database. The main goal of the project is to give me a chance to play with some frameworks that I haven't had a chance to work with yet, give some focus to my reading, explore the bits of IT that I have no experience of ( blogging, hosting and domain names came up straight away !) and if at the end of it, there's an opportunity to make some money out of it, bonus. This blog is to support this project.

What's the aim of the blog ?
In almost every project I've been involved in, normally a couple of years afterwards, there's been a period of reflection and asking "Why did I choose to do it that way ?" The main aim of the blog is to capture those decisions and the rationale behind them. By making it public, there might be a chance to get some external review of the decisions - one of the main problems of a one man project, which does limit the learning experience.

It's also going to be a place to put links to articles that have contributed to those decisions, problems encountered and the solutions adopted.

And the name comes from ?
The name comes from the hundreds of decisions that will have to be made along the way, and the choices that are open. First steps are going to be to think about the supporting infrastructure, then the stack I'm going to use.

Or at least, that's the idea anyway.