Thoughts For Beans

Mobile-friendly websites

2012-09-20T17:30:00.000-07:00

Many people these days rely more on their mobile devices - phones, tablets etc - to access the Internet than on desktop / laptop computers. As such, it is a common requirement in many websites today that we design / test / tune the usability not just for different computer browsers but also for mobile devices.

While not every website has to cater to mobile devices this article gives some ideas on how you can make new and existing websites more mobile-friendly without requiring lots of coding or duplication.

If you are building a new website from scratch there are few tools that can help kick-start your own project such as:

H5BP: http://html5boilerplate.com/

Initializr: http://www.initializr.com/

These tools are project skeletons you can use to start your own projects. They already integrate / implement several best practices, proven technologies and tricks that can save you a lot of time.

If you already have a website and you can't rewrite / refactor it on top of one of the projects above then you have few options:

You can write a completely separate mobile site - some sites take this route since this allows full separation between the "PC version" and the mobile version of their sites. The obvious drawback is that now you have 2 sites to maintain.

Another option is to pick few tweaks from the projects above and add it in your existing site. For example, many times we can make existing websites more mobile-friendly with few simple changes:

Automatically adjust page layout and font sizes

Just include this single line in the head element of all your HTML pages:

<head>
    …
    <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0">
    …
</head>

This is how www.eventfy.com looks like in the iPhone without and with this single-line code:

Eventfy without viewport tweak

Eventfy with viewport tweak

It may not look like a lot but with this minor tweak users don't need to constantly zoom in and scroll horizontally in every page. Also, it does not break usability in PC browsers.

Use responsive layouts

If you use fixed layouts where we position DIVs regardless of the client's resolution users will likely have to scroll horizontally. You can improve things considerably if you use percentages to set the widths of your DIVs. This will avoid the scroll bars but may result in tiny columns for users with smaller screens.

A better approach would be to setup the layout so that it properly adapts to the client's resolution / screen width. Check out Bootstrap's live grid example here. Note that when you open it with your PC browser maximized the 9 columns (those marked with '1') are listed horizontally. When you resize the browser reducing the window width the layout will automatically change so that all '1' columns are stacked on top of each other.

It sure helps if you can use something like Twitter Bootstrap, but if not you can still achieve similar results with custom CSS code. More specifically, you can use the @media tag to query information about your client and tweak your CSS classes. For example:

@media all {
    // add common CSS here
}

/*--- iPad ---*/
/*--- full screen mode ---*/
@media (device-width: 768px) and (orientation: portrait) {
    .panel {
        height                       : 934px; /*1004-80*/
    }
}
/*--- browser mode ---*/
@media (device-width: 768px) and (orientation: landscape) {
    .panel {
        height                       : 668px; /*748-80*/
    }
}

/*--- iPhone / small screens ---*/
/*--- full screen mode ---*/
@media (max-width: 480px) and (orientation: portrait) {
    .panel {
        height                       : 380px; /*460-80*/
    }
}

With the code above the "panel" class will be tweaked based on the client's resolution.

You can find a better example of what you can do with CSS and media queries in the www.eventfy.com website. In common PC resolutions Eventfy's top menu bar lists all items horizontally. In smaller screens it consolidate items in a drop-down list to save space. This is what it looks like:

Eventfy's top menu bar - full monitor resolution

Eventfy's top menu bar - browser window resized at 360px width

Eventfy's top menu bar, expanded menu - browser window resized at 360px width

Another thing I did to make better use of small mobile screens is to hide some unimportant content. For example, the long marketing message below the Eventfy title only shows up in big browser windows, and is hidden in smaller devices.

Setting this up was simple, I created a CSS class "details", and in the @media with "max-width: 480px" sections I override it with a "display: none;". This way all the extra content that fits nicely in the big PC browser window but doesn't in smaller devices gets hidden.

Conclusion

If you are building a new website, consider making it compatible with mobile devices from the start. It isn't extra work if you are doing it from the beginning.
If you already have existing websites, determine if it makes sense to make them mobile-friendly. Not every website would benefit with support for mobile devices.
Test your sites with mobile devices. Pick someone to test it and give you feedback, or pretend you are one of your users. Identify main usability items that can be improved (e.g. reduce zooming / scrolling, adjust layout, hide extra content etc).
Try tools to help improve layout in different devices / browsers without increasing maintenance. These can save you a lot of time coding / tweaking things by hand.
Implement tweaks to improve usability.

And that's it for now, happy coding folks!

My GoDaddy domain is down

2012-09-10T11:49:00.001-07:00

My domain registrar (GoDaddy) is currently down (9/10/2012 11:20AM AZ) in what seems to be an attack from the group Anonymous. Not only their services are down, even their main webpage is down right now! Wow, that was ugly!

I am not hosting any of my projects with them. Everything is hosted on Google infrastructure (AppEngine, Apps for Business, Blogger etc). Unfortunately they are all down too because I still rely on GoDaddy's DNS servers! Grrrrr! :(

The only positive thing I can take from this issue is that my web monitor app Pingr worked as designed and quickly notified me about the problem. In this case there isn't much I can do to fix it though, have to wait for GoDaddy figure it out on their side. At least Pingr will tell me once their system is back up. Until then Pingr's main URL (http://pingr.eventfy.com/) will be unavailable.

P.S. If anyone else wants to receive a notification from Pingr when your system is backup you can use it from its Google AppEngine's URL: http://pingr-eventfy-com.appspot.com/.

Update: at 3:18PM AZ (about 4 hours since I found out about this issue!) my domain eventfy.com finally got back up.

It seems some sites are still affected but GoDaddy is slowly getting there. You can find a discussion about this issue here (Reddit).

Update: yesterday night (9/13) GoDaddy sent me an email taking the blame, stating that it wasn't caused by an external hacker but by themselves. Not sure if (i) I buy it and (ii) it makes me feel any safer if this is true. Anyways they gave me a 30% off new product or renewal valid for 7 days. Man, I am just happy I don't depend on this for my livelihood! :)

Server resource utilization monitoring

2012-09-05T15:10:00.001-07:00

Pretty much every Linux distribution has tools to help sys admins monitor the utilization of common server resources such as CPU, memory and storage. For advanced monitoring and reporting we may need to install 3rd-party tools (e.g. Nagios or FAN) but sometimes we just want a simple way to get warnings when critical server resources may be close to overload.

Here is a set of simple shell scripts (tested in RHEL and CentOS) you can use to monitor CPU, memory and storage space. If utilization goes over a threshold these scripts will send you an email warning hopefully soon enough so that you can fix it before a system crash. Assuming you have access to basic commands like vmstat, df, mail and crontab these scripts can be setup even if you do not have root access to the server.

CPU Monitor

This will send you an email notification if the average CPU utilization of the last 10 seconds is higher than the configured threshold.

#!/bin/sh

#
#CONFIGURATION BLOCK:
#

#Set the alarm threshold here
#e.g. set to 90 to make it send notifications if average utilization goes past 90%
THRESHOLD=90
#Add your email address here
MAILTO="myemail@eventfy.com"
#The temp file below will be replaced everytime this script runs
TEMPFILE=/tmp/cpu.temp

#
#LOGIC BLOCK:
#

HOSTNAME=`hostname`
rm -f $TEMPFILE 

#Calculate the CPU Utilization with the command below
#Note: this command will calculate the avg CPU utilization of last 10 secs
CPU=$(vmstat 10 2 | tail -1 | awk '{print $15}' | sed 's/%//')
let "CPU = 100 - $CPU"
#echo "`date` - CPU Utilization: $CPU"

#Compare the current value with the threshold one.
if [ $(expr $CPU ">=" $THRESHOLD) -ne 0 ]
then
echo "Warning - CPU utilization on server $HOSTNAME is ${CPU}%" >> $TEMPFILE
fi

#Send an email if /tmp/cpu.temp is present.
if [ -e $TEMPFILE ]
then
mail -s "$HOSTNAME CPU Warning" $MAILTO < $TEMPFILE
fi

rm -f $TEMPFILE

Memory Monitor

This will send you an email notification if the percentage of used RAM + swap is higher than the configured threshold.

#!/bin/sh

#
#CONFIGURATION BLOCK:
#

#Set the alarm threshold here
#e.g. set to 90 to make it send notifications if memory utilization goes past 90%
THRESHOLD=90
#Add your email address here
MAILTO="myemail@eventfy.com"
#The temp file below will be replaced everytime this script runs
TEMPFILE=/tmp/memory.temp

#
#LOGIC BLOCK:
#

HOSTNAME=`hostname`
rm -f $TEMPFILE

#Calculate the SWAP Utilization with the below command.
#Note: this script will take avg utilization of last 10 secs
TOTAL_RAM=$(vmstat 1 2 -s | grep "total memory" | awk '{print $1}')
#echo "`date` - TOTAL_RAM: $TOTAL_RAM"
USED_RAM=$(vmstat 1 2 -s | grep "used memory" | awk '{print $1}')
#echo "`date` - USED_RAM: $USED_RAM"

TOTAL_SWAP=$(vmstat 1 2 -s | grep "total swap" | awk '{print $1}')
#echo "`date` - TOTAL_SWAP: $TOTAL_SWAP"
USED_SWAP=$(vmstat 1 2 -s | grep "used swap" | awk '{print $1}')
#echo "`date` - USED_SWAP: $USED_SWAP"

PERCENT_USE=0
let "PERCENT_USE=100 * ($USED_RAM + $USED_SWAP) / ($TOTAL_RAM + $TOTAL_SWAP)"
#echo "`date` - Memory Utilization: $PERCENT_USE"

#Compare the current value with the threshold one.
if [ $(expr $PERCENT_USE ">=" $THRESHOLD) -ne 0 ]
then
echo "Warning - memory utilization on server $HOSTNAME is ${PERCENT_USE}%" >> $TEMPFILE
fi

#Send an email if /tmp/memory.temp is present.
if [ -e $TEMPFILE ]
then
mail -s "$HOSTNAME Memory Warning" $MAILTO < $TEMPFILE
fi

rm -f $TEMPFILE

Disk Space Monitor

This will send you an email notification if a target partition's percentage used space is higher than the configured threshold.

#!/bin/sh

#
#CONFIGURATION BLOCK:
#

#Set the alarm threshold here
#e.g. set to 70 to make it send notifications if partition utilization goes past 70%
THRESHOLD=70
#Add your email address here
MAILTO="myemail@eventfy.com"
#The temp file below will be replaced everytime this script runs
TEMPFILE=/tmp/diskspace.temp
#Set target partition with unique substring identifier based on the output of 'df -P'
TARGET_PARTITION="LogVol00"

#
#LOGIC BLOCK:
#

HOSTNAME=`hostname`
rm -f $TEMPFILE

#Calculate the partition utilization with the command below
PERCENT_USE=$(df -P | grep "$TARGET_PARTITION" | awk '{print $5}' | sed 's/%//')
#echo "`date` - Disk space usage: $PERCENT_USE"

#Compare the current value with the threshold one.
if [ $(expr $PERCENT_USE ">=" $THRESHOLD) -ne 0 ]
then
echo "Warning - low disk space (${PERCENT_USE}%) available on partition ${TARGET_PARTITION} in server $HOSTNAME" >> $TEMPFILE
fi

#Send an email if /tmp/diskspace.temp is present.
if [ -e $TEMPFILE ]
then
mail -s "$HOSTNAME Disk Space Warning" $MAILTO < $TEMPFILE
fi

rm -f $TEMPFILE

Setup

1. With a text editor (e.g. vi) create new shell script files so you can copy/paste the code above.
2. Make sure you edit the configuration part of each script (e.g. your email address, target partition etc).
3. Setup crontab to execute these scripts every X minutes:

Use "crontab -e" to edit and save your changes. Assuming you will keep these scripts in "/root/monitor" and want to run each one every 10 minutes your crontab configuration would look like this:

0,10,20,30,40,50 * * * * /root/monitor/cpu.sh
1,11,21,31,41,51 * * * * /root/monitor/memory.sh
2,12,22,32,42,52 * * * * /root/monitor/diskspace.sh

Notes

These scripts do not require special permissions and should run fine with non-root users in most systems. It may be a good idea to setup these scripts as root if you have root access though.
These scripts are very lightweight but it may still be a good idea to set them so they don't run at the same time like above.
You can use "crontab -l" to list and review the crontab configuration.

Additional Notes

You should test these scripts in your server if you plan to rely on these notifications. You can run a quick test by setting the thresholds to some low number and validate that you are getting emails.

If you get error "mail: command not found" you may need to run this as root:

yum install mailx

And finally, keep in mind that these scripts will be sending you emails until the problem is fixed. It would be simple to change it to only send emails when the situation changes, but I was okay with emails every 10 mins until the problem is fixed.

My Amazon Wish List

2012-08-27T05:00:00.000-07:00

Selected items from

My Amazon Wish List

Lots of juice to power your gadgets during those long flights

If you have a compatible Mac get this and the Samsung SSD laptop kit for the best upgrade you can possibly get on your laptop!

Includes laptop upgrade kit accessories

This kit comes also with 4 rechargable batteries (2xAA and 2xAAA) and spacers to convert the batteries into C or D sizes

Hide your backyard and pool stuff in style

No, really, how cool is that?

Zap those flying bugs and reclaim your backyard!

Yes it is expensive as heck but this is a wish list right? :)

Get few of these and save energy (and $$$)

Helps you find the energy hogs in your house

Automate your house's cooling / heating
Set the temperatue with your smartphone

You should probably be extra careful shaving with this one

Is there any all-in-one better than this yet?

For the (weekend) handyman (wannabe)

This magical item not only gives you darkvision but if the need arises you can also use it as a club!

Hook this up on your living (or media) room HTPC and its "couch potato" time

Guess this is good for guitarists?

Translate your website with Google Website Translator

2012-08-24T17:30:00.000-07:00

Oftentimes we want to support multiple languages in our websites. One "easy" way is to keep one separate resource bundle for each language we support, but this isn't as simple as it seems. First you need a tag / mapping mechanism to match labels in your pages with tags in the property files (frameworks like JSF already provide it) but then you need to maintain multiple sets of labels. Any new field / screen etc will require changes all resource bundles, which can quickly become a burden. In many cases this burden may be acceptable in exchange for full control and performance, but in other cases we can use Google's Website Translator tool.

Follow these steps to find out how to easily include a styled language selector into your website, and to remove the Google Website Translator toolbar. I recently applied these steps in Eventfy and the final result looks like this:

First login with your Gmail account and sign your site up to Google Website Translator. Just click the “+ Add new website” button:

Select your site’s default language, and all other languages you want the GWT widget to list to your users. At the end it will give you a HTML code snippet like this:

<div id="google_translate_element"></div><script type="text/javascript">
function googleTranslateElementInit() {
  new google.translate.TranslateElement({pageLanguage: 'en', includedLanguages: 'en,es,fr,it,nl,pt', layout: google.translate.TranslateElement.InlineLayout.SIMPLE}, 'google_translate_element');
}
</script><script type="text/javascript" src="//translate.google.com/translate_a/element.js?cb=googleTranslateElementInit"></script>

Just add this code where you want the language selector to appear in your page. In Eventfy’s case I added it inside a <li style="padding-top:5px;"> block to get it aligned with the other menu items. This is all there is to get it to work, give it a try!

This is how Eventfy looked with this default code:

Default language selector with Google icon

Google translator toolbar hiding Eventfy's top menu and bad Portuguese translation

I like how easy it was to include it into Eventfy but I thought the language selector and the toolbar didn't look good with the rest of the site. The next step was then to override Google's default style with custom CSS so it looks as part of Eventfy's design. I removed the Google icon and the Google translator toolbar with the following style overrides:

1. Remove Google icon and "linky" behavior from language selector:

.goog-te-gadget-icon {display:none;}
.goog-te-gadget-simple a {text-decoration: none !important;}

2. Remove the Google translate toolbar:

.goog-te-banner-frame.skiptranslate {display: none !important;}
body { top: 0px !important; }

Note: you may want to add a <span class="notranslate"> anywhere you don't want translations (e.g. email addresses like “someone at somedomain dot com”).

Please note that while this is very easy to setup, since this is all computer-based some translations are way off. Thankfully we - site admins and users - can improve / override the automated translations with rules and suggestions (always managed by the website admin). More specifically, users can click on a translated phrase and suggest a better translation. Then, the website's admin can see and approve / reject suggestions from the Google Website Translator website.

Final result - no toolbar, no Google icon and improved Portuguese translation

I am still testing this service but if it works well it may be a simple (and cheap) approach to support multiple languages.

Happy translations to all! :)

Hide input type file from older iOS devices

2012-08-22T16:46:00.002-07:00

I recently added support for user image upload in Eventfy. You can find more details and screenshots here.

Safari mobile in iOS devices like iPhone and iPad do not yet support file uploads and render input type file disabled. It may change with the upcoming update to iOS 6, but until then our web applications must take that limitation into account.

I wanted to hide the file update controls in iOS and show a note in its place. Instead of checking the browser version I am simply checking the state of the input file, which may also cover other devices with similar limitations.

The code below describes how to set it up with jQuery, but it is simple to do the same with plain Javascript.

HTML part:

<div id="imageUploadDiv" class="control-group">
 <label class="control-label" for="imageUpload">Image</label>
 <div class="controls">
   <input id="imageUploadInt" name="imageUploadInt" class="input-file" type="file" />
   <p class="help-block">You can optionally include a picture or image in your invite</p>
   <span id="imgPreview" class="thumbnail" style="display:none;width: 250px;"></span>
 </div>
</div>
<div id="imageUploadiOSWarning" class="alert alert-info" style="display: none;">
 <button type="button" class="close" data-dismiss="alert">×</button>
 <strong>Note</strong> Image upload isn't supported by iOS before version 6.0
</div>

Javascript / jQuery part:

<script>
$(document).ready(function(){
    if ($("#imageUploadInt").prop("disabled")) { 
        $('#imageUploadiOSWarning').show();
        $('#imageUploadDiv').hide();
    }
}
</script>

In this simple example DIV "imageUploadInt" is presented by default. But as soon the document loads we check if the input file was automatically disabled by the browser, and if so we hide "imageUploadInt" and show instead DIV "imageUploadiOSWarning".

Note: this DIV structure and CSS classes are defined in Twitter Bootstrap. You can find more about Bootstrap control forms here, on section 'Validation States'.

Update: I tested the image upload function in Eventfy yesterday (9/19) with iOS 6 and this code works great. Now users can upload pictures (if their device supports it), otherwise they are presented with a message explaining why the control is disabled.

Here are some screenshots to show how it works with iOS 6:

Input type file is enabled with iOS 6

Photo selection

Photo uploaded to Eventfy

My Amazon Favorite Buys

2012-08-19T10:30:00.000-07:00

A list of

My Amazon Favorite Buys

Read the disclaimer or check this list out and you may find something interesting

Great case for your small USB drive and USB cable

Small but large capacity USB drive, fast transfers and no separate power chord, excellent for backups
Note: they released a new version available here

Here's a relatively inexpensive item (compared to this one, that is) that can make a big difference when you're flying.

Tiny wireless N adapter

Tire inflator with a built in pressure gauge and adapters for bikes, balls etc
Quite noisy but strong enough for SUVs

Cool mist humidifier, good for those in dry climates

Works great with iPhones

Hook up 2 computers to the same monitor, keyboard, mouse and audio
Worked great with my URB keyboard and mouse

Find those da$% scorpions! Or check out CSI-style how clean your hotel room really is, it may surprise you

This is one of those things you never need... until you need it, that is!

More HDMI ports for your TV
Switch inputs automatically so you don't even have to use the included control

Good UPS for computers
Get this if you want extra time to save your work during a blackout, or to ensure a graceful shutdown if you hook it via USB

Don't use it very often but has a good price, looks great and so far it worked for years without problems

Nice gift for the wife, it looks great (on her, that is!)

Great radio for the bathroom countertop. I wouldn't keep it in the shower like some people at Amazon though
If this one is OOS you can find a similar version here

Best kitchen (or general purpose) radio ever, great quality sound and design

Excellent classic D&D game for under $10. If you like D&D and CRPG but you never played BG2 do yourself a favor and get it now. Great stories with an amazing replay value

Amazing board game set on the Game of Thrones' universe

Broom? What broom? Every house should have one of these
Note: this product doesn't replace a full blown vacuum

I bought 2 more expensive ones (the most expensive from Norelco and one from Wahl) but ended up coming back to this cheap one

This compact fabric steamer worked so well for my sister-in-law's wedding the whole family used it. So much so they kept it as a "gift" and we now gotta buy another one for the next weddings

Great gift for babies, it plays different sounds and songs in English and Spanish

Hands free soap dispenser. Get it for the kitchen and the bathrooms

Gift

Disclaimer

I purchased all items on this list either from Amazon or from other merchants and I am very happy with them. I am sharing this list so that if you buy anything you like here there will be a (tiny) smaller chance for them to be taken out of the market. However, your own experience with these products may differ.
I found that in most cases Amazon has the best prices or is at least close. You may want to check it with Google Shopping, other price comparison tools or in deal sites like Slickdeals. Still, Amazon reviews are great to help you find your own great buys, even if you buy elsewhere.

This blog is back

2012-08-17T19:30:00.000-07:00

After a (too) long hiatus I finally decided to get rid of the cobwebs and write here again. I didn't have lots of cool ideas to discuss so I decided to start a new pet project to give me some inspiration. After few weekends this is what I came up with:

Eventfy (http://www.eventfy.com), a simple web tool to setup events. You can use Eventfy instead of group emails to organize and track events. You can even setup conditional events like those poker (or RPG) games you can't confirm yet because you're not sure enough people will show up. In this case, once enough people confirm their presence Eventfy automatically notifies everybody that the game is on. If not enough people confirmed by the deadline you set Eventfy sends an event cancellation note to all invited. Thanks to responsive design techniques all functions work great from computers and from small devices like smartphones and tables.
Pingr (http://pingr.eventfy.com), a basic web-based availability monitoring tool. Just enter a URL from your site and Pingr will monitor it for you. If the site goes down or performance (ping or download speed) drops below normal thresholds it will quickly send you a notification.
Eventfy Blog (http://blog.eventfy.com), where I will post updates about anything hosted in the Eventfy.com domain.

These are quite simple projects but they are mobile-friendly (the screens won't look crappy on smaller screens and as such users won't have to zoom in / out in iPhone and iPad) and it can transparently scale to meet the demand (hosted on Google App Engine's PaaS solution).

Check out after the break all the different topics I am planning to discuss here soon, all inspired from my experiences with Google cloud platform, public internet domains, responsive websites, HTML5, Twitter Bootstrap, jQuery and more.

I spent just few weekends on these small projects but I end up getting inspired (and thankfully a bit more knowledgeable too) to write about and share code / details on the following topics:

Blogging platforms: Wordpress vs Blogger
Custom domains

How to find a good name for cheap
Setup DNS zones: A, CNAME, MX etc

Centralize multiple services in Google Apps for free

Use it to manage your custom domain and get

email addresses like yourname@yourdomain.com backed by Gmail
many Google services like Blogger, Calendar, Gmail, Docs mapped to URLs like blog.yourdomain.com, calendar.yourdomain.com, mail.yourdomain.com etc

Integrate with your Google App Engine projects

Cloud computing on the cheap

PaaS vs IaaS
Google App Engine vs Amazon

Google AdSense in websites and mobile sites

Set up ads (and get $$$) on responsive websites

Build responsive modern-looking websites with HTML5 + Twitter Bootstrap + jQuery

A single site for both computers, smartphones and all in-between
Kickstart your project with H5BP
Start from a boilerplate project with Initializr
Bootstrap themes

Functional web tests with Selenium
A simple way to escape user input and prevent code and sql injection attacks
Cool graphs with jqPlot
User authentication with OAuth, Google Accounts

Don't spend time writing your own auth mechanism
Let users securely sign in with their Google or Facebook accounts

Resource optimization in Google App Engine

Reduce datastore I/O with caching
Improve performance with batch operations

Date and time hell - yes, this is more complicated than you think

Timezone conversions
Internal vs Human friendly formats (e.g. server uses UTC)
Device-specific date / time formats

Datetime picker mechanism that works with both PC and iOS devices
Google App Engine cron tasks
iCalendar attachments in emails
Website performance optimization
LESS vs CSS in the real world
Build responsive UIs with AJAX and JSON
Message templates in Java

I am happy with all the different ideas and fun these projects gave me so far. If you ever find yourself looking for ideas to write about try starting up / build something new and I am sure you too will find lots of cool things to discuss.

If you are interested in any of this just keep an eye here or subscribe and you will find updates on these topics - in no particular order - pretty soon. Until then feel free to give Eventfy and Pingr a try and leave me a comment to let me know what you think.

Kindle media converter Windows script

2010-05-18T18:30:00.000-07:00

Check this post out for a Windows batch script to convert images into PDFs optimized for the Amazon Kindle e-reader.

So a while back I got an Amazon Kindle as a birthday gift from my brother-in-law (many thanks Junior!).

Coincidentally, I also got from my lovely wife a book with 20 years of Dilbert - that's funny, even though I read it for years I had no idea there were 20 years of Dilbert! The book, which is pretty huge by the way, thankfully came with a CD with images of all its pages.

Needless to say what happened, right? Since Kindle's light weight makes it much more comfortable to read than the book I wanted to find a good way to put all the images there.

Amazon did include support in the Kindle for photo-albums, or folders with image files (see detailed step by step here). This mechanism works well for photos but not so well when there are a proper sequence of images - for some reason it frequently presented some random page instead of the next page. Also, since the images weren't sized specifically for the Kindle it had to resize it on the fly (the Kindle isn't exactly a speed daemon to handle anything other than text books) and the images were using more space with resolution and colors its grayscale screen just isn't able to reproduce.

Then few months back I put together a Windows batch script to improve on that. I decided to share it, perhaps it might be useful for other e-reader users, or someone who wants to learn from a working script on how to use this amazing open source image processing tool ImageMagick. The script uses Windows and ImageMagick commands to process these 3 steps:

1. Iterate through all sub-folders for images (GIF, PNG, JPG and BMP) and sort them alphabetically by path and file names

This script can correctly handle directory hierarchies like:
.\volume1\image1.png
.\volume1\image2.gif
.\vol2\img1.gif
.\v3\img1l.gif
.\v3\img1r.gif
etc

2. Convert all images from their original format to GIF, resizing it to 600x800 with 16 shades of gray for optimal Kindle memory use.

If you have a different e-reader you can still use this mechanism with some minor modifications to resolution and image depth - keep reading for more info

3. Generate a PDF file with all processed images - this step ensures Kindle will follow the proper page sequence.

Please note that this isn't a super efficient solution or script, and I am sure some Windows batch file genius can point how to better implement some of these functions. I actually learned quite a bit about Windows batch scripts while I was implementing it (and especially how bad Windows scripting is compared to Unix shell scripting).

Installation

1. ImageMagick
ImageMagick is a set of image commandline-based manipulation tools. You can download the latest stable build here: http://www.imagemagick.org/script/download.php

Following their recommendation I downloaded ImageMagick-6.6.1-10-Q16-windows-dll.exe. You should use the latest stable version available on their website. During the installation there is a checkbox labeled "Add application directory to your system path" - make sure this is checked (default value). This is how the converter batch script will be able to use ImageMagick. I installed without changing any options, just Next / Next / Finish.

2. Ghostscript
Ghostscript is a PDF manipulation tool. You can download the latest stable build here: http://pages.cs.wisc.edu/~ghost/

Since we are using Windows we don't need to download the sources (the first set of files in the download page) - just scroll down and download one of the 2 Windows binary builds (32 or 64bit).

I downloaded file gs871w32.exe from this page just for compatibility reasons. If you have a 64bit Windows you can use instead the latest 64bit build for improved performance. Again, I installed it with all default settings.

3. Testing
Open a command prompt window and type this command:

convert.exe -version

If ImageMagick's installation worked properly the result should look like this:

About the script

The complete script can be downloaded here. Follow the text below for detailed instructions on how it works and how you can use it.

You should NOT download and execute batch scripts on your computer from someone you don't know without taking a good look at the code to ensure it is not going to cause your computer any harm. I am just sharing it here because it could be useful for others and I do not provide any guarantees that it will work for you or that it will not somehow break your computer. However, at least I didn't knowingly design it to break anything in your computer. If you download and execute this script you are doing so on your own risk.

The 2nd warning is, you should NOT use this script on content you do not own. I am sharing this just as a way to convert what you already have into your e-reader. Support good authors.

Okay, so now that the "no-guarantee liability babble" is out of the way let me explain how it works:

1. Image consolidation

mkdir work

REM Copy all images to current directory with unique names
    ECHO Pass 1/9 - Consolidate images
pushd %*
for /F "delims=" %%j in ('dir /B /O /S *.gif') do (
  rem echo file=%%j
  set fileName=%%j
  set uniqueName=!fileName:\=_!
  set uniqueName=!uniqueName::=!
  copy "!fileName!" "work\!uniqueName!"
)
popd

This block creates a temporary "work" sub-directory to consolidate images from all other sub-directories. It copies all images from other sub-directories with new names based on the image's path + original name, so the path and name sorting is kept and there are no naming collisions. For example, ".\vol1\1.png" would be copied as to "vol1_1.png" and ".\vol2\1.png" would be "vol2_1.png".

Let me re-state this - the script make copies of all images from all sub-directories into "work". It will NOT remove / rename / process the original images in any way, but only these temporary copies in "work".

2. Remove unwanted images

REM Remove unwanted images
del "work\*Credi*"

This block removes all unwanted images from the "work" directory based on the image name. If your source has images you do not want to transfer to the Kindle you can use this step to filter them out and squeeze the final PDF size a bit.

3. Convert all temporary images to 4bit color GIFs

set MAGICK_TMPDIR=work

REM Convert all GIFs
ECHO Pass 2/9 - GIF to 4bit color GIF
FOR %%a in (work\*.gif) DO (
    ECHO Processing file: "%%~nxa"

    REM Convert GIF to 4bit color GIF
    convert.exe -type GrayScaleMatte +level 0,15 -level 0,15 -quality 100 "%%a" "t1-%%~na.gif"
)

I know PNG is a newer and better format than GIF for various reasons, but I found that if I switch from GIF to PNG above then for some reason I got weird image size issues in the final PDF file.

4. Divide pages in two images

I didn't need this one for my book but it may be useful for other ones. If every original image has two pages side by side (e.g. a full image scan of the two pages) you can uncomment the block below to divide each image into two:

REM Divide pages in two images
rem FOR %%a in (t1*.gif) DO (
rem     convert.exe "%%a" -crop 50%%x100%% +repage "t2-%%~na%%d.gif"
rem )
REM Remove temporary images
rem del t1*.gif

5. Rename all temporary GIFs based on their proper page number

REM Rename all GIFs to format 00000.gif
    ECHO Pass 5/9 - Rename
pushd %*
set /A cnt=0
for /F "delims=" %%j in ('dir /B /O t*.gif') do (
  rem echo cnt1=!cnt!
  set /a cnt+=1
  set cnt2=!cnt!
  if "!cnt2:~1,1!"=="" set cnt2=0!cnt2!
  if "!cnt2:~2,1!"=="" set cnt2=0!cnt2!
  if "!cnt2:~3,1!"=="" set cnt2=0!cnt2!
  if "!cnt2:~4,1!"=="" set cnt2=0!cnt2!
  rem echo cnt2=!cnt2!
  ren "%%j" "!cnt2!.gif"
)
popd

6. Trim, resize, rotate and compress

REM Crop, Trim, Resize
    ECHO Pass 6/9 - Border
    mogrify.exe -bordercolor #FFFFFF -border 30 "*.gif"

    ECHO Pass 7/9 - Trim
    mogrify.exe -bordercolor #FFFFFF -border 30 -fuzz 10%% -trim +repage "*.gif"

    ECHO Pass 8/9 - Resize
rem    mogrify.exe -resize ^^!600x800^^! -colors 16 -compress LZW -quality 30030 "*.gif"
    mogrify.exe -rotate -90 -colors 16 -compress LZW -quality 30030 "*.gif"

For the specific set of images I was converting resizing to 600x800 didn't give me the best results, so I commented it out. It may be useful for other sources, however. Also, I found I had to rotate the images to better use Kindle's screen real state, which may also not be ideal for other sources.

Depending on what you are converting you might want to disable rotation and/or enable resize.

7. Generate the final PDF including all GIFs in the proper order

REM Convert *.GIF into PDF
    ECHO Pass 9/9 - GIF to PDF
    pushd %*
    set gifs=
    set /A cnt=0
    for /F "delims=" %%j in ('dir /B /O *.gif') do (
        echo Appending %%j
        set gifs=!gifs! "%%j"
        set /A cnt+=1
        if !cnt!==600 (
            convert.exe -limit memory 128mb -limit map 256mb -compress zip -quality 100 -density 200 "out.pdf" !gifs! "out.pdf"
            set /A cnt=0
            set gifs=
        )
    )
    if !cnt! GTR 0 (
        convert.exe -limit memory 128mb -limit map 256mb -compress zip -quality 100 -density 200 "out.pdf" !gifs! "out.pdf"
    )
    popd

The step above usually is where this script takes the most time - on my Core2Duo desktop it took few hours to convert all the images into one PDF file. Since I left it running throughout the night it didn't bother me much. I also found that it required a good amount of free space in the disk, to store temporary files.

8. Clean-up of all temporary files

REM Remove GIF files leaving only the PDF
    del "*.gif"
    rd /q /s work

The commands above are used to remove all GIFs from the current directory, and not from any sub-directory. Then, it also removes the temporary "work" directory for a complete clean-up.

Some weird error messages may pop up while the script is running - this wasn't happening with the version of ImageMagick I used to write it, but found it is with the current stable version. Thankfully it does not affect functionality, as far as I could see

If everything worked properly, in the end of the process you will have in the current directory all your original sub-directories and the final PDF file.

So, for a quick test I processed about 5 Dilbert images and divided in two sub-directories, simulating multiple volumes we want to consolidate into a single Kindle-optimized PDF.

The path structure looked like this:
C:\Temp\convertToKindle - convertToKindle.bat
C:\Temp\convertToKindle\v1 - v1-1.gif, v1-2.gif
C:\Temp\convertToKindle\v2 - v2-1.gif, v2-2.gif, v2-3.gif

Then after I executed convertToKindle.bat I had a new file, out.pdf with all 5 images, looking like this:

Like I said, these settings work well for this particular source - depending on what you're converting and your e-reader you might need to play with it to find optimal settings for rotation and resize.

Tip - process multiple documents in one shot

If you have multiple sources and want to convert each on a separate PDF file you can follow these instructions to process all of them one at a time:

1. setup your directories like this:
- root
- source1
- source2
- source3

2. copy convertToKindle.bat to source1, source2 and source3

3. create a new Windows batch file named processAll.bat in the root folder and paste the commands below:

cd source1
call convertToKindle.bat
cd ..

cd source2
call convertToKindle.bat
cd ..

cd source3
call convertToKindle.bat
cd ..

Now if you execute processAll.bat it will execute convertToKindle.bat on every source directory, generating one PDF for each directory.

That's it, I hope this can be helpful for others.
Pedro

Update - 05/27/2010

I found few similar tools that may be easier / better to use available elsewhere. So, just in case this batch script seems too complicated you might also want to try one of these instead:

1. Mangle: http://foosoft.net/mangle/

2. KindleManga: http://code.google.com/p/kindlemanga/

3. Manga2PDF (Linux): http://www.mobileread.com/forums/showthread.php?t=12790&highlight=manga2pdf

4. ePubizator (Multiplataform): http://www.mobileread.com/forums/showthread.php?t=36677

Note: I did not try any of these tools, so your mileage may vary.

Quest to find the best Linux for slow computers, Fedora 12 LXDE upgrade to 13

2010-05-10T18:35:00.000-07:00

I replaced Kubuntu Netbook Remix (KNR) on my netbook with the LXDE spin of Fedora 12. KNR looks nice and has many nice features for devices with small screens but it was just painfully slow on my netbook. LXDE is (in theory) a more crappy-hardware-friendly Linux flavor, so I decided to give it a try.

LXDE's UI looks uglier and less flexible than KNR's, but performance is indeed better. With that in mind I put together a development environment based on a Fedora LXDE virtual machine I can use to test my own projects in Linux. Nothing terribly resource-intensive but things like Grails, Gradle, AppFuse, etc. I wanted to know if this VM would be fast enough so I could use it as my development environment.

Click on "Read more" below to check out the rest of this post on a development environment VM and instructions on how to update Fedora 12 LXDE with the unofficial release 13.

I know this approach adds some overhead when compared to a native development environment, but if the performance hit is not noticeable enough it could be interesting. For example, the next time I upgrade my laptop I can simply copy my VMs and keep my fully configured environment. Also, since all servers I work with are Unix-based, there would be more script reuse and less environment-specific issues (not that I get many of these, but some of my colleagues insist in hardcoding Windows-style paths sometimes). And lastly, I can keep project-specific VMs to organize my development environments. So far this approach has been working well for the past few days.

But I wanted to see how Fedora 13 LXDE is looking like (we still have few days before the official launch). Since I couldn't find where to download anything but the main version of Fedora 13 I upgraded a copy of my VM following these steps:

Update current libraries

su
yum update

Download and install Fedora 13 release RPM

a. Download this file: http://mirror.aarnet.edu.au/pub/fedora/linux/releases/test/13-Beta/Fedora/i386/os/Packages/fedora-release-13-0.7.noarch.rpm
b. Install it with the command below:

rpm -Uvh fedora-release-13-0.7.noarch.rpm

Verify the release has changed

cat /etc/fedora-release
Fedora release 13 (Goddard)

Upgrade all libraries to Fedora 13

yum upgrade -y

And after one hour or two I had a fully functional Fedora 13 LXDE VM. I just don't see lots of changes other than few icons and the wallpaper. I guess I will have to wait until the official launch... :(

NFJS 2010 - Northern Virginia Software Symposium II

2010-05-05T09:28:00.000-07:00

This is my last post about the No Fluff Just Stuff Software Symposium here in Reston, VA. Sad to say I didn't get any of the 3 iPads! I did see my name scrolling down for the last iPad (the most expensive model!) but the system had to pick someone else's name just 2 or 3 positions from mine! Oh well... guess I am not destined to get one, since I am not gonna buy one and my luck isn't gonna get me a free one either. This is something else I might consider buying, if it ever sees the light of day Notion Ink Adam.

Anyways, the symposium was amazing - Jay Zimmerman put together a great set of speakers and enough varied content for everybody. Thanks Jay! These are the sessions I attended, along with my comments:

Click on the "Read more" link below to check out my notes on Friday's presentations.

Friday

Implementing Evolutionary Architecture - Neal Ford

Testing the Entire Stack - Neal Ford

Smithing in the 21st Century - Neal Ford

Like I said on my last post, I really like Neal's presentations. He described REST in detail, and how we can use it to keep systems loosely coupled. Also, he gave some great recommendations on tools and strategies for testing different aspects of common web and desktop applications.

What Stops You From Delivering? - David Hussman

Discussed about ways to find your process bottlenecks and how to deal with them.

Saturday

Pragmatic Architecture - Ted Neward

Discussed about architecture and how oftentimes we either under or over-architect systems.

It is quite common in the J2EE world to find teams building systems using so many different design patterns just because they want to use it, not because it is the easiest / best way to solve the problem. In the other extreme we have teams that just go coding stuff without minimal or any architecture discussion.

In the end architecture should be defined at a high-level before we start coding things, otherwise it will be defined with much less control and may not end up being quite what you expected.

Architect for Scale - Michael Nygard

Software Architecture for the Cloud - Michael Nygard

I am a big fan of Michael since I read his book "Release It!: Design and Deploy Production-Ready Software (Pragmatic Programmers)
". His presentations focused on architecture for systems designed to handle volumes orders of magnitude higher than what I am used to. It was really interesting to glimpse on what kind of problems we may see soon, with more and more people using computer systems, and what kind of solutions we are using today to support it with high availability and performance. He described techniques such as process parallelization, architecture partitioning (sharding) and content delivery networks (e.g. Akamai) to support scalability.

Building RESTful Apps with SpringMVC - Craig Walls

He demonstrated a single SpringMVC-based web application with support for both browsers and REST web service clients. It was a good demo, I will try that later.

300 Sessions

This was cool, each presenter got only 5 minutes to present a topic. I wish they had shared these PPTs too.

Sunday

Cloud Computing - Rohit Bhardwaj

This one was really interesting - Rohit described and demonstrated how we can use Amazon EC2 and S3 as a low-cost approach for on-demand infrastructure services. He also demonstrated a simple Google AppEngine project, and compared the costs and specifics of each major player in the cloud computing world. It might be just me but I didn't know Google gave us free resources for our projects in Google App Engine! If you just want to test out an idea you can setup a virtual environment in no time, and as long you don't use lots of resources (up to 500MB, up to 5 million page hits per month, limited CPU use etc) you can set it up for FREE!

Google's offer is free but limited: no database (you have to use BigTable instead), not all parts of Java are supported (e.g. no Thread) etc. They basically hide the servers from you, so you can focus only on the application side.

Amazon's offer is not free but cheap, and they give you control over the infrastructure. You can try out different server architectures, create server templates etc without buying any physical infrastructure. This is great for temporary needs, like a performance test where after the test proved a set of hypothesis it can be discarded.

One approach is to build and test your app in Google, and then migrate it to Amazon or to your in-house infrastructure. As long you don't build close ties with Google's infrastructure that shouldn't be too complicated.

Now that I have cheap infrastructure I just need to have the killer idea on how to use it to make tons of money! Ideas anyone? Please? :D

Enter the Gradle - Ken Sipe

Gradle is like a new Maven, a project build system. From this presentation I got that it is still not mature yet for my purposes (though big projects are already using it) but it made me more interested in Groovy, the language used to build Gradle. I will definitely spend more time learning Groovy and Grails in good part because of this presentation.

Maintaining Source Code Quality - David Bock

David described how we can easily include project analyzers to determine code quality and use it as metrics towards better quality. Great stuff, but I used most of it on previous projects (PMD, FindBugs, CheckStyle etc). I did learn about some new source code analyzers though. He also pointed out the importance for enforcing a project-specific code style - it is not about cosmetic preferences, but to simplify merges in the source code versioning system.

Programming Clojure - Aaron Bedra

What else can I say about this one? I think I twisted my brain hahaha... really, this was the last session I attended in the conference, and the most complex one. Aaron had to consolidate a 4 hours presentation into just 1.5 hours, and while I think he did a good job at it describing Clojure's syntax my opinion is that it just made people want to avoid it.

NFJS 2010 - Northern Virginia Software Symposium

2010-04-30T20:33:00.000-07:00

I am attending the No Fluff Just Stuff Software Symposium here in Reston, VA. The schedule this year is really good, with names like Neil Ford, Jeff Brown, Matthew McCullough and so many other experienced speakers and book authors discussing things like REST, polyglot development with languages like Ruby, Clojure etc, cloud computing, testing, Agile development etc. Many of these guys are either book authors, code committers or both for really cutting edge technologies that are shaping up what most of us will be doing and working with years from now. Sometimes it is hard to decide which session you want to attend - at any given time during the event they have 6 parallel sessions going on!

I attended today 2 sessions with Neal Ford, "Implementing Evolutionary Architecture" and "Testing the Entire Stack". He is not just a gifted speaker but also makes it obvious he knows well what he is talking about from hands on experience. If you ever have the chance to attend one of his sessions at NFJS or some other symposium make try not to miss it, I am sure you will learn something you can use. His keynote "Smithing in the 21st Century", where he presented his views of how (and to what extent) we can predict the future and to where we are headed in the near future was great, and it was interesting not just for developers / architects but to pretty much anyone interested in technology.

Unfortunately I just found out about the symposium less than 2 weeks back so I couldn't convince my company to fund my tickets (who knows if they change their mind in the next weeks? that would be a good surprise :D). Despite of that, a good thing is that I got to do volunteer work helping the event organizers on some basic tasks (things like attendee registration, organizing the presentation printouts etc), got to meet some nice people and should even get some credit back. Also, NFJS will be giving away 2 iPads, books and other prizes at random for the attendees - who knows, if my name gets picked for one of the iPads that wouldn't be bad at all, especially because this is likely the only way I would ever get one! :D

I will let you know on Sunday if I got one of the iPads hehehe... along with my personal feedback of the sessions I attended. I know these things are subjective, but who knows, if you get the NFJS guys in your town this could help you pick your sessions.

Quick web app prototyping with AppFuse

2010-04-10T07:39:00.000-07:00

Quick web app prototyping with AppFuse

Sometimes we just need to put together a quick web app for prototyping, discuss screen flows, design layouts, create web services etc. I found AppFuse to be a pretty good option for this purpose.

You can follow the steps in this article to setup your environment and create a new web application with AppFuse. The best thing is that once your environment has all the requirements setup it is super quick to create as many new web apps whenever you need. Even if you aren't very familiar with things like AppFuse, Maven, Spring, Hibernate etc but still want to be able to generate new web apps, just try to endure the first steps - once you see the web app up and running on your computer I am sure you will be more motivated to learn it all through AppFuse.

Click on the "Read more" link below to get started with AppFuse and build a working web application in no time.

I used AppFuse 2.1.0 M1 on my examples for this article since this is what I would use if I had to start a new project today. If you prefer to use the last full release of AppFuse 2.0.2 or the snapshot release for M2 check out the additional sections at the end of this article.

Prerequisites

1. Java Development Kit 1.5.x or newer
You can download the latest one here:
http://java.sun.com/javase/downloads/widget/jdk6.jsp

2. Apache Maven 2.0.x (AppFuse 2.0.2) or Maven 2.2.x (AppFuse 2.1.0+)
You can download both versions here:
http://maven.apache.org/download.html

3. Any email server
You can download an open source implementation here:
Java Apache Mail Enterprise Server (JAMES) - http://james.apache.org/download.cgi#Apache_James_Server

An email server is not exactly required for you to use AppFuse but it is used by the standard automated tests. If this is your first time using AppFuse it is a good idea to start James just to avoid having to disable or modify jUnit tests.

No database is required (how cool is that?) if you can use one of the supported embedded database such as H2 or HSQLDB. Sure, you can easily configure AppFuse to use other databases such as MySQL or Oracle, but while this is a good idea for a real enterprise application it is out of the scope for this quick guide.

Eclipse (or any other Java IDE) isn't actually required if you just want to run AppFuse's starter web application - however, if you are planning on customizing it, play with changing configuration and properties files etc you really want to use it. I included instructions to setup the project in Eclipse but if you use some other IDE you might want to check AppFuse's website or forum for more information.

Environment setup

You will need to execute the steps in this section only once. After you set up all requirements on your development environment you will be able to create and start new web applications with just 4 or 5 commands, without even touching code or configuration files!

For this example I will install the JDK, Maven and James in "C:\Users\Pedro\Env\Java" and create my AppFuse-based projects in "C:\Users\Pedro\Projects" - but you can use whatever paths you want – especially if your name isn't "Pedro"! From now on I will refer to "C:\Users\Pedro\Projects" just as PROJECT_ROOT.

1. I always create a batch file (in Windows) or shell script (in Unix) to set up my environment variables. Usually this is a better practice than set environment variables only in the system configuration since it is easier to clone on other machines and make the project less coupled with the underlying environment. If you want to do the same, on PROJECT_ROOT create a file called setenv.bat with the content below:

set JAVA_HOME=C:\Users\Pedro\Env\Java\jdk1.6.0_17
set PATH=%JAVA_HOME%\bin;%PATH%
set M2_HOME=C:\Users\Pedro\Env\Java\apache-maven-2.2.1
set MAVEN_HOME=C:\Users\Pedro\Env\Java\apache-maven-2.2.1
set PATH=%MAVEN_HOME%\bin;%PATH%
set MAVEN_OPTS=-Xms256m -Xmx512m

Remember to replace the paths in setenv.bat described above with the ones you use on your environment.

Let's create a new web project!

1. Start your mail server:

Execute C:\Users\Pedro\Env\Java\james-2.3.2\bin\run.bat

2. Open a command prompt, go to PROJECT_ROOT and execute setenv.bat

3. Create a new AppFuse project:

AppFuse supports different combinations of various frameworks - in the example below uses Spring, Hibernate and JSF. If you want to use other frameworks you can check out AppFuse's official site for the appropriate Maven commands.

Also, note that the first time you create an application with AppFuse it will take Maven several minutes to download all the dependencies from the internet into your local Maven repository. However, the next times since it will already have the JARs the build will finish much faster.

[AppFuse 2.1.0 M1]

On the same command prompt window copy and paste the commands below to generate the web application:

I am using simple names in the example below, but for a real project you might want to replace "com.mycompany" with a more meaningful Java package, and "appfusem1" with your project name.

mvn archetype:generate -B -DarchetypeGroupId=org.appfuse.archetypes -DarchetypeArtifactId=appfuse-basic-jsf-archetype -DarchetypeVersion=2.1.0-M1 -DgroupId=com.mycompany -DartifactId=appfusem1
cd appfusem1
mvn appfuse:full-source
mvn -P hsqldb
mvn -P hsqldb install

Because of the many tests being executed as part of the build process it is normal for warning and error messages to popup in the command prompt, just give it some time and wait for it to complete.

If you got a SUCCESS message like the one shown below your web application was generated correctly and it is ready to run.

Build, test and run:

mvn -e -P hsqldb jetty:run-war

Wait for the message highlighted above.

Open your internet browser and go to URL "http://localhost:8080/".
You can login as username "admin" password "admin" or username "user" password "user".

Disable database rebuild, unit and UI tests

Database rebuild, unit and web tests are enabled by default, which means that even with the JARs the builds will take some time to complete. It is a good idea to run and keep these tests in sync with the code so you can avoid surprises, but for times when we just want a quick build we just need to add property "-DskipTests=true" in the command line to disable it all.

For example:

mvn -e -P hsqldb jetty:run-war -DskipTests=true

Import project in Eclipse:

You can use Maven to generate Eclipse's .project file:

mvn -P hsqldb org.apache.maven.plugins:maven-eclipse-plugin:2.5.1:eclipse

Open Eclipse with PROJECT_ROOT as the workspace. If Eclipse does not ask you for the workspace when it starts, go to File / Switch Workspace / Other and enter the path to your PROJECT_ROOT

In Eclipse, select File / Import, expand General and select Existing Project into Workspace

Click Next and for the root directory just click Browse and Ok to select the current workspace, make sure your new project is listed and checked, and click Finish.

Once the import is finished you will see many dependency issues – to correct this you must add a variable classpath in Eclipse named "M2_REPO" pointing to your Maven repository
eg. M2_REPO=C:\env\.m2\repository

You can do that via Eclipse: right-click the project root node, select Java Build Path, Libraries, Add Variable, New, and create M2_REPO pointing to your Maven repository.

You can also do that using Maven: in the command prompt window just execute the command below:

mvn -Declipse.workspace=C:\Users\Pedro\Projects eclipse:add-maven-repo

Instructions for different versions of AppFuse

[AppFuse 2.0.2]

mvn archetype:generate -B -DarchetypeGroupId=org.appfuse.archetypes -DarchetypeArtifactId=appfuse-basic-jsf -DarchetypeVersion=2.0.2 -DgroupId=com.mycompany -DartifactId=appfuse202
cd appfuse202
mvn appfuse:full-source

To correct a minor bug with AppFuse + embedded databases in Windows, edit and replace in pom.xml:

<jdbc.url><![CDATA[jdbc:hsqldb:${java.io.tmpdir}/appfuse202;shutdown=true]]></jdbc.url>

with:

<jdbc.url><![CDATA[jdbc:hsqldb:tmp/appfuse202;shutdown=true]]></jdbc.url>

mvn -e -P hsqldb
mvn -e -P hsqldb install eclipse:eclipse

How to disable database rebuild, unit and UI tests in 2.0.2: add "-Dmaven.test.skip=true". For example:
mvn -e -P hsqldb jetty:run-war -Dmaven.test.skip=true

[AppFuse 2.1.0 M2 SNAPSHOT]

mvn archetype:generate -B -DarchetypeGroupId=org.appfuse.archetypes -DarchetypeArtifactId=appfuse-basic-jsf-archetype -DarchetypeVersion=2.1.0-M2-SNAPSHOT -DgroupId=com.mycompany -DartifactId=appfusem2
cd appfusem2
mvn appfuse:full-source
mvn -P hsqldb
mvn -P hsqldb install eclipse:eclipse

J2EE Performance Tuning

2010-03-17T08:00:00.000-07:00

Found a very good overview on "How to Diagnose a Performance Problem in a J2EE System" written by John Bley: http://weblogic.sys-con.com/node/43024

Even though this article was written in 2003 somehow it is still pretty up to date. Guess the ways we build our systems are not changing so fast afterall.

Virtual Machines - managing virtual drives

2010-03-10T16:33:00.000-08:00

From time to time I have to expand the disks on my virtual machines. Or convert disks from preallocated 9+GBs into sets of 2GB files (so they fit in FAT32 USB sticks, for example). I found this tool vmware-vdiskmanager to be the best way to manage these tasks, but it does not come bundled with VMWare Player. It does come, however, in the also free-as-in-beer VMWare Server. So if you need it, download and install VMWare Server here so you can use the examples below.

# VMware Virtual Disk Manager Utility (vmware-vdiskmanager.exe):
- Documentation: http://www.vmware.com/support/ws55/doc/ws_disk_manager_running.html

# Examples:

- Convert existing virtual disk from fixed preallocated size to growable, split into 2GB files (trust me, it helps wonders to set all your virtual disks this way to avoid future headaches):

C:\"Program Files"\VMware\"VMware Server"\vmware-vdiskmanager.exe -r data.vmdk -t 1 data-1.vmdk

- Defragment existing virtual disk (unless you have a SSD you should do this every now and then both in the VM and in the host OS):

C:\"Program Files"\VMware\"VMware Server"\vmware-vdiskmanager.exe -d data.vmdk

- Shrink virtual disk:

C:\"Program Files"\VMware\"VMware Server"\vmware-vdiskmanager.exe -k data.vmdk

- Expand virtual disk:

C:\"Program Files"\VMware\"VMware Server"\vmware-vdiskmanager.exe -x 32GB data.vmdk

Note: when you expand an already partitioned virtual disk it will not show immediately the new size when you boot your virtual machine. However, now you can use tools inside the virtual machine to expand the partition so the operating system can see and use it. In Linux, for example, you can do this:

df <- run this to list all mounted partitions
Filesystem ... Mounted on
/dev/sda1 ... /
/dev/sdb1 ... /data <- this is the partition I want to expand

Note: the block below worked for me, but as always, MAKE BACKUPS OF EVERYTHING if you don't want to risk losing your precious data. Or don't backup, but don't blame me if you mess up and lose your whole filesystem...

login as root and run these commands:
umount /dev/sdb1

fdisk -l /dev/sdb - write down the start and end cylinders for the target partition

fdisk /dev/sdb

in fdisk use the [d] command to delete the partition - yeah, I know, this is scary, but if you REALLY took the time to backup your virtual machine then go ahead; if everything works your data will not be lost in fdisk

use the [n] command to add a new primary partition, using the same partition number and start cylinder, but use the default value for end cylinder for it to use all the space you made available in fdisk

use the [w] command to write the partition table to the disk and exit - if it asks you to restart your server, do that before proceeding, otherwise keep going

login as root again and run these commands:

e2fsck -f /dev/sdb1

resize2fs /dev/sdb1

mount /dev/sdb1

df -h

If everything worked as planned you should now have a bigger partition, and all your precious data should still be there. If something didn't work, well, you still have your backups, right? RIGHT? ;D

Solid State Drives - Part 2

2010-03-06T08:59:00.000-08:00

On my previous post I tried to convince you that it makes sense to buy an overpriced storage device called SSD if you use computers for a living. Now, to conclude my discussion on SSDs I will go into more low-level details such as:

1. Why I bought specifically Intel's I-25M?
2. How I installed it, and why I cloned my old drive instead of starting it "fresh"?
3. What are the tweaks one needs to apply on a XP machine with SSD?
4. Good comparison of SSDs x HDDs

--------------------

1. Why I bought specifically Intel's I-25M?

- Quick answer: because it currently has the best value if you weight compatibility, performance and price among the top SSDs.

- Verbose answer:
If you are joining the SSD bandwagon you should try to pair it up with a modern operating system, such as Windows 7. Also, plan on doing a clean install of the OS instead of imaging your current hard drive into the SSD. That will simplify things a lot, since older operating systems were not designed with SSDs in mind and may cause issues ranging from sub-optimal performance to even significantly decrease the life of SSDs.

Sadly I could not follow this advice myself - I wanted to install my SSD on my work laptop, since this is what I use 99% of the time for "money-making" tasks. Ya know, unless we rob a bank or do some other illegal stuff we need to do something they call work to get money to buy cool stuff like electronics. My company currently doesn't allow me to use Windows 7 yet, and I didn't want to mess with my VPN. Not having the option to switch my work computer to Windows 7 I had to figure out a way to make it work with XP.

Some SSDs do not support XP well. More specifically, many of them only support something called TRIM in Windows 7. You can look for TRIM and SSD on Google or Wikipedia, but it's enough to say that you NEED the ability to run TRIM at least every week or month, otherwise your SSD's performance and life will sink fast. Intel's SSD works pretty well with XP - in fact, their SSDs are one of the simpler devices to use, requiring the least amount of tweaks to work in top shape with XP. Sure it also helps I got a decent deal from Newegg the last Black Friday! :D

2. How I installed it, and why I cloned my old drive instead of starting it "fresh"?

I had to find a way to clone my original HDD into the SSD so I would not have to reinstall everything, including apps, network settings and more importantly the VPN. Then, after digging through zillions of forums I found that the most important thing you need to pay attention to when cloning XP drives into an SSD is something called partition alignment. You can look it up, but basically it has to do where the XP partition starts in the disk, and for you to get the best performance it must start on a sector that is a multiple of your SSD's page size. Since XP wasn't built with SSDs in mind it always start its partitions on a position that doesn't match the start of a page in your SSD. What this means is that when XP needs to read or write something to the SSD, the SSD needs to do double the work to access two pages instead of one. Some SSDs handle it better than others, but it is a performance constraint nevertheless. If you partition your SSD using Vista or Windows 7 you should be safe, these operating systems always start a partition on sector 2048, which is compatible with most SDDs out there.

Note that partition alignment can give a good performance boost even for regular HDDs, as described by Microsoft here (http://www.microsoft.com/technet/prodtechnol/windows2000serv/reskit/prork/pree_exa_oori.mspx?mfr=true). For (much) more detailed information on benefits of partition alignment for SSDs see here (http://www.ocztechnologyforum.com/forum/showthread.php?48309-Partition-alignment-importance-under-Windows-XP-%2832-bit-and-64-bit%29-why-it-helps-with-stuttering-and-increases-drive-working-life).

- SSD partitioning:
To ensure proper partition alignment I used the Disk Management tool from my Windows 7 computer to remove the old and re-create a new partition on the SSD. This tool comes bundled with Windows 7 and Vista, just go to Control Panel, Administrative Tools, Computer Management. There I removed all partitions from the SSD (it was a new drive, so I had no data to lose), created a new partition and formatted it with NTFS.

- Disk cloning:
I first used a free tool called Macrium Reflect (http://www.macrium.com/reflectfree.asp) to backup my original HDD. This tool is great for disk image backups, you can even mount the image as a virtual drive in Windows Explorer if you need to retrieve files without imaging a disk. Unfortunately, I found that every time I imported the original HDD image into the SSD Macrium was also bringing the bad alignment from the original disk. I could not find a workaround for it, but at least I was happy to backup the original disk in case something bad happened.

Looking for a (free) solution I found on some forums people recommending a trial version of a tool called Drive Snapshot. It didn't work for me for some reason, like Macrium this tool was reverting my SSD to the original bad alignment.

Then I found something that finally worked, a free tool called XXClone (http://www.xxclone.com/). I thought it wouldn't work, it seems the latest version of this tool was released in 2007. I was surprised to find that the SSD was able to boot after XXClone, and better yet, the partition was still properly aligned!

3. What are the tweaks one needs to apply on a XP machine with SSD?

Ok, so after cloning was completed I got the disk to boot on my laptop. Now it was time to tweak XP so it would work with the best performance and wouldn't decrease the life of my SSD.

Below are the changes I made to the XP registry for it to work better with the SSD. If you want you can just copy/paste this block into a new file called "SSD.reg" and double-click it to apply the changes. While it worked fine for me I WON'T BE LIABLE IF IT BREAKS YOUR COMPUTER, SO USE IT ON YOUR OWN RISK.

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\PrefetchParameters]
"EnablePrefetcher"=dword:00000000

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Dfrg\BootOptimizeFunction]
"Enable"="N"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\OptimalLayout]
"EnableAutoLayout"=dword:00000000

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management]
"LargeSystemcache"=dword:00000001

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management]
"DisablePagingExecutive"=dword:00000001

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management]
"ClearPageFileAtShutdown"=dword:00000000

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem]
"NtfsDisable8dot3nameCreation"=dword:00000001

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem]
"NtfsDisableLastAccessUpdate"=dword:00000001

[HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Explorer]
"AlwaysUnloadDLL"=dword:00000001

4. Good comparison of SSDs x HDDs

On my previous post I presented some synthetic benchmarks taken with CrystalMark. That really doesn't mean much in terms of real life experience. To get a glimpse of how SSDs perform when compared to regular HDDs take a look at videos like these on YouTube:
- http://www.youtube.com/watch?v=T_Jz7IMwBt4&feature=related
- http://www.youtube.com/watch?v=96dWOEa4Djs&feature=related

Solid State Drives - Part 1

2010-02-21T00:05:00.000-08:00

I have been reading a lot about Solid State Drives (SSDs) for quite some time. SSDs are storage devices we can use to replace standard Hard Disk Drives (HDDs), but offer less space and cost much much more.

HDDs are based on mature (i.e. old) technology, basically consistinting of multiple spinning magnetic disks read and written by moving heads. Suppose you want to read a part of a certain file: the reading head must move to a specific track / cylinder and wait until the data in the spinning disk passes just under the reading head. This procedure of one physical move operation and then wait for the data is called seek time. And as much it sounds complex, it is actually amazingly fast in modern HDDs - both the heads and the disks are freaking fast. It still, however, takes some time, and by relying on moving parts this whole mechanism is quite sensitive to impacts. When my external Maxtor 500GB USB harddrive fell on the table (yes, it didn't drop on the floor or anything, it just fell from vertical to horizontal position on the table!) I could only imagine its disks spinning at 7200 RPM with the writing heads scratching on everything just like an old vinyl LP! Poor HD, it is now on the HD heavens, you will be remembered!

SSDs currently offer less space than HDDs, but since it has no moving parts (it is based on the same memory we use on digital cameras, small USB drives etc) it would still be "alive", one point to SSDs here. But also, without heads and spinning disks etc SSDs run cooler, use less energy (great for laptops) and can get the data without any delay - the seek times are ridiculously fast! Or so the manufacturers say...

It took me some time to find courage (i.e. $$$) to try it myself, but I finally got one on last year's Black Friday. Then I had to read even more to set it up properly. After 4 months using it every day in one of my work laptops I am certain that SSDs are to become mainstream in the next couple years, with falling prices and improved performance. Though they may not replace HDDs anytime soon, it makes a lot of sense to setup a mixed system, with SSDs to store the operating system and programs, and HDDs to store music, photos, movies, backups etc.

Now, if you are interested in more comprehensive reviews and bechmark comparisons, I am not dreaming of competing with sites like AnandTech (http://anandtech.com/storage/showdoc.aspx?i=3747&p=1), ARS Technica (http://arstechnica.com/hardware/news/2009/03/anand-investigates-ssd-performance-shenanigans.ars) or Tom's Hardware (http://arstechnica.com/hardware/news/2009/03/anand-investigates-ssd-performance-shenanigans.ars) - but, given how complex and expensive SSDs currently are, I wanted to share my own perspective as a Java developer.

# SSDs benefits:
- read and write speeds are much better than HDDs
- more resistant to shocks due to no moving parts
- when its lifetime is over you can still read your data and transfer it to other storage devices - something that is not easily accomplished with HDDs in most cases

# SSD drawbacks:
- still costly, but prices are going down as new and better products are being offered
- less storage space - unless you have lots of money to burn you may have to get used to work with less space in your laptop than what you have now
- many operating systems, especially older ones (XP, this is you!), were not designed to use SSDs efficiently, causing sub-optimal performance and increased wear
- limited lifetime - due to technical limitations in flash memory, over time it will stop being able to write new data. No one knows exactly how long each of these SSDs last. All manufactures provide some bogus unrealistic metrics, but if I had to guess it would not be much longer than 3 to 5 years

I said I would not make benchmarks but just to give you a glimpse of how faster SSDs are I posted these two images below. In the left we have a 160GB Western Digital HDD, and in the right we have a 80GB Intel X-25M - both tests executed on a Dell Latitude D630 laptop.

The Seq row represents scenarios similar to when we are reading / writing big files (i.e. a DVD movie) to the disk. The 4K row is, however, far more significative for the overall disk performance - it represents reading / writing of small chunks of data to the disk, an activity that is far more common on a workstation throughout the day.

Without getting into further details, I think it is pretty evident that SSDs are significantly faster than common HDDs. Should you buy one? Well it depends... but you can be sure it made sense for me to buy it! To explain why I believe in that, these two are the biggest concerns people have with SSDs, and also the main reasons why I bought it:

1. "SSDs are too costly" - I work with computers 8 to 10 hours per day. Every day. And when I am not working for my employer I am likely to be testing something for myself to learn new tricks. I always need to keep few resource hog applications up all the time (e.g. Oracle, Eclipse, virtual machines etc), which causes a lot of disk I/O. When I build a complex project with Ant or Maven it can take around 5 minutes - yeah, it seems fine, but sometimes I need to do that many many many times during the day, and you can guess it adds up fast.

Now, let us assume the SSD costs $200 and that my employer pays me $40 / hour. If the SSD's improved performance saves me 15 minutes per day in project builds, application startups etc, it would take me only 1 month to "recover" my investment. And after that, it is just profit - boom baby! Also, because I can now provide consistent faster results, especially on tasks associated with high disk I/O (e.g. build virtual machines, install Oracle etc), perhaps my boss will have good news on my next performance review and give me a raise! Yeah, I know I am dreaming now...

2. "SSDs will die after 5 or so years" - Ok, I wish you could see my face of "not caring". Seriously, in 5 years I want to be able to buy a few petabytes of quantum-based storage for the same $200. By then I will just not care if a 80GB or 160GB SSD is not able to write anymore. Even its speed, which is great for todays standard, is going to be dog slow!

Ok, now that we agree it makes sense in some cases (e.g. mine) to replace HDDs with SSDs, which one we should buy? Man, I wish things were simple... even though we have a gazillion companies selling these things, it is still something I would never ask my father to buy and install one by himself, and he plays with computers since I was 6 years old.

First, you need to pick the right SSD - and like I said, there are too many companies selling them. And I can tell you, you REALLY do not want to pick one of the bad SSDs - many of them are worse than regular HDDs, and they get even worse over time! Since this is not a super consumer-friendly (i.e. plug-and-play-and-forget) kind of device, your best bet is to stick with the top 2 vendors - the ones with most modern technology and the biggest number of fellow "SSD-ers" to help you on forums if you ever need help. Currently I consider Intel and OCZ as the two top contenders in the SSD market, with Intel having the upper hand in value. OCZ's top model (Vertex LE) is somewhat faster than Intel's I-25M G2 but it is way more expensive - the 100GB costs around twice as much of what I paid for Intel's 80GB SSD! Sure, there are others (e.g. GSkill, SuperTalent etc), but at this time I think it is just safer to stick with one of these two. Kingston entered the market re-branding Intel drives, so apart from the niceties in the Kingston bundle they count as Intel hardware.

Intel has two interesting models, the I-25M with 80GB or 160GB. Both use Intel's own controller.

OCZ has just too many different models and capacities, but the most popular ones seem to be the Vertex (Indilix controller) and Sumit series (Samsung controller).

If you really want to jump in the SSD bandwagon I recommend you keep an eye for deals on sites like Amazon, Newegg, SlickDeals, DealDump etc (see my previous post about the Black Friday for a more deals sites) for either one of these 3 models (Intel, Vertex and Summit) with the storage size your wallet can afford. These short-lived deals can make a good difference for your wallet, or even allow you to get a bigger drive for the same $$$.

Ok, this post ended up quite longer than I expected. I will leave for my next post to describe how I installed, setup, cloned and tweaked the SSD on my laptop, and why I picked Intel over other alternatives.

Java magic with AspectJ

2010-02-04T07:10:00.000-08:00

Sometimes in our programming projects we can start things from scratch and implement them exactly how we want them. In these happy and creative times we so much have control of everything, if we don't like some of our own designs, APIs or mechanisms we can just go and change it, simple like that. Bad thing oftentimes we aren't so lucky, and we have to work on more constrained settings, using existing crappy code and macaroni architectures. In times like these we gotta be happy if we at least have the source code we are supposed to extend / use!

Recently I had to work on a project whose a big chunk of the code was implemented by another team. And we were not supposed to change that code. But we were still somehow supposed to change the exception handling and logging for a certain kinds of Java classes on both that codebase and our own custom code.

Since we are on a pretty bad economy, instead of spending time to update my resume I thought of using this as an opportunity to test and learn AspectJ. It was a long shot but I read about it in the past, but never had a strong need to use it until now and thought it was worth trying. The thing is, I was interested in it a while back but thought it would be just like other technologies, like Spring, Hibernate, JSF etc. Think about it, if you need to do real work and have only 2 or 3 days, can you do something with Spring + Hibernate + JSF from scratch if you never worked with it before? The learning curve for most of these things is just too high if you need quick results!

Man, I am glad I tried AspectJ. I know its inner workings use some arcane technology (bytecode manipulation isn't for the faint of hearth), but it was SO much easier than I expected to hook my custom exception handling logic in the existing code I felt it was almost too easy to be true! :P

Since I think other people may benefit from this mechanism let me share some basic but still useful information about it. In a nutshell AspectJ is an arcane technique to alter behavior of existing Java binaries. You can transparently inject logic into certain "point cuts", like when a certain method name is executed, or when classes from a certain package are loaded etc etc. Few cool examples of what you can do with that are:
- a custom global logging mechanism to existing classes from a certain package (e.g. my.company.*)
- custom exception handling mechanism for certain exception types
- method profiling, for performance monitoring and tuning
- reduce LOC and complexity on your new code using aspects to handle multiple functional concerns transparently, outside your business classes

There are 2 main ways of using AspectJ:

1. Static-weaving: You can use AspectJ's ajc compiler to alter existing class files (either .java or .class), injecting your custom aspects and pack all changed files into a new JAR file. Then you put the new JAR in your classpath instead of the old classes (in most cases it may also work to just put the JAR first in the JVM's CLASSPATH). The Eclipse team divides this method as Compile-Time Weaving, when ajc compiles *.java into *.class injecting the aspects, and Post-Compile Weaving, when ajc injects aspects into pre-compiled class files. The performance penalty here is 0, it will be the same thing if you recompiled the original classes with your aspect's logic in it.

2. Load-time weaving: This is where AspectJ's magic is more evident - your original classes and JARs are modified when they are loaded by the JVM! You do not have to replace any JARs, it will all be done dynamically. In most cases the performance overhead imposed by AspectJ to "weave" your aspects in the existing classes is negligible when loading the classes, but note that there will be no runtime overhead - once AspectJ injects your aspects it will not mess with that class again.

If I have to choose between one of the 3 methods described above I would try Load-Time Weaving (LTW) first, because I think in most cases the performance overhead during class load times (note it is "load times" and not "run time") should not be enough to overshadow the benefits of not having to bother updating JAR files every time the original class must be updated. If there is a significant performance hit due a broad scope of weaved classes then it is easy to move to one of the two static weaving methods described in item 1 above.

Ok, enough talk, how can we actually use it? Let me describe it with a very simple example that can serve as the basis for whatever else you want to do with AspectJ: (yeah, you guessed) a custom logging and performance monitoring mechanism!

# Dev env setup

1. Download and install AspectJ Development Tools (AJDT) for Eclipse. You can download it here.

2. Download my AspectJ examples project here.

3. Extract the AspectJ examples project somewhere in your development environment and import it in Eclipse.

# Project notes

1. "test.Test.java" is a simple test class. Their methods will be intercepted by AspectJ to transparently inject additional behavior without us having to change any code in Test.java.

2. Our two aspects LogAspect.aj and MethodProfilerAspect.aj are located in package "aspects".

3. Eclipse with AJDT show us what aspects affecting each method in class Test.java in the Cross References view (we can see it in the screenshot above, in the bottom right). Just select a method in the Java editor and it will list the aspects in that view. Or select an aspect method in LogAspect.aj and it will list the intercepted methods in that view. This is pretty useful to control scoping, just keep in mind that AJDT will only list methods in its own classpath, and it may be different (i.e. the aspects may catch more classes) when you deploy the aspect JAR in the app server.

Without any aspects, this is the output of test.Test.java:

Test.method1
Test.method2
Test.method3

Enabling just LogAspect.aj we then get this output:

[LogAspect] before public static void test.Test.main(java.lang.String[])
[LogAspect] param[0]='[Ljava.lang.String;@12ac982'
[LogAspect] before public void test.Test.method1()
Test.method1
[LogAspect] after public void test.Test.method1()
[LogAspect] before public java.lang.Object test.Test.method2()
Test.method2
[LogAspect] after public java.lang.Object test.Test.method2()
[LogAspect] returned 'null'
[LogAspect] before public java.lang.Object test.Test.method3(java.lang.String)
[LogAspect] param[0]='InputValue'
Test.method3
[LogAspect] after public java.lang.Object test.Test.method3(java.lang.String)
[LogAspect] returned 'ReturnValue'
[LogAspect] after public static void test.Test.main(java.lang.String[])

And finally, enabling just MethodProfilerAspect.aj we get this output:

[MethodProfilerAspect] entering public static void test.Test.main(java.lang.String[])
[MethodProfilerAspect] entering public void test.Test.method1()
Test.method1
[MethodProfilerAspect] exiting public void test.Test.method1() in 31ms
[MethodProfilerAspect] entering public java.lang.Object test.Test.method2()
Test.method2
[MethodProfilerAspect] exiting public java.lang.Object test.Test.method2() in 62ms
[MethodProfilerAspect] entering public java.lang.Object test.Test.method3(java.lang.String)
Test.method3
[MethodProfilerAspect] exiting public java.lang.Object test.Test.method3(java.lang.String) in 16ms
[MethodProfilerAspect] exiting public static void test.Test.main(java.lang.String[]) in 109ms

If you look at the aspects' source code you will see how little coding was required for that. But this is just the tip of the iceberg, there are many other cool things we can do with AspectJ, like dynamically change the return value of certain methods, catch, modify and re-throw Exceptions etc.

So this is pretty powerful... but "with power comes responsibility" - you must be careful and not depend on too many aspects, or your project could become a maintenance nightmare!

# Additional interesting articles about AspectJ

Pointcuts and Advices: http://www.eclipse.org/aspectj/doc/next/adk15notebook/ataspectj-pcadvice.html
Join Points and Pointcuts: http://www.eclipse.org/aspectj//doc/next/progguide/language-joinPoints.html

Black friday madness

2009-12-18T18:30:00.000-08:00

November, sweet november... month of the most important of holidays on Earth - the Black Friday! When all geeks can join in a mad rat race for the best deals, from clothing to electronic gadgets. For about 3 days people get in lines throughout the night, take enough red bull and 5 hour energetic drinks to glow green in the dark just to save some dough for the holidays. This kind of sacrifice may sound stupid for those who never experienced it, but since I came here to the US I can say this is the funniest holidays ever. Which other holiday can we find 47' HDTVs for less than half the regular price? Or get into a Walmart of Target store and see people fighting over cheap doorbuster deals? And hordes of geeks fighting online and killing web servers with DOS attacks trying to purchase half-price electronics? It is not for the faint of hearth, that is for sure!

This has been my approach during the past Black Fridays:
1. Monitor "official" BF sites for leaked ads weeks before the Black Friday - the ones I usually look into are:

• http://bfads.net/
• http://www.blackfriday.info/
• http://www.black-friday.net/

2. Based on the leaked ads, make a list of your targets - when the time comes there is no room for you to decide what to buy, if you hesitate you lose the deal!

2. Create a webmail account just for deals and other junk - apply for email lists on every store you can think of. Stores sometimes send good deals in its email lists and this is sometimes the fast way to score it! Just make sure you create a "junk" email account to use only for that purpose or set rules in your existing email client (webmails like hotmail and gmail are good at that too) to move all emails from these companies to a separate folder. Some stores I get deals via email:

• Newegg - http://www.newegg.com/
• Amazon - http://www.amazon.com/
• BestBuy - http://www.bestbuy.com/
• Target - http://www.target.com/
• CostCo - http://www.costco.com/
• TigerDirect - http://www.tigerdirect.com/
• CompUSA - http://www.compusa.com/

And so much more... some stores also give you discount coupons when you signup, so make sure you do not overlook it.

3. Monitor deal sites and forums for the best deals - the ones I use the most are:

• DealDump - http://www.dealdump.com/
• SlickDeals HotDeals forum - http://slickdeals.net/forums/forumdisplay.php?f=9
• FatWallet - http://www.fatwallet.com/

* - make sure you also follow the BF forum at SlickDeals - you need all the information as fast as possible!

4. And lastly, refrain from sleeping. This is key - if you sleep during the Black Friday week you WILL lose good deals. If you sleep during Thanks Giving / Black Friday / Saturday you will lose the BEST deals! In times like these you can't rely on a cup of coffer, you need something "military grade" - in times like these 5 Hours Extra is your best friend! :D

5. Some brick-and-mortar deals are inviting - but think wisely before stepping into the streets. Many times the online stores match or even top the deal. Also, shopping online you can easily cover dozens of stores - while other people are in line for one single store!

I hope you already figured I made a few jokes so far, I am not that crazy to spend 8 hours in line during a freezing night like some people to score a TV or laptop for peanuts. But for those who still think I am a materialistic person, I guess that may be true... but I have to say that most of the Black Friday fun is actually in digging and "fighting" (by fighting I mean trying to get your order thru, not actually fighting a dude over a TV!) for deals than actually spending tons of cash on stuff. And if you are picky with what you buy you do not need to spend a lot on stuff, spend good time with friends - I have had lots of good times with the wife and a friend on Black Fridays - and get out of it with good stuff you can use.

I just wish I had the idea to write this post BEFORE this years' BF... oh well, perhaps no one will notice if I copy / paste it for next year's BF hehehe :D

Setup Oracle's SGA size

2009-12-13T18:00:00.000-08:00

I always work with Oracle databases on my laptop. I prefer to install them myself so I can tune it to run without using all my memory. Also, instead of installing Oracle directly on my laptop I always do so on virtual machines because this way it is easier to share my environments with my colleagues and to setup individual sandboxes for each project I am working on.

When I have to work with databases installed by others I found I usually have to decrease how much memory is assigned to Oracle. The scripts below describe how to do just that.

• update Oracle's SGA size:

-- show current SGA size
select sum(value)/1024/1024 from v$sga;

-- set SGA target size to 256MB
alter system set sga_target = 256M scope=both;

-- list buffer size
select current_size from v$buffer_pool;

-- list all pools
select pool, sum(bytes)/1024/1024 Mbytes from v$sgastat group by pool;

-- additional Oracle memory settings:
-- make sure you don't setup SGA to the same or higher size than MEMORY
alter system set memory_max_target = 512M scope=spfile;
alter system set memory_target = 512M scope=spfile;
alter system set sga_max_size = 256M scope=spfile;
alter system set sga_target = 256M scope=spfile;
-- you will need to restart Oracle for the changes to take effect (e.g. dbshut and dbstart)

UML diagram editors

2009-12-11T23:07:00.000-08:00

From time to time people ask me to write UML diagrams. Sometimes it makes sense, but oftentimes I could be better off using my time on something else, like writing code, prototyping etc. Please don't get me wrong, I am not against good documentation or UML per se, but many people like to focus too much time on pretty documents with beautiful colorful diagrams while others, more pragmatic ones, prefer instead to have a system built on time, with quality and running in production. I like to think of myself as part of the "pragmatic" group.

I KNOW documentation is important, it can make the difference of people using your software or not, or your development team implementing what your client actually needs instead of something completely outlandish. That is not my point. My point for this post is that we should generally try to stick with what we are good on, but sometimes we just dont have a choice and have to work on something we KNOW isn't one of our strong skills. I thought PMs were supposed to limit such times, making sure tasks are assigned to the best resources based on their talents, but from my experience this is not exactly how it generally works. Not necessarily because of the PMs, sometimes they just don't have better resources to tackle the job.

But regardless what I think, as much I try to avoid wasting a lot of time writing documents that no one will read, sometimes I have no choice and just have to do it. I guess that is why they call it work, we can't like all that we do for a living. But what I can do is to try to finish it as quickly as possible, while still developing good quality documents. Which bring us to graphical design talent... and this is one area of talent that, I must confess, I am not especially gifted. I've worked with people born with natural talent for graphical design, they could use pretty much any tool and come up in few minutes with amazing diagrams or images or web site layouts etc. Think like someone opening Microsoft Paint and drawing a Mona Lisa in few minutes - yeah, I know, that was a stretch, but the point is, this is one of the things in life that you are either born with or you are not. Sure you can learn techniques to improve, but your true potential is more or less apparent from the early start.

Now the question is, how someone who's not a superfast gifted designer can create decent looking diagrams fast? My answer is "with especialized tools that format most things for you and you just have to enter the data". Think something a bit more especialized than Microsoft Visio. I can spend hours on Visio, moving lines and boxes, changing colors etc, and no matter what I do on it it will always end up looking like a pig. Before people say I did not try enough I can say that I tried learning from articles and books about it, but after some time I just accepted that I'm better of spending my time designing server-side frameworks, integrating enterprise applications, working with web services etc than creating striking-looking diagrams and working with web design. Which is actually quite liberating, to accept that I should leave such things for other people more gifted than myself while focusing on what, I like to think, am good on and like a lot to do. We should not try to do everything, but to focus on what we LIKE to do, right?

So what can I do when I need to write diagrams so they at least look decent and describe the concepts and ideas I am trying to document? And are finished quick so I can move on to other tasks? Well... tools that automatically generate the diagrams for me is a good start, I would think.

For example, NetBeans can automatically generate a UML class diagram from your source code. The generated diagram will be complex and unless you do some additional editing to filter out some of the noise no developer would ever waste time reading it, they would obviously prefer following the source code instead (assuming your source code is decently written - it is, right??), but it does look decent to see on a document or presentation. For those who dont know, it would appear that you put a lot of time on it... yeah, right! :P

Ok, so NetBeans is fine... but most times when I need to write UML diagrams I still do not have the classes yet, I am still discussing the solution design with a team. So I need something more flexible than that. But not too flexible that I would have to spend ages moving lines and boxes here and there! That is when I found a ridiculously small application called Quick Sequence Diagram Editor (QSDE).

Different than most similar tools, in QSDE instead of moving graphical elements around with your mouse you actually PROGRAM the sequence diagrams! I initially thought it was stupid and it would take much longer to write diagrams this way than just using the mouse... man I was wrong, and I am glad I kept trying to use this little gem. Now I can draw sequence diagrams real fast using QSDE.

Another decent UML tool that I used for other UML diagrams than sequence is StarUML. It is free and pretty decent for class diagrams, check it out.

This is just a simple (and lame) diagram made with StarUML

# References:

• StarUML - http://staruml.sourceforge.net/

• Quick Sequence Diagram Editor - http://sourceforge.net/projects/sdedit/

Setting up a Maven repository proxy with Nexus

2009-11-21T15:22:00.000-08:00

Background

Hey, quick hint: if you need quick get-to-the-point instructions on how to setup a free Maven repository proxy you can skip this section and scroll down to the next one. But, if you have some time to spare and are curious to know why one would need to setup his own Maven repository proxy, please bear with me a little and keep reading...

It should be a bit more than 3 years since I stopped using Ant in favor of Maven to automate builds, tests etc. After years using Ant for both personal and professional projects I got into a (then) new project where the dev team was using Maven. By that time I had already read about Maven and tried it briefly on minor prototypes just to evaluate it against Ant. In those first contacts it just seemed much more complicated than Ant, everything was considerably different etc.

Like I said, in this new project other analysts had already set it up the whole development environment, and since I had other pressing issues to focus on I did not learn much about it. The guys who set it up were amazing technical analysts, they picked excellent tools (they used AntHill Pro, among some others) and had a deep understanding of best practices for build management, test automation, quality control etc, and because of that I never had to trouble with nor had to deal with Maven or the build process myself.

Then, about another 1 and a half years ago, I began as the leading developer (and pretty much THE ONLY developer, designer, architect etc, I should add) on another project I wanted to stick with Maven. I guess I just got spoiled with things like automated dependencies download and the ability to integrate with such amazing apps like Hudson and Trac. Finally, the fact that AppFuse is heavily based on Maven also helped me made my mind.

A quick note on AppFuse: it is amazing how simple AppFuse makes it for us to kick-start our solutions rather than spend time on mundane things like defining your project hierarchy, setup Maven, Spring, Hibernate etc, but it comes with the price of being another level of abstraction – it help make things work even if you don't quite know how they work. Based on that, my advice is to use it or a similar super framework to save you time on redundant tasks but do take the time to learn how it does what it does!

So, long story short, that project I was working almost by myself grew, it got more developers and made its way to a production environment. During one of our code drops I got a message from the release manager that the build server was not able to build the application, something to do with a missing dependency. Well, yeah, if you ever worked with Maven you know one of the first things you learn is that it can download dependencies for you from the Internet. But this time it did not work. I built it locally, and it worked fine, no problem. I then removed my Maven repository, and that is when I got the exact same error we got in the build server – Maven was unable to contact one of the public Maven repositories the project depended on.

It is funny that I remember speaking to the build manager couple weeks before that incident that we should setup our project's own Maven repository to avoid something like that. But, as much as an outage of a Maven public repository is to be expected and therefore should be prevented, since I had never seen it before it got a very low priority in our To-Do list.

I found a quick fix for that by simply pointing the build server to my personal Maven repository, which had all required dependencies and did not need to download anything from the internet. So, in the end, there was no big harm other than me and the build manager having to spend about one hour to figure it out, upload files to build server and finally get the build completed.

What we can learn from this story? I don't know about you, but I learned that if we blindly depend on things beyond our control, as stable as they look like or use to be, they will fail us on unpredictable times.

It is my responsibility to shield my project from any outages on systems I do not control. The same way I always build simulators and stubs for external systems I must do something similar for external Maven repositories. And that is where a local project Maven repository fits in.

A Maven repository proxy is just a simple mechanism that acts as a mediator between your Maven-based applications and the Maven public repositories in the internet. It gives you control over your application dependencies – say, if you built your application yesterday with success, and assuming then that your Maven repository proxy is up to date, the public Maven repositories can burn and you will still be able to build your application. I hope they don't actually burn though, that was just an expression to make the point! Also, since you are not downloading hundreds of JAR files over the internet but from a local server, it may even improve your build times considerably.

So, which one to use? I tried 3 or 4 different Maven repository proxies and found Nexus to be the best one of the bunch, on my humble opinion. It could not be much simpler to install and set it up, and once you do you can forget about it, it just works!

If you are one of my 2 or 3 readers who might had the patience to get to this point with me, be happy this section is over. I will describe in the section below how easy it is to install and setup Nexus as your free-as-in-beer local Maven repository proxy.

To-the-point Nexus installation and configuration

Nexus is released both as a stand-alone server and as a standard WAR file. This section will describe how to install the WAR file release only.

To install Nexus as a WAR file just follow these instructions:

1. You need a servlet container to install Nexus WAR file. While any recent application server should work just fine I only tested it with Tomcat 6.0.20. If you do not have a servlet container I recommend you to download and install the latest stable release of Tomcat 6.x into your build server. For the purposes of this article I will assume you have Tomcat installed in $TOMCAT_ROOT (wherever it is) running on 127.0.0.1, port 8080. Make sure you replace IP address, port and paths on the next steps based on your environment.

2. Download the Nexus WAR file – usually I would recommend you to download the latest stable version but I had some trouble with Nexus 1.4.0 on Tomcat. The previous version, 1.3.6, works fine though and you can get it directly from this URL: http://nexus.sonatype.org/downloads/older/nexus-webapp-1.3.6.war

3. Rename the WAR file to nexus.war – this will make your Nexus portal URL much simpler.

4. With your Tomcat running, copy or upload the file nexus.war into “$TOMCAT_ROOT/webapps” - this should deploy Nexus in just few seconds, as shown in the image below.

If everything worked you should now be able to access Nexus web console in this URL: http://127.0.0.1:8080/nexus/

Click on Repositories link in the left menu to see which repositories are available and their URLs, as shown in the figure below. Nexus comes pre-configured with the most popular repositories including Maven Central, Codehaus and Apache. If your application requires only the most popular libraries the odds are that you may not need to change Nexus' configurations at all. If that is the case, the only thing you need to do is to edit your application's pom.xml to use Nexus repository paths (e.g. “http://127.0.0.1:8080/nexus/content/repositories/central/”). If you want to confirm exactly where each one of these URLs are pointing to or if you need to include new repositories you need to log in as an administrator.

To log in as an administrator and configure Nexus click on the link “Log In” on the top-right portal. It will ask for credentials, to witch you can use admin / admin123.

Once you log in, click on Repositories and select “Maven Central”. As an ADMIN you will see new tabs, click on “Configuration”. In this screen you will find that URL “http://127.0.0.1:8080/nexus/content/repositories/central/” is associated with a remote repository URL “http://repo1.maven.org/maven2/”, the Maven Central public repository. So, in essence, when your application tries to download a library from the Nexus URL that is not in your Nexus repository, Nexus will download it from Maven Central, but the next time it will not need to, since Nexus will keep the library in its local repository.

For my particular project I had to setup 2 or 3 additional repositories in Nexus. That was also easy, I just “cloned” one of the existing entries in the Nexus web console. I then updated the repositories section on my project's pom.xml to point to the repository paths listed in Nexus' web console. I was amazed my project got built successfully the first time, it was so simple even my humble self got it right the first time!

If you tried this but found that I missed some important step this may help - there is much more detailed information on how to install Nexus here: http://www.sonatype.com/books/nexus-book/reference/install-sect-as-a-war.html

References

• Nexus: http://nexus.sonatype.org/

• AppFuse: http://appfuse.org/

• AntHill Pro: http://www.anthillpro.com/

Oracle database quick reference

2009-11-06T11:35:00.000-08:00

Similar to the Unix/Linux quick reference post, this one lists multiple snippets about the Oracle database. Feel free to drop a comment with your own Oracle "gems".

# how to list all triggers affecting a table
select * from all_triggers where triggering_event like '%INSERT%' and table_name='PUT_TABLE_NAME_HERE';

# if you get error ORA-24170 when dropping an user see the article below
drop user myuser cascade;
ORA-24170: MYUSER.SOME_QUEUE_NAME is created by AQ, cannot be dropped directly
read: http://forums.oracle.com/forums/thread.jspa?messageID=1184946

# set password for sys/system oracle users
ALTER USER SYSTEM IDENTIFIED BY password;
ALTER USER SYS IDENTIFIED BY password;

# create a full export of an Oracle database:
exp system/password file=database.dmp full=y statistics=none

# import an Oracle database dump file:
imp USERNAME/PASSWORD@ORACLE_SID file=database.dmp full=y
imp USERNAME/PASSWORD@ORACLE_SID file=database.dmp fromuser=ORIGINAL_USER touser=NEW_USER grants=n

# run Oracle imp as a SYSDBA:
imp \'/ as sysdba\' file=database.dmp

# purge and disable Oracle 10g recyclebin tables:
-- login as sysdba
purge dba_recyclebin;
alter system set recyclebin = OFF scope=both;

# list tables with a field with a given name:
select * from all_tab_columns where column_name like '%PUT_A_COLUMN_NAME_HERE%';

# rebuild all Oracle indexes for performance tuning:
select 'ALTER INDEX ' || OWNER || '.' || OBJECT_NAME || ' REBUILD;' from dba_objects where status = 'VALID' and object_type in ('INDEX')

# improve indexes performance:
select 'analyze table ' || OWNER || '.' || object_name || ' compute statistics;' from dba_objects where object_type in ('TABLE');

# Oracle database performance optimizer - run it every time we have significant changes on a database schema
sqlplus /nolog
connect system/
begin
dbms_stats.gather_schema_stats (
ownname=>'PUT_SCHEMA_NAME_HERE',
degree=>3);
end;
/
quit;

# compile all invalid objects in Oracle:
The best way to compile all database objects that are invalid is to use a script in the $ORACLE_HOME/rdbms/admin directory named utlrp.sql. This script finds all objects in the data dictionary that are invalid and compiles them. This script is typically mentioned in patch notes but you can use it any time a schema change occurs.
I also included other alternative approaches just in case.

1. best way:
sqlplus system/password
@$ORACLE_HOME/rdbms/admin/utlrp.sql

2. by schema:
select 'ALTER ' || OBJECT_TYPE || ' ' || OWNER || '.' || OBJECT_NAME || ' COMPILE;' from dba_objects where status = 'INVALID' and object_type in ('PACKAGE','FUNCTION','PROCEDURE') and owner='PUT_SCHEMA_NAME_HERE'
union
select 'ALTER PACKAGE ' || OWNER || '.' || OBJECT_NAME || ' COMPILE BODY;' from dba_objects where status = 'INVALID' and object_type in ('PACKAGE BODY') and owner='PUT_SCHEMA_NAME_HERE';
set pages 50000
spo compile.sql
/
spo off
@compile.sql

3. using the utl_recomp APIs:
connect sys/password as sysdba;
select 'utl_recomp_begin: ' || to_char(sysdate, 'HH:MI:SS') from dual;execute utl_recomp.recomp_serial();
select 'utl_recomp_end: ' || to_char(sysdate, 'HH:MI:SS') from dual;
quit;

# how to enable table shrink in a Oracle 10g database:

alter table mytable enable row movement;
alter table mytable shrink space;

# how to shrink multiple schemas in a Oracle 10g database:

select 'ALTER TABLE ' || OWNER || '.' || OBJECT_NAME || ' ENABLE ROW MOVEMENT;' from dba_objects where object_type in ('TABLE') and owner in ('MY_SCHEMA1','MY_SCHEMA2','MY_SCHEMA3');

select 'ALTER TABLE ' || OWNER || '.' || OBJECT_NAME || ' SHRINK SPACE CASCADE;' from dba_objects where object_type in ('TABLE') and owner in ('MY_SCHEMA1','MY_SCHEMA2','MY_SCHEMA3');

# how to shrink datafiles:
select 'ALTER TABLE ''' || OWNER || '''.''' || TABLE_NAME || ''' SHRINK SPACE COMPACT' from all_tables;

SELECT VALUE FROM V$PARAMETER WHERE upper(NAME) = 'DB_BLOCK_SIZE';

SELECT 'ALTER DATABASE DATAFILE ''' || FILE_NAME || ''' RESIZE ' || CEIL( (NVL(HWM,1)*&&BLKSIZE)/1024/1024 ) || 'M;' SHRINK_DATAFILES FROM DBA_DATA_FILES DBADF,
(SELECT FILE_ID, MAX(BLOCK_ID+BLOCKS-1) HWM FROM DBA_EXTENTS GROUP BY FILE_ID ) DBAFS
WHERE DBADF.FILE_ID = DBAFS.FILE_ID(+) AND CEIL(BLOCKS*&&BLKSIZE/1024/1024)- CEIL((NVL(HWM,1)* &&BLKSIZE)/1024/1024 ) > 0 ;

# how to find which objects have extents at the end of a datafile.
Relocating these objects makes shrinking of relevant datafile possible.

select *
from (
select owner, segment_name,
segment_type, block_id
from dba_extents
where file_id =
( select file_id
from dba_data_files
where file_name = &FILE )
order by block_id desc
)
where rownum <= 5;

UNIX / LINUX quick reference

2009-11-04T11:26:00.000-08:00

I have compiled the Unix/Linux code snippets below as a quick reference covering the "how to" on various topics. Please feel free to leave a comment with your own "Unix command line gems".

How to:

# list top memory processes on a Linux machine:
ps aux | awk '{print $4"\t"$11}' | sort | uniq -c | awk '{print $2" "$1" "$3}' | sort -nr

# show how many CPUs on a Linux machine:
cat /proc/cpuinfo

# create a symbolic link:
ln -s originalPath newLinkPath
e.g. ln -s /usr/local/apache/logs /tmp/apachelogs

# list largest files in Linux (good for file system cleanup):
find / -type f -size +20000k -exec ls -lh {} \; 2> /dev/null | awk '{ print $NF ": " $5 }' | sort -nk 2,2

# list largest directories in Linux:
du -h / | grep ^[0-9.]*G | sort -g
du -h / | grep ^[0-9.]*M | sort -g | more
find / -type d -size +1G

# split big text files into smaller and more manageable files
split --lines=100000 --suffix-length=3 application.log application.log.a

# redirect Linux output:
ls 1> out.log 2> err.log

# do global string replaces in the VI editor:
:%s/old_string/new_string/g

# set executable tag in shell script files recursivelly:
find . -name "*.sh" -exec chmod +x+x+x {} \;

# delete bak and log files recursivelly:
find . -name "*.bak" -exec rm {} \;
find . -name "*.log" -exec rm {} \;

# recursively determine how much space a directory is taking:
df -h : shows how much space we have on each partition
du -sh * : show how much space the current directory is taking
du -h | grep -v '/' | awk '{print $1}' : does the same without iterating thru the subdirs

# find files that contain a given text string
grep -lir "/data/platform" *

# find files that contain a given text string recursively
find . * -exec grep -li "/data" {} \;
find . * -exec grep -li "/data" {} \; > files.txt

# poll the number of files in a folder every 3 seconds
ksh
while true
do
ls *.retry.xml | wc -l
sleep 3
done

# archive a folder and sub-directories in tar/gz
tar -zcvf archive.tgz directory
e.g. tar -zcvf apacheLogs.tgz /usr/local/apache/logs

# extract a tar/gz archive
tar -xvf archive.tgz directory
e.g. tar -xvf apacheLogs.tgz .

These are just some of the ones I use the most. You can also find many more at the Unix Toolbox here.

Automated handling of header notices in source files

2009-11-03T07:22:00.000-08:00

I recently implemented a web-based application that grew from a proof-of-concept into a real solution for a telecom operator, and it has also been offered to other potential clients. And, since it uses only open technologies such as JSF, Spring and Hibernate, it is pretty easy for other developers to reuse it for demos and prototypes without training on proprietary technologies. It is so that other teams in my company have been asking me to share the source code so they too can use it on their own projects.

While I've worked with few developers in the past who don't quite like the idea of sharing "their" code with other teams, I'm quite proud of this app and have no problems sharing it internally if it could help my company. However, to protect my company’s IP I thought I should at least include copyright messages on every single source code file before sharing it. I know I should have included it from the beginning but when we are working around the clock and we do not have to share the code these kinds of things are easily overlooked. So then I had a project with hundreds of classes and other artifacts without any copyright message. That should be a simple and common task to solve, right? Well, I thought as much... but when looking for a good way to handle it I found that my development tools do not have any functionality I could use to easily handle copyright messages on existing projects.

There are many possible ways to automate this task, from SED-based Unix shell scripts to regular expressions in search/replace functions of more advanced text editors, but I wanted to find something easier to use and more well-integrated to my development environment.

After digging for something in those lines I found a good free solution in an Eclipse plugin called Eclipse Releng Tools plugin, or simply Releng. It can be used to automate the task of including a header notice stating the code’s ownership and/or its license. This article will guide you step by step on how to install, setup and use this Eclipse plugin.

But before we start, a note of warning – YOU should make backups before following these instructions, install and run Releng. It did not cause me any troubles but I should not be held liable in any way if it causes you any problem.

Installation

If you still do not have Eclipse installed, download it here. You should select the distribution package that best suits your specific project, or just pick "Eclipse IDE for Java EE Developers" since it works best for most projects. Note that for Eclipse to run you also need a Java Development Kit – and if you do not have it and do not know how to install it you are not the target audience for this article and should probably not follow its instructions.

Once the download is finished extract the ZIP archive and start Eclipse. It will ask for a workspace folder. You can enter the path where your existing project is, or enter something else if you wish to start a new project or check out from a source code repository system. Click on the Workbench icon to close the welcome screen.

Eclipse's welcome screen

Since the Eclipse Releng Tools plugin is not installed by default in Eclipse we must install it now. In Eclipse’s top menu select “Help” / “Install New Software”.
In the “Work with” tab select “--All Available Sites--”. This will list all the available Eclipse plugins. In the filter bar enter “Releng” to filter the results and show only the plugin we need to install. Check the “Eclipse Releng Tools” plugin and complete the installation process.

Eclipse's available plugins window

After the plugin is installed Eclipse will ask you to restart it. Do that so that it is activated when Eclipse comes back up.

Open your project and note that when you right-click the project root you see the new menu item Fix Copyrights. Do not click on it now; this is just to show that the plugin was installed correctly.

Releng's new menu item

Setup

In Eclipse’s top menu select “Window” / ”Preferences”. In the Preferences window click on Copyright Tool. This will present you with the copyright template.

Releng's default copyright template

Replace the default message with this one below, including the empty lines (one at the top and two at the bottom):



This file contains proprietary information of MyCompany.
Copying or reproduction without prior written approval is prohibited.
Copyright (C) ${date}  MyCompany.  All rights reserved.

Set the year fields to 2009 or the current year.
Optionally you can enable or disable the copyright messages on properties and XML files. You can also instruct the plugin to replace existing copyright messages with this new template.

Usage

Now that Releng is properly setup, right-click on your project’s root node and select “Fix Copyrights”. All of your source code should now include a header message with your notice, as shown in the image below.

Project processed by Releng