Remove reviews

Passing UTF-8 trough HTTP

Attila-Mihaly Balazs — Wed, 29 May 2013 20:36:00 +0000

These days we should write every code as if it will be used by international people with a wide variety of personal information (just look at Falsehoods Programmers Believe About Names for some headscratchers). I would like to do add my small contribution to this by showing how UTF-8 encoded strings can be passed into GET/POST parameters.

For this I'll be using the following small PHP script, which can be quickly run by the command line PHP webserver added in PHP 5.4:

<?php header('Content-Type: text/html; charset=utf-8'); ?>
<code><pre>
GETs: <?php print_r($_GET); ?>
POSTs: <?php print_r($_POST); ?>
</pre></code>

We'll test this with the following Python script:

#!/usr/bin/python
# vim: set fileencoding=utf-8 :
import urllib
import urllib2

params = {'name': u'東京'}
params = { k: v.encode('utf-8') for k, v in params.iteritems() }
data = urllib.urlencode(params)

url = 'http://localhost:8000/?' + data
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)

print response.read()

This all works well and nicely, so here are some conclusions:

GET and POST variables need to be UTF-8 encoded after which they need to be urlencoded ("% encoded"). See this StackOverflow answer.
Based on the same answer: hostnames use Punycode instead (but we are not concerned with hostnames here)
You might need to add the following header for POST requests to work: "Content-Type: application/x-www-form-urlencoded; charset=UTF-8"
Failing to observe this sequence leads to an UnicodeEncodeError in urllib.urlencode

Connecting to the MtGox market data feed using Perl

Attila-Mihaly Balazs — Wed, 29 May 2013 17:52:00 +0000

For a recent project I needed some realistic market data for an electronic exchange. Seeing how MtGox provides free and open access to theirs (thank you!) I chose them. However none of the examples floating around the internet seemed to work, so I whipped one up using Net::Async::WebSocket::Client. Enjoy:

use IO::Async::Loop;
use Net::Async::WebSocket::Client;

my $client = Net::Async::WebSocket::Client->new(
        on_frame => sub {
                my ( $self, $frame ) = @_;
                print "\n", $frame, "\n";
        },
);

my $loop = IO::Async::Loop->new;
$loop->add( $client );

$client->connect(
        host => 'websocket.mtgox.com',
        service => 80,
        url => "ws://websocket.mtgox.com:80/mtgox",
        on_connected => sub {},
        on_connect_error => sub { die "Cannot connect - $_[-1]" },
        on_resolve_error => sub { die "Cannot resolve - $_[-1]" },
);

$loop->loop_forever;

(it is basically the sample program for the module, with the MtGox market data URL hardcoded).

Converting datetime to UTC in python

Attila-Mihaly Balazs — Thu, 07 Feb 2013 19:06:00 +0000

So you need to convert a python datetime object which has a timezone set ("aware" in the Python nomenclature) to an UTC one with no timezone set ("naive"), for example because NDB on GAE can't store anything else. The solution will look something like this:

date = date.astimezone(tz.tzutc()).replace(tzinfo=None)

For searcheability: the exception thrown by NDB if you fail to do this is "NotImplementedError: DatetimeProperty updated_at can only support UTC. Please derive a new Property to support alternative timezones."

Recovering your RoTLD password for domains registered trough Gandi.NET

Attila-Mihaly Balazs — Tue, 05 Feb 2013 16:08:00 +0000

If you need to "recover" your RoTLD password when the .RO domain is registered trough Gandi.NET (I say "recover" because you didn't set it in the first place :-)) - do this:

In the Gandi interface go to Account Management -> Update account information and set "Anti-spam system" to No
Go to the RoTLD password recovery page and reset your password. Now the password recovery email should have arrived to your inbox.
(Optional) go back to the Gandi interface and re-enable the anti-spam system

I needed to do this to enable CloudFlare on IT-Events.RO because RoTLD allows the changing of nameservers only trough their website.

Free software for Windows

Attila-Mihaly Balazs — Fri, 04 Jan 2013 01:36:00 +0000

Inspired by this post I decided to list the software I use/recommend under Windows.

Free/Libre Open Source Software:

LibreOffice (for of OpenOffice) - a very capable office solution. Most people don't need anything more than this. If you are installing it for a non-technical user, make sure to set the default file formats to the Microsoft ones (.doc, .xls, ...)
Far Manager - a very cool two-panel file manager, for those of us who like text-based things (don't let the looks fool you - it is very modern and very capable). You could also take a look at NDN or DNOSP, but FAR is my personal favorite.
VLC - THE video player. It can play 99.999% of the media out there and it won't pollute your system with all kinds of DLLs.
ffdshow-tryouts - DirectShow / VFW codecs for all the formats VLC can play (in fact they are both based on the ffmpeg project). Use this to play back videos in programs which are DirectShow based (like WMP or Winamp).
7-zip - for all your zipping and unzipping needs. It supports a lot of other formats too (mainly for extracting) so no need to install the shareware version of WinRar
Firefox - 'nuff said. There are also a lot of plugins which one might find useful like Firebug, NoScript, RequestPolicy, etc. As a download manager I would recommend DownThemAll, but I found that with recent increases in Internet access speeds I don't need a download manager.
Notepad2 for your small (text) editing needs. There is also Notepad++ and notepad2-mod.
PDFForge to create PDFs from any program which can print (it acts as a virtual printer which outputs its results to a PDF file). Sidenote: LibreOffice can natively save to PDF, no need for this if you're using it only for that.
Pidgin - you might know it as GAIM, a multi protocol instant messenger. And it is very multi protocol. The only downside is that some advanced protocol features are not always functional (*cough* file-transfer, *cough*), but I find that I rarely use those anyways. If you are installing it for someone else however, make sure to ask them what features they consider essential (like custom background/emoticons) and act accordingly.
WinSCP to copy files trough SSH/SCP, and the FileZilla client to do the same over FTP.
Various specialized programs: GIMP for photo-editing, Inkscape for vector-based graphics, VirtualDub and Avidemux for (linear) video editing, Wireshark for network analysis, Audacity for audio editing, PuTTY as an SSH client, VirtualBox for running virtual machines and Eclipse for development (it can do more than Java!). There is also the IntelliJ Community Edition for developement which is Open Source Software, but be aware of the limitations.
XAMPP for quickly setting up a LAMP environment under Windows

Free (as in beer) - these may try to trick you into installing toolbars / changing your homepage / your search engine so watch out:

TeamViewer - a nice remote control solution (also cross platform - although it doesn't run perfectly on other OSs). I especially like the fact that it can run as a service and that it takes care of the NAT traversal problem.
CDBurnerXP - for burning optical media. Nothing special, but it works.
IrfanView - a very capable image viewer / converter. Don't forget to install the plugins to take full advantage of its features! It is also so lightweight that won't believe how quickly the installation finishes. Watch out though, it tries to install sponsored programs :-(
FreeCommander - a two panel graphic file manager. Recommended if you're a Total/Windows Commander fan rather than a Norton Commander one :-)
The MS Visual Studio Express series - a good way to get your feet wet with MS specific development (also good for university projects), but be aware that you'll quickly hit a wall with it on professional projects.
uTorrent for my downloading needs.
Daemon Tools Free for my ISO (CD/DVD/BR) mounting needs. Attention during install! It will try to "upgrade" you several times during install and also try to install additional software / change your home page / search provider if you just click trough next. Don't let it, form a technical point of view it is a great product. There is also the unsupported Virtual CD product from Microsoft and Windows can mount ISOs natively starting from Windows 7 I think.
The FoxIt PDF Reader - a lightweight PDF reader, although Adobe Reader X caught up nicely I feel (and they also auto-update to eliminate security vulnerabilities), so you could give it a second go.
BB FlashBack Express for screen recording. Little annoying to install (you need to give them an email account, they try to upgrade you to the paid version and you need to "register" afterwards), but after installing it is all good and I found it to be a very capable product (even at the free version level).
Chrome and Opera as alternative browsers (and no, Chrome is not Open Source, Chromium is).
Paint.NET for advanced but "not photoshop level" image editing.
foobar2000 or Winamp (with the classic skin :-)) for music. foobar is very lightweight and quick, but it might lack some features. Winamp is very complete, but tries to make all kinds of changes to your system. You also probably don't need the Winamp Agent to run in the background all the time :-)
Dropbox for file synchronization / small-scale file serving. Alternatively there is SkyDrive, but it isn't very Linux friendly.
Windows Live Writer - the best blog publishing software I could find. Unfortunately Microsoft ruined it with the Office 2007 look and now seems to want to abandon it completely
Skype for video-call / teleconference, although lately I've been dropping it in favor of Google Hangouts

Update: this software looks very promising: Ninite. It purports to auto-install and auto-update a lot of common Windows software. Will take it for a spin the next time I install Windows.

What every programmer should know about X

Attila-Mihaly Balazs — Mon, 31 Dec 2012 11:24:00 +0000

Piggybacking on some memes floating around on the internet I would like to publish my list of "what every programmer should know".

A couple of introductory words: in my opinion the two most important things to learn for new programmers are terminology - to know what things / ideas / algorithms / concepts are called so that they can search for them on the internet and discuss their ideas) and humility (if something doesn't exists or doesn't work the way we expected, the first thing we should ask ourselves is: "what am I missing?" instead of proclaiming the predecessors to be idiots). Moving along to the list:

What every programmer should know about X - the Breaking Eggs And Making Omelettes blog contains a very eclectic list of articles. The only thing I would add is the PDF link to "What every programmer should know about memory" rather than the LWN version. Also worth checking out the discussion in the comments.
Latency numbers every programmer should know and the discussion on hackernews (this is not the original source of the data, but a particularly good visualization of it nevertheless - see also my extended version which allows you to transform the numbers into whichever time-unit - s, ms, us - you need).
From the "Falsehoods programmers believe about X" department comes:
- Falsehoods programmers believe about build systems (additionally to the comments on the post, also see the discussion on Reddit and a follow-up blogpost)
- Falsehoods Programmers Believe About Names
- Falsehoods programmers believe about time and More falsehoods programmers believe about time
A video giving an overview of the different Google systems (search, map-reduce, GFS, etc). Nothing revolutionary, but a good first-time overview:

Happy holiday reading/watching to all!

Ensuring the order of execution for tasks

Attila-Mihaly Balazs — Fri, 21 Dec 2012 21:13:00 +0000

This post was originally published as part of the Java Advent series. If you like it, please spread the word by sharing, tweeting, FB, G+ and so on! Want to write for the Java Advent blog? We are looking for contributors to fill all 24 slot and would love to have your contribution! Contact Attila Balazs to contribute!

Sometimes it is necessary to impose certain order on the tasks in a threadpool. Issue 206 of the JavaSpecialists newsletter presents one such case: we have multiple connections from which we read using NIO. We need to ensure that events from a given connection are executed in-order but events between different connections can be freely mixed.

I would like to present a similar but slightly different situation: we have N clients. We would like to execute events from a given client in the order they were submitted, but events from different clients can be mixed freely. Also, from time to time, there are "rollup" tasks which involve more than one client. Such tasks should block the tasks for all involved clients (but not more!). Let's see a diagram of the situation:

As you can see tasks from client A and client B are happily processed in parallel until a "rollup" task comes along. At that point no more tasks of type A or B can be processed but an unrelated task C can be executed (provided that there are enough threads). The skeleton of such an executor is available in my repository. The centerpiece is the following interface:

public interface OrderedTask extends Runnable {
    boolean isCompatible(OrderedTask that);
}

Using this interface the threadpool decides if two tasks may be run in parallel or not (A and B can be run in parallel if A.isCompatible(B) && B.isComaptible(A)). These methods should be implemented in a fast, non locking and time-invariant manner.

The algorithm behind this threadpool is as follows:

If the task to be added doesn't conflict with any existing tasks, add it to the thread with the fewest elements.
If it conflicts with elements from exactly one thread, schedule it to be executed on that thread (and implicitly after the conflicting elements which ensures that the order of submission is maintained)
If it conflicts with multiple threads, add tasks (shown with red below) on all but the first one of them on which a task on the first thread will wait, after which it will execute the original task.

More information about the implementation:

The code is only a proof-of-concept, some more would would be needed to make it production quality (it needs code for exception handling in tasks, proper shutdown, etc)
For maximum performance it uses lock-free* structures where available: each worker thread has an associated ConcurrentLinkedQueue. To achieve the sleep-until-work-is-available semantics, an additional Semaphore is used**
To be able to compare a new OrderedTask with currently executing ones, a copy of their reference is kept. This list of copies is updated whenever new elements are enqueued (this is has the potential of memory leaks and if tasks are infrequent enough alternatives - like an additional timer for weak references - should be investigated)
Compared to the solution in the JavaSpecialists newsletter, this is more similar to a fixed thread pool executor, while the solution from the newsletter is similar to a cached thread pool executor.
This implementation is ideal if (a) the tasks are (mostly) short and (mostly) uniform and (b) there are few (one or two) threads submitting new tasks, since multiple submissions are mutually exclusive (but submission and execution isn't)
If immediately after a "rollup" is submitted (and before it can be executed) tasks of the same kind are submitted, they will unnecessarily be forced on one thread. We could add code rearrange tasks after the rollup task finished if this becomes an issue.

Have fun with the source code! (maybe some day I'll find the time to remove all the rough edges).

* somewhat of a misnomer, since there are still locks, only at a lower - CPU not OS - level, but this is the accepted terminology

** - benchmarking indicated this to be the most performant solution. This was inspired from the implementation of the ThreadPoolExecutor.

Meta: this post is part of the Java Advent Calendar and is licensed under the Creative Commons 3.0 Attribution license. If you like it, please spread the word by sharing, tweeting, FB, G+ and so on! Want to write for the blog? We are looking for contributors to fill all 24 slot and would love to have your contribution! Contact Attila Balazs to contribute!

Java Runtime options

Attila-Mihaly Balazs — Fri, 21 Dec 2012 21:06:00 +0000

The Java runtime is a complex beast - and it has to be since it runs officially on seven platforms and unofficially on many more. Give this, it is normal that there are many knobs and dials to control how things function. The more well known ones are:

-Xmx for the maximum heap size
-client and -server for selecting the default set of parameters from classes of defaults
-XX:MaxPermGen for controlling the permanent generation size

Other than these, it is (very) rarely the case that you need to change the defaults. However, thanks to Java being open source you can see the list of options, their default values and a short explanation directly from the source code. Currently there are almost 800 options in there!

An other way to see the options (but one which doesn't display the explanations unfortunately) is the following command:

java -XX:+UnlockDiagnosticVMOptions -XX:+UnlockDiagnosticVMOptions -XX:+PrintFlagsFinal -version

These options are well worth studying. Not for tweaking them (since there is a wealth of testing behind the defaults the extent of which would be very hard to replicate), but rather to understand the different functionalities offered by the JVM (for example why you might not see stacktraces in exceptions).

Changes to String.substring in Java 7

Attila-Mihaly Balazs — Wed, 12 Dec 2012 10:37:00 +0000

It is common knowledge that Java optimizes the substring operation for the case where you generate a lot of substrings of the same source string. It does this by using the (value, offset, count) way of storing the information. See an example below:

In the above diagram you see the strings "Hello" and "World!" derived from "Hello World!" and the way they are represented in the heap: there is one character array containing "Hello World!" and two references to it. This method of storage is advantageous in some cases, for example for a compiler which tokenizes source files. In other instances it may lead you to an OutOfMemorError (if you are routinely reading long strings and only keeping a small part of it - but the above mechanism prevents the GC from collecting the original String buffer). Some even call it a bug. I wouldn't go so far, but it's certainly a leaky abstraction because you were forced to do the following to ensure that a copy was made: new String(str.substring(5, 6)).

This all changed in May of 2012 or Java 7u6. The pendulum is swung back and now full copies are made by default. What does this mean for you?

For most probably it is just a nice piece of Java trivia
If you are writing parsers and such, you can not rely any more on the implicit caching provided by String. You will need to implement a similar mechanism based on buffering and a custom implementation of CharSequence
If you were doing new String(str.substring) to force a copy of the character buffer, you can stop as soon as you update to the latest Java 7 (and you need to do that quite soon since Java 6 is being EOLd as we speak).

Thankfully the development of Java is an open process and such information is at the fingertips of everyone!

A couple of more references (since we don't say pointers in Java :-)) related to Strings:

If you are storing the same string over and over again (maybe you're parsing messages from a socket for example), you should read up on alternatives to String.intern() (and also consider reading chapter 50 from the second edition of Effective Java: Avoid strings where other types are more appropriate)
Look into (and do benchmarks before using them!) options like UseCompressedStrings (which seems to have been removed), UseStringCache and StringCache

Hope I didn't strung you along too much and you found this useful! Until next time
- Attila Balazs

(Re)Start me up!

Attila-Mihaly Balazs — Sat, 01 Dec 2012 18:24:00 +0000

This post was originally published as part of the Java Advent series.

There are cases where you would like to start a Java process identical to the current one (or at least using the the same JVM with tweaked parameters). Some concrete cases where this would be useful:

Auto-tuning the maximum memory parameters (ie. you have an algorithm to determine the optimal value - for example: 80% of the system memory - and your JVM wasn't started with that particular value)
Creating a cluster of processes for high(er)-availability (true HA implies multiple physical nodes) or because processes have different roles (like the components in MongoDB).
Daemonizing the current process (that is, the background process should run even after the launching process has terminated) - this is a very frequent modus-operandi for programs on *nix systems where you have the foreground "control" process and the background "daemon" process (not to be confused with the "daemon" threads).

Doing this is relatively simple - and can be done in pure Java - after you find the correct API calls:

List arguments = new ArrayList<>();
// the java executable
arguments
  .add(String.format("%s%sbin%sjava",
    System.getProperty("java.home"), File.separator,
    File.separator));
// pre-execuable arguments (like -D, -agent, etc)
arguments.addAll(ManagementFactory.getRuntimeMXBean()
  .getInputArguments());

String classPath = System.getProperty("java.class.path"), javaExecutable = System
  .getProperty("sun.java.command");
if (classPath.equals(javaExecutable)) {
 // was started with -jar
 arguments.add("-jar");
 arguments.add(javaExecutable);
} else {
 arguments.add("-classpath");
 arguments.add(classPath);
 arguments.add(javaExecutable);
}

// we might add additional arguments here which will be received by the
// launched program
// in its args[] paramater
arguments.add("runme");

// launch it!
new ProcessBuilder().command(arguments).start();

Some explanations about to the code:

It is largely inspired from this project
We suppose that the java executable is named java and is located in bin/java relative to java.home. We use File.separator for the code to be portable.
getInputArguments is used to get specific arguments passed to the JVM (like -Xmx). It does not include the classpath.
Which is taken from java.class.path
Finally, there is one heuristic step: we try to detect if we were launched using the -jar myjar.jar syntax or the MyMainClass syntax and replicate it.

This is it! After that we use ProcessBuilder (which we should always favour over Runtime.exec because it auto-escapes the parts of the command line for us).

A final thought: if you intend to use this method to "daemonize" a process (that is: to ensure that it stays running after its parent process has terminated) you should do two things:

Redirect the standard input and output. By default they are redirected into temporary buffers and the JVM will seemingly randomly terminate when those buffers (pipes) fill up.
Under Windows use javaw instead of java. This ensures that the process won't be tied to the console it was started from (however it will still be tied to the user login session and will terminate when the user logs out - for a more heavy-duty solution look into the Java Service Wrapper).

This is it for today, hope you enjoyed it, fond it useful. If you run the code and it doesn't work as advertised, let me know so that I can update it (I'm especially interested if it works with non Sun/Oracle JVMs). Come back tomorrow for an other article!

Meta: this post is part of the Java Advent Calendar and is licensed under the Creative Commons 3.0 Attribution license.

Upgrading from MySQL to MariaDB on Ubuntu

Attila-Mihaly Balazs — Sun, 25 Nov 2012 11:14:00 +0000

So you decided that Oracle doesn't know its left foot from the back of his neck when it comes to open source (how's that for a mixed metaphor), but you are not ready just yet to migrate over to PostgreSQL? Consider MariaDB. Coming from Monty Widenius, the original author of MySQL, it aims to be 100% MySQL compatible while also being truly open-source.

Give that it's 100% MySQL compatible, you can update in-place (nevertheless it is recommended that do a backup of your data first). The steps are roughly adapted from here.

Go to the MariaDB repository configuration tool and generate your .list file (wondering what's up with the 5.5 vs 10.0 version? See this short explanation). You don't know the exact Ubuntu version you're running? Just use lsb_release -a.
Save the generated file under /etc/apt/sources.list.d/MariaDB.list as recommended and do an sudo aptitude update. You should see an output complaining about some public keys.
Do sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 0xCBCB082A1BB943DB to add those keys (replace the last number with the one you saw in the previous output).
Issue sudo apt-cache policy mysql-common and you should see mariadb as an upgrade option.
Finally do sudo aptitude upgrade mysql-common libmysqlclient18 and watch your MySQL database being transformed into a MariaDB one and all keeping chugging along just as usual!

Cluj-Napoca (Romania) wins the title of European Youth Capital 2015

Attila-Mihaly Balazs — Sun, 25 Nov 2012 10:58:00 +0000

We interrupt our regular (lack of) posting to bring you this news:

Cluj-Napoca is European Youth Capital 2015. Congratulation to everyone involved!

Writing beautiful code - not just for the aesthetic value

Attila-Mihaly Balazs — Tue, 06 Nov 2012 14:33:00 +0000

This article was originally published in the 6th edition of TodaySoftMag in Romanian and on the Transylvania JUG blog in English. Reprinted here with the permission of the author / magazine.

Most mainstream programming languages contain a large set of features and diverse standard libraries. Because of this it becomes important to know not only “how” you can achieve something (to which there are usually several answers) but also “what is the recommended way”.

In this article I will argue that knowing and following the recommended ways of coding doesn’t only yield shorter (easier to write), easier to read, understand and maintain code but also prevents programmers from introducing a lot of bugs.

This particular article needs a drop of Java language knowledge to savour, but the fundamental idea can be generalized to any programming language: there is more to using a language efficiently than just knowing the syntax.

Example 1: Double Trouble

Lets start with a snippet of code: what does it print out?

Double d1 = (5.0d - 5.0d) *  1.0d;
Double d2 = (5.0d - 5.0d) * -1.0d;
System.out.println(d1.equals(d2));

What about the following one?

double d1 = (5.0d - 5.0d) *  1.0d;
double d2 = (5.0d - 5.0d) * -1.0d;
System.out.println(d1 == d2);

The answer seems to be clear: in both cases we multiply zero with different values (plus and minus one respectively), thus the result should be zero which should compare as equal regardless of the comparison method used (calling the equals method on objects or using the equality operator on the primitive values).

If we run the code, the result might surprise us: the first one displays false while the second one displays true. What’s going on? On one level we can talk the technical reasons behind this result: floating point values are represented in Java (and many other programming languages) using the sign-and-magnitude notation defined in the IEEE Standard 754. Because of this technical detail both “plus zero” and “minus zero” can be represented by variables of this type. And the “equals” method on Double (and Float) objects in Java considers these values to be distinct.

On another level however we could have avoided this problem entirely by using the primitive values as shown in the second code snippet and as suggested by Item 49 in the Effective Java book^[1]: Prefer primitive types to boxed primitives. Using primitive types is also more memory efficient and saves us from having to create special cases for the null value.

Sidenote: we have a similar situation with the BigDecimal class^[2] where values scaled differently don’t compare as equal. For example the following snippet also prints false:

BigDecimal d1 = new BigDecimal("1.2");
BigDecimal d2 = new BigDecimal("1.20");
System.out.println(d1.equals(d2));

The answer in this case (given that there is no primitive equivalent for this class) would be to use the compareTo method and assert that it returns zero instead of using the equals method (a method which can also be used to solve the conundrum in the Double/Float case if we are not worried about nulls).

Example 2: Where is my null at?

What does the following snippet of code print out?

Double v = null;
Double d = true ? v : 0.0d;
System.out.println(d);

At first glance we would say: null, since the condition is true and v is null (and null can be assigned to a reference of any type, so we are allowed to use it). The actual result is however a NullPointerException at the second line. This is because the right-hand type of the assignment is actually double (the primitive type) not Double (as we would expect) which is silently converted into Double (the boxed type). The generated code looks like this:

Double d = Double.valueOf(true ? v.doubleValue() : 0.0d);

This behavior is described in the Java Language Specification^[3]:

“If one of the second and third operands is of primitive type T, and the type of the other is the result of applying boxing conversion (§5.1.7) to T, then the type of the conditional expression is T.”

I would venture to guess that not many of us have read the JLS in its entirety and even if we would have read it, we might not have realized the implications of each phrase. The recommendation from EJ2nd mentioned at the previous example saves us again: we should use primitive types. We can also draw a parallel with Item 43: Return empty arrays or collections, not nulls. Would we have used a “neutral element”, which is analogous to using empty arrays/collections, the problem would not have appeared. (The neutral element would be 0.0d if we use the value later in summation or 1.0d if we use it in multiplication.)

Example 3: We come up empty

What is the difference between the following two conditions?

Collection<V> items;
if (items.size() == 0) { ... }
if (items.isEmpty()) { ... }

One could argue that they do exactly the same thing as being empty is equivalent to having zero items. Still, the second condition is easier to understand (we can almost read it out loud: “if items is empty then …”). But there is more: in some cases it can be much, much faster. Two examples from the Java standard libraries where the time needed to execute “size” grows linearly with the number of elements in the collection while “isEmpty” returns in constant time: ConcurrentLinkedQueue^[4] and the view sets returned by TreeSet’s^[5] headSet/tailSet methods. And while the documentation for the first mentions this fact, it doesn’t for the second.

This is yet another example how nicer code is also faster.

Example 4: Careful with that static, Eugene!

What will the following snippet of code print out?

public final class Test {
        private static final class Foo {
                static final Foo INSTANCE = new Foo(); // 2
                static final String NAME = Foo.class.getName(); // 3
                Foo() {
                        System.err.println("Hello, my name is " + NAME);
                }
        }
        public static void main(String[] args) {
                System.err.println("Your name is what?\nYour name is who?\n");
                new Foo(); // 1
        }
}

It will be

Your name is what?
Your name is who?

Hello, my name is null
Hello, my name is Test$Foo

The (probably) unexpected null value happens because we obtain a reference to a partially constructed object:

We start to create an instance of Foo at point 1
This being the first reference to Foo, the JVM loads it and starts to initialize it
Initializing Foo involves initializing all its static fields
The initialization of the first static field contains a call to the constructor at point 2 which is dutifully executed
At this point the NAME static field is not yet initialized, so the constructor will print out null

This code demonstrates that static fields can be confusing and we shouldn’t use them for things other than constants (but even then we should evaluate if the constant is not better declared as an Enum). By the same token we should also avoid singletons which make our code harder to test (thus avoiding them will make the code easier to test).

We should however favor static member classes over non-static ones (Item 22 in EJ2nd). Static classes in Java are entirely distinct conceptually from static fields and it is unfortunate that the same word was used to describe them both.

We should also run static analysis tools on our code and verify their output frequently (ideally at every commit). For example the bug presented is caught by Findbugs^[6] and tools incorporating Findbugs.

Example 5: Remove old cruft

Name four things wrong with the following snippet:

// WRONG! DON’T DO THIS!
Vector v1;
...
if (!v1.contains(s)) { v1.add(s); }

They would be:

The wrong container type is used. We clearly want to have each string present at most once which suggests using a Set<> which has the benefits of shorter and faster code (the above method gets linearly slower with the number of elements)
Doesn’t use generics
It unnecessarily synchronizes access to the structure if it is only used from a single thread
If the structure is actually used from multiple threads, the code is not thread safe, only “exception safe” (as in: no exceptions will be raised, but the data structure can be silently corrupted possibly creating a lot of headache downstream)

All of these can be avoided by dropping Vector and its siblings (Hashtable, StringBuffer) and using the Java Collection Framework (available for 14 years^[7]) with generics (available for 8 years^[8]).

Conclusion

There are many more examples one could give, but I think the point is well made that knowing a programming language means more than just knowing the syntax at a basic level. I’m urging you if you are using Java: get yourself a copy “Effective Java, 2nd edition” and “Java™ Puzzlers: Traps, Pitfalls, and Corner Cases” each and read through them if you haven’t done so already. Also, use static analysis on your code (Sonar^[9] is a good choice in this domain) and consider fixing the issues signaled by it, or at least read up on them.

Again, the conclusions is similar for other languages:

Try reading up on best practices/idiomatic ways to write code in the given language. For example for Perl the best book currently is “Modern Perl^[10]” by chromatic
Look to see if there is a good quality static analysis / lint program for your language. For Perl there is Perl::Critic^[11], for Python there is pep8^[12] and pylint^[13], all of which are free and open source

Being good a programmer (or an architect, or a business analyst, etc) is process of lifelong learning and these are the tools which can help us truly learn a programming language.

[1]Joshua Bloch: Effective Java, Second Edition. ISBN: 0321356683

[2]http://docs.oracle.com/javase/7/docs/api/java/math/BigDecimal.html

[3]http://docs.oracle.com/javase/specs/jls/se7/html/jls-15.html#jls-15.25

[4]http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ConcurrentLinkedQueue.html#size()

[5]http://docs.oracle.com/javase/7/docs/api/java/util/TreeSet.html

[6]http://findbugs.sourceforge.net/bugDescriptions.html#SI_INSTANCE_BEFORE_FINALS_ASSIGNED

[7]http://en.wikipedia.org/wiki/Java_version_history#J2SE_1.2_.28December_8.2C_1998.29

[8]http://en.wikipedia.org/wiki/Java_version_history#J2SE_5.0_.28September_30.2C_2004.29

[9]http://www.sonarsource.org/

[10]http://www.onyxneon.com/books/modern_perl/index.html

[11]http://search.cpan.org/~thaljef/Perl-Critic-1.118/lib/Perl/Critic.pm

[12]http://pypi.python.org/pypi/pep8

[13]http://pypi.python.org/pypi/pylint

A (mostly) cross-platform clickable map of Romanian counties

Attila-Mihaly Balazs — Thu, 01 Nov 2012 10:06:00 +0000

TL;DR - I've created a map of Romania's counties (judete) using Javascript which can be used as an alternative for drop-down boxes during form input. It works great with FF and Chrome :-). See it in action and browse the source code.

The map is based on the SVG drawings from Wikimedia Commons postprocessed in Inkscape to simplify the paths (and to remove the Black Sea from it). After that it was converted to Raphael.js using readysetraphael. I also use ScaleRaphael to be able to render it at an arbitrary size. According to BrowserStack (a great service well worth its money BTW if you do web development!) it works with Firefox (starting with version 3.6), Chrome and Safari. IE has a hard time with it, but hopefully as the underlying library improves, it will start to work :-).

Every end is a new beginning

Attila-Mihaly Balazs — Sat, 27 Oct 2012 23:03:00 +0000

TL;DR: I'm shutting down the twitfeeder project (it was on lifesupport for a long time) so I mirrored a technical article from the blog in the hope that it might be useful for somebody someday.

Proxying URL fetch requests from the Google App Engine

Hosting on the Google App Engine means giving up a lot controls: your application will run on some machines in one of Google's datacenters, use some other machines to store data and use yet an other set of machines to talk to the outside world (send/receive emails, fetch URLs, etc). You don't know the exact identity of these machines (which for this article means their external IP address), and even though you can find out some of the details (for example by fetching the ipchicken page and parsing the result or by sending an e-mail and looking at the headers), there is no guarantee made that the results are repeatable (ie. if you do it again and again you will get the same result) in the short or long run. While one might not care about such details most of the time, there are some corner cases where it would be nice to have a predictable and unique source address:

You might worry that some of the exit points get blocked because other applications on the Google App Engine have an abusive behavior

You might want a single exit point so that you can "debug" your traffic at a low (network / tcpdump) level

And finally, the main reason for which the Twit Feeder uses it: some third-party services (like Twitter or Delicious) use the source IP to whitelist requests to their API

To be fair to the GAE Architects: the above considerations don't affect the main usecase and are more of a cornercase if we look at the average application running on the GAE. Also, having so few commitments (ie. they don't stipulate things like "all the URL fetches will come from the 10.0.0.1/24 netblock") means that they are free to move things around (even between datacenters) to optimize uptime and performance which in turn benefits the application owners and users.

Back to our scenario: the solution is to introduce an intermediary which has a static and well known IP and let it do all the fetching. Ideally I would have installed Squid and and be done with it, but the URL fetch service doesn't have support for proxies currently. So the solution I came up with looks like this:

Get a VPS server. I would recommend one which gives you more bandwidth rather than more CPU / memory / disk. It is also a good idea to get one in the USA to minimize latency from/to the Google datacenters. I'm currently using VPSLink and I didn't had any problems (full disclosure: that is a referral link and you should get 10% off for lifetime if you use it).

Install Apache + PHP on it

Use a simple PHP script to fetch the page encoded in a query parameter using the php_curl extension.

To spice up this blogpost ;-), here is a diagram of the process:

A couple of points:

Taking care of a VPS can be challenging, especially if you aren't a Linux user. However failing to do so can result in it being taken over by malicious individuals. Consider getting a managed VPS. Also, three quick security tips: use strong passwords. move the SSH server to a non-standard port and configure your firewall to be as restrictive as possible (deny everything, and open ports to only a limited number of IP addresses).

Yes, this introduces a single point of failure into your application, so plan accordingly (while your application is small, this shouldn't be a problem - as it grows you can get two VPS's at two different data centers for example for added reliability).

The traffic between Google and your VPS can be encrypted using TLS (HTTPS). The good news is that the URL fetcher service doesn't check the certificates, so you can use self-signed ones. The bad news is that the URL fetcher doesn't check the certificates, so a determined attacker can relatively easily man-in-the-middle you (but it protects the data from the casual sniffer).

Be aware that you need to budget for double of the amount of traffic you estimate using at the VPS (because it needs to first download it and then upload it back to Google). The URL fetcher service does know how to use gzip compression, so if you are downloading mainly text, you shouldn't have a bandwidth problem.

PHP might seem like an unusual choice of language, given how most GAE users have experience in either Python or Java, but there are a lot of tutorials out there on how to install it (and on modern Linux distribution it can be done in under 10 minutes with the help of the package manager) and it was the one I was most comfortable with as an Apache module.

Without further ado, here are the relevant sourcecode snippets (which I hereby release into the public domain):

The PHP script:

<?php
error_reporting(0);
require 'privatedata.php';

if ($AUTH != @$_SERVER['HTTP_X_SHARED_SECRET']) {
   header("HTTP/1.0 404 Not Found");
   exit;
}

$url = @$_GET['q'];
if (!isset($url)) {
   header("HTTP/1.0 404 Not Found");
   exit;
}

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_ENCODING, '');
curl_setopt($ch, CURLOPT_TIMEOUT, 30);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, false);

if (isset($_SERVER['HTTP_USER_AGENT'])) {
   curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
}
if (isset($_SERVER['PHP_AUTH_USER']) && isset($_SERVER['PHP_AUTH_PW'])) {
   curl_setopt($ch, CURLOPT_USERPWD, "{$_SERVER['PHP_AUTH_USER']}:{$_SERVER['PHP_AUTH_PW']}");
}

if (count($_POST) > 0) {
   curl_setopt($ch, CURLOPT_POST, true);
   $postfields = array();
   foreach ($_POST as $key => $val) {
       $postfields[] = urlencode($key) . '=' . urlencode($val);
   }
   curl_setopt($ch, CURLOPT_POSTFIELDS, implode('&', $postfields));
}

$headers = array();
function header_callback($ch, $header) {
   global $headers;
   // we add our own content encoding
   // also, the content length might vary because of this
   if (false === stripos($header, 'Content-Encoding: ')
       && false === stripos($header, 'Content-Length: ')
       && false === stripos($header, 'Transfer-Encoding: ')) {
       $headers[] = $header;
   }
   return strlen($header);
}

curl_setopt($ch, CURLOPT_HEADERFUNCTION, 'header_callback');

$output = curl_exec($ch);
if (FALSE === $output) {
   header("HTTP/1.0 500 Server Error");
   print curl_error($ch);
   exit;
}
curl_close($ch);

foreach ($headers as $header) {
   header($header);
}
print $output;

The "privatedata.php" file looks like this:

<?php

$AUTH = '23tqhwfgj2qbherhjerj';

Two separate files are being used to avoid submitting the password to the source repository, while still keeping all the sourcecode open.

Now with the code in place, you can test it using curl:

curl --compressed -v -H 'X-Shared-Secret: 23tqhwfgj2qbherhjerj' 'http://127.0.0.1:1483/fetch_url.php?q=http%3A//identi.ca/api/help/test.json

As you can see, a custom header is used for authentication. An other security measure is to use a non-standard port. Limiting the requests to the IPs of the Google datacenter from the firewall would be the ideal solution, but given that this was the problem we are trying to solve in the first place (the Google datacenters not having an officially announced set of IP addresses), this doesn't seem possible.

Finally, here is the Python code to use the script from withing an application hosted on the GAE:

from urllib import quote
from google.appengine.api import urlfetch

# ...

fetch_headers['X-Shared-Secret'] = '23tqhwfgj2qbherhjerj'
result = urlfetch.fetch(url='http://127.0.0.1:1483/fetch_url.php?q=%s' % quote(url), payload=data,
 method=urlfetch.POST if data else urlfetch.GET,
 headers=fetch_headers, deadline=timeout, follow_redirects=False)

A little more discussion about the code:

The method of using a custom header for authorization was chosen, since the forwarding of authentication data (ie. the "Authorization" header) was needed (specifically this is what the Twitter API uses for verifying identity)

Speaking of things the script forwards: it does forward the user agent and any POST data if present. The forwarding of other headers could be quite easily be added.

Passing variables in a GET request is also supported (they would be double-encoded, but that shouldn't be a concern in but the most extreme cases)

If we are talking about sensitive data, cURL (and the cURL extension for PHP) has the ability to fetch HTTPS content and to verify the certificates.

While this method might look cumbersome, in practice I found it to work quite well. Hopefully this information will help others facing the same problem. A final note: if you have questions about the code or about other aspects, post them in the comments. I will try to answer them as fast as possible. I'm also considering launching a "proxy service" for GAE apps to make this process much more simpler (abstracting away the setup and administration of the VPS), so if you would be interested in paying a couple of bucks for such a service, please contact me either directly or trough the comments.

Helper for testing multi-threaded programs in Java

Attila-Mihaly Balazs — Sat, 27 Oct 2012 22:48:00 +0000

This post was originally published on the Transylvania JUG blog.

Testing multi-threaded code is hard. The main problem is that you invoke your assertions either too soon (and they fail for no good reason) or too late (in which case the test runs for a long time, frustrating you). A possible solution is to declare an interface like the following:

interface ActivityWatcher {
 void before();
 void after(); 
 void await(long time, TimeUnit timeUnit) throws InterruptedException, TimeoutException;
}

It is intended to be used as follows:

“before” is called before the asynchronous task is delegated to an execution mechanism (threadpool, fork-join framework, etc) and it increments an internal counter.
“after is called after the asynchronous task has completed and it decrements the counter.
“await” waits for the counter to become zero

The net result is that when the counter is zero, all your asynchronous tasks have executed and you can run your assertions. See the example code. A couple more considerations:

There should be a single ActivityWatcher per test (injected trough constructors or a dependency injection framework)
In production code you will use a dummy/noop implementation which removes any overhead.
This only works for situations where the asynchronous are kicked of immediately. Ie. it doesn’t work for situations where we have periodically executing tasks (like every 5 seconds) and we would want to wait for the 7th tasks to be executed for example.

One thing the above code doesn’t do is collecting exceptions: if the exceptions happen on different threads than the one executing the testrunner, they will just die and the testrunner will happily report that the tests passs. You can work around this in two ways:

use the default UncaughtExceptionHandler to capture all exceptions and rethrow them in the testrunner if they arrise (not so nice because it introduces global state – you can’t have two such tests running in parallel for example)
Extend activity watcher and code calling activity watcher such that it has a “collect(Throwable)” method which gets called with the uncaught exceptions and “await” rethrows them.

Implementing this is left as an exercise to the reader :-).

GeekMeet talk about Google App Engine

Attila-Mihaly Balazs — Sat, 27 Oct 2012 22:45:00 +0000

The GAE presentation I've given for the 12th edition of Cluj Geek Meet can be found here (created using reveal.js).

You can find the source code here.

Lightning talk at Cluj.PM

Attila-Mihaly Balazs — Sun, 19 Aug 2012 15:22:00 +0000

The slides from my Cluj.PM lightning talk:

It was a stressful (but fun!) experience. Thanks to the organizers!

Running pep8 and pylint programatically

Attila-Mihaly Balazs — Sun, 19 Aug 2012 15:12:00 +0000

Having tools like pep8 and pylint are great, especially given the huge amount of dynamism involved in Python - which results in many opportunities to shooting yourself in the foot. Sometimes however you want to invoke these tools in more specialized ways, for example only on the files which changed since the last commit. Here is how you can do this from a python script and capture their output for later post-processing (maybe you want merge the output from both tools, or maybe you want to show only the lines which changed since the last commit, etc):

import pep8
try:
  sys.stdout = StringIO()
  pep8_checker = pep8.StyleGuide(config_file=config, format='pylint')
  pep8_checker.check_files(paths=[ ...path to files/dirs to check... ])
  output = sys.stdout.getvalue()
finally:
  sys.stdout = sys.__stdout__

from pylint.lint import Run
from pylint.reporters.text import ParseableTextReporter

reporter = ParseableTextReporter()
result = StringIO()
reporter.set_output(result)
Run(['--rcfile=pylint.config'] + [ ...files.., ], reporter=reporter, exit=False)
output = result.getvalue()

It is recommended that you use pylint/pep8 installed trough pip/easy_install rather than the Linux distribution repositories, since they are known to contain outdated software. You can check for this via code like the following:

if pkg_resources.get_distribution('pep8').parsed_version < parse_version('1.3.3'):
    logging.error('pep8 too old. At least version 1.3.3 is required')
    sys.exit(1)
if pkg_resources.get_distribution('pylint').parsed_version < parse_version('0.25.1'):
    logging.error('pylint too old. At least version 0.25.1 is required')
    sys.exit(1)

Finally, if you have to use an old version of pep8, the code needs to be modified to the following (however, this older version probably won't be of much use and will most likely annoy you - you should really try to use an up-to-date version - for example you could isolate this version using virtualenv):

result = []
import pep8
pep8.message = lambda msg: result.append(msg)
pep8.process_options(own_code)
for code_dir in [ ...files or dirs... ]:
    pep8.input_dir(code_dir)

Clearing your Google App Engine datastore

Attila-Mihaly Balazs — Wed, 15 Aug 2012 10:08:00 +0000

Warning! This is a method to erase the data from your Google App Engine datastore. There is no way to recover your data after you go trough with this! Only use this if you're absolutely certain!

If you have a GAE account used for experimentation, you might like to clean it up sometimes (erase the contents of the datastore and blobstore associated with the application). Doing this trough the admin interface can become very tedious, so here is an alternative method:

Start your Remote API shell

Use the following code to delete all datastore entities:

while True: keys=db.Query(keys_only=True).fetch(500); db.delete(keys); print "Deleted 500 entries, the last of which was %s" % keys[-1].to_path()

Use the following code to delete all blobstore entities:

from google.appengine.ext.blobstore import *
while True: list=BlobInfo.all().fetch(500); delete([b.key() for b in list]);  print "Deleted elements, the last of which was %s" % list[-1].filename

The above method is inspired by this stackoverflow answer, but has the advantage that it does the deletion in smaller steps, meaning that the risk of the entire transaction being aborted because of deadline exceeded or over quota errors is removed.

Final caveats:

This can be slow
This consumes your quota, so you might have to do it over several days or raise your quota
The code is written in a very non-pythonic way (multiple statements on one line) for the ease of copy-pasting

Relaxed JSON parsing

Attila-Mihaly Balazs — Tue, 27 Dec 2011 13:40:00 +0000

This blogpost was originally posted to the Transylvania JUG blog.

JSON is a good alternative when you need a lightweight format to specify structured data. But sometimes (for example when you want the user to specify JSON manually) you would like to relax the formalism required to specify "valid" JSON data. For example the following snippet is not valid as per the spec, although its intent is quite clear:

   [{ foo: 'bar' }]

To make this standard compliant we would need to write it as:


  [{ "foo": "bar" }]

We shouldn't run out and blame the standard of course since it needs to balance many contradictory requirements (ambiguity of encoded data, ease of understanding, ease of writing parsers, etc). If you decide that you want to strike the balance differently (make the definition of valid data more relaxed) you can do this easily with the Jackson parser:


  JsonParser parser = new JsonFactory()
	.createJsonParser("[{ foo: 'bar' }]")
		.enable(JsonParser.Feature.ALLOW_UNQUOTED_FIELD_NAMES)
		.enable(JsonParser.Feature.ALLOW_SINGLE_QUOTES);
JsonNode root = new ObjectMapper().readTree(parser);

assertEquals("bar", root.get(0).get("foo").asText());

If your tool of choice is gson, it is slightly more complicated but still doable. See the linked source code for a complete example.

JSON is a good tool for semi-structured data and using a relaxed parsing can make the programs you write easier to use.

Updating the root certificates for Java

Attila-Mihaly Balazs — Thu, 20 Oct 2011 15:36:00 +0000

One usually thinks of SSL in the context of HTTPS, but there are also other protocols which rely on it to provide security. See this link for a short overview of SSL – it only mentions HTTPS, but the same applies for IMAPS, FTPS, etc – SSL is independent of the wrapped protocol. You can have issues with your Java programs in where the party you are communicating with provider changes their certificate and the program rejects it as invalid. The exception is something like:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: 
    PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: 
    unable to find valid certification path to requested target

One cause of the problem can be that the server uses an SSL provider which is based on a root certificate that wasn’t included with the particular version of Java you are using (this is especially true for really old versions like Java 1.5). The issue can be solved by updating to the latest version, but it might be that this isn't an option. Fortunately I found the following article: No more ‘unable to find valid certification path to requested target’

How to use it:

Compile the program javac InstallCert.java
Run it with the target host/port. For example in our case it would be: java InstallCert imap.mailprovider.com:993 (993 is the port for IMAPS)
navigate trough the menus and select which certificate to import
now you have a file called jssecacerts. You need to copy this to $JAVA_HOME/jre/lib/security/cacerts (back up the existing file first!)
Now the root certificate is imported (you can confirm this by rerunning InstallCert)

HTH

Vagrant and VirtualBox on Windows

Attila-Mihaly Balazs — Thu, 20 Oct 2011 15:27:00 +0000

Vagrant is a collection of scripts written in Ruby to manage VirtualBox images in a shared environment (like the QA boxes inside a company): install them, update them, etc. Unfortunately installing it under Windows is not as straight forward as one would want, so here are some useful tips:

Read the Windows setup page and the Windows x64 setup page (if you are on 64 bit)

If you are on a 64 bit Windows install:

Check out this post if your JRuby is using the 32 bit JVM on a x64 Windows setup
You need to use version 4.0 of VirtualBox (rather than the latest). You can get it from here

You need to use an older version of Vagrant:

       jgem install jruby-openssl jruby-win32ole
jgem install --version '=0.7.8' vagrant

If the vagrant box download stops around 4G, check that you have a NTFS filesystem (rather than FAT) and deactivate any "network" scanning capabilities of installed security software (I had problems with NOD32 particularly)

HTH

Another friend blogging

Attila-Mihaly Balazs — Thu, 13 Oct 2011 18:20:00 +0000

Another friend ~~bit the dust~~ started blogging: Cleonte's GitHub blog - bookmarks at the moment but looking forward to more involved posts :-).

Using Jython from Maven

Attila-Mihaly Balazs — Thu, 13 Oct 2011 18:16:00 +0000

This blogpost was originally posted to the Transylvania JUG blog.

On the surface it looks simple: just add the dependency and you can run the example code.

However what the jython artifact doesn’t get you are the standard python libraries like re. This means that as soon as you try to do something like the code below, it will error out:

PythonInterpreter interp = new PythonInterpreter();
try {
  interp.exec("import re");
} 
catch (PyException ex) {
  ex.printStackTrace();
}

The solution? Use the jython-standalone artifact which includes the standard libraries. An other advantage is that it has the latest release (2.5.2) while jython lags two minor revisions behind (2.5.0) in Maven Central. A possible downside is the larger size of the jar.

<dependency>
    <groupId>org.python</groupId>
    <artifactId>jython-standalone</artifactId>
    <version>2.5.2</version>
</dependency>