Tarlog on Java

DevOps: Making Fast Deployments of Java Servers using Maven and Nexus

2013-05-21T10:26:00.000+03:00

A Warning: this post is theoretical. I have never tried something like this yet. Maybe I will try it in the future. But currently it's just a nice idea.
In addition, if you know about somebody who works in a similar way, I would really like to know. So please comment!

If you provide a SAAS service you probably have multiple Java servers running in some sort of a cluster. If your SAAS solution is complicated, and if your solution is multi-tier, you should have multiple servers types. And now comes a question: How to make quick deployments to the production?

The common solution suggests that you build a package and release it. It might be a war, or a zip, or a rpm if you are running on Linux.
Once released, you upload the package to the server, unzip/copy it to the relevant folder and restart the server.

The problem with this solution might be if your packages are large. (And if you are using OSGi, your packages are usually very large!) So the upload itself takes time. It also uses traffic which might become expensive if you perform a lot of deployments. And the really funny thing is that most of the upload is redundant: most of the jars in your package are third parties that do not change between the deployments at all!

The common solution suggests pre-uploading the third party jars to the server and exclude them from the package. I've seen such a solutions and in my opinion they are the exact opposite of a good solution: in this way you split the package, the third parties become manages in two (sometimes more) places and each deployment involves at least additional (probably manual!) step of checking if the third parties were changed and if additional deployment if third parties is required.

But if you use Maven. And if you upload your released packages to Nexus (or actually any other Maven repository). This Nexus repository contains all the third parties, all the released packages and the most important: The pom file that was used to build your project!
If you download this pom file, you will be able to build the package on the production server! Pay attention that you don't need to do the full build that includes the compilation, testing and so on. You just need your package, so considering that you deploy a war, you only need to run the "mvn war:war" (Once again: I never tried it myself and the actual execution might be more complex, but I think that the idea is clear).
Sometimes, if you a running a java application with a main class (pure old java and not some kind of JEE inside the application server or a servlet container), you don't even need a package, you just need a correct classpath and Maven will be happy to assist you: mvn dependency:build-classpath.

So I guess that the idea is clear now. Each time Maven will download only the relevant jars and save them to the local repository. The dependencies are managed in the same pom file that is used to release the application, so when making a package, or creating a classpath on a production machine, the exact same dependencies will be used.
And the deployment process will become much faster!

I know that this idea is somewhat different from the usual process. Instead of doing some like "build, deploy, run", we do something that might look even more complicated: "build, deploy descriptor only, package, run". But this should be much faster. So I definitely think that this idea is worth trying.

P.S. The idea described in this post relates only to the package itself: building, packaging and running. The deployment may contains additional steps like changing the local configuration files and so on. These steps are not covered here as they are usually not covered in a build process, but part of release notes. The possible solution can be deploying the relevant scripts to Nexus repository and somehow describe them in a pom file. When downloading the pom, the relevant scripts will be also downloaded and executed.

P.P.S. The idea also doesn't cover the tool that makes the whole process. Although it describes that the tool is using Maven, it says nothing about the actual implementation. It might be a java process. Or a shell script. Or even Ant.

P.P.P.S. Notice that downloading files from Nexus using Maven makes important checks for you, for example it makes an integrity check, which is very important in case of a bad network between the Nexus with releases and a production site.
In addition, you can make some optimizations on Nexus. For example, if you have several production sites all over the world, each site may have a Nexus pointing to the main release repository and caching it. This will make the deployments even faster.

Deadlock in Jetty or Be Careful while Synchronizing

2013-02-13T16:53:00.001+02:00

About nine months ago I reported a bug to the Jetty community that session timeout doesn't work properly. The bug was fixed quite quickly, but nine months later I have discovered that the fix leads to a deadlock in some scenarios.

Deadlock in Jetty illustrates an interesting coding guideline that you must follow while writing your code.

So what happened in Jetty?

Consider a class A that carries state and lives in a multi-threaded environment. Obviously this class must be synchronized.
Consider that you can subscribe to events of class A. So let's say class you must implement an interface I that will be notified when something important in class A happens.
Let's assume that the method in which class A invokes instances of I is synchronized (A carries state, remember?)
Let's also assume that your implementation of I also carries state and must be synchronized as well.

And now let's see what happens:

Thread-1: Some event on A occurs. It's wants to notify I, but first acquires LOCK_A and then invokes method of I. Method of I tries to change state of I, so it tries to acquire LOCK_I, but it was already acquired by Thread-2.

Thread-2: Runs on I. It changes the state of I, so it acquires the LOCK_I. During the change it needs some information of A. It tries to get it, but LOCK_A was already acquired by class A.

And here we have a deadlock.

So what is wrong here?
The most wrong part is of class A: it invokes method of some other class while it is locked. BAD! Finish the lock before calling someone else! And when I say "someone else" I include the other methods of the class! (What really happened in Jetty is that in class A method f1() was synchronized. Method f1() called to f2(), which called to f3(), which called to f4(), which called to I. It was clear in f4() that no synchronization is needed. But the mistake is actually in f1()!)
So you have some member to change? acquire the lock, change them and release the lock.

In addition, the situation could be a little improved if Read-Write lock was used instead of synchronized: most of the access to class are to read data. May be if LOCK_A was split to READ_LOCK_A and WRITE_LOCK_A; and LOCK_I was split to READ_LOCK_I and WRITE_LOCK_I, it was not causing the deadlock. But this is not about preventing the situation, but about improving.

Summary

The main point of my post is that when synchronizing, find the critical section and synchronize it only! Do not call other methods (even if they are of the same class) from the critical section: gather all information before and notify everyone else after.

Initiate Infinispan with a Custom JChannel

2013-01-14T16:53:00.001+02:00

In all Infinispan sample the JGroups part is almost omitted. You are told that you should provide a configuration xml and the infinispan-core.jar contains three samples: udp, tcp and ec2.

But what happens if you don't want to use samples? What happens if you want to have a dynamic configuration that changes based on the environment/database/external configuration tool? What happens if you want to configure it from code?

It appears to be not an easy task, but not very complicated. A special interface org.infinispan.remoting.transport.jgroups.JGroupsChannelLookup exists that allows an injection of org.jgroups.Channel.

So you can write something like:

public classJGroupsChannelLookupImpl implements JGroupsChannelLookup {
   
    public Channel getJGroupsChannel(Properties p) {
        return new JChannel();  // of course the code here might be quite complex, initializing the channel with a custom configuration
    }

   
    public boolean shouldStartAndConnect() {
        return true; // change to false if you start and connect the channel yourself
    }

    
    public boolean shouldStopAndDisconnect() {
        return true; // change to false if you stop and disconnect the channel yourself
    }
}

Make sure to have a default public noargs constructor or a static getInstance() method that returns an instance of JGroupsChannelLookup.
(Personally I prefer a static method, so I have control how the instance is created and managed)

Now you need to tell the Infinispan to use your JGroupsChannelLookup:

Configuration conf = new ConfigurationBuilder().clustering().cacheMode(CacheMode.REPL_SYNC).sync().build();
GlobalConfiguration globalConf = new GlobalConfigurationBuilder().clusteredDefault().transport()
                    .addProperty(JGroupsTransport.CHANNEL_LOOKUP, JGroupsChannelLookupImpl.class.getName()).asyncTransportExecutor()
                    .build();

            
DefaultCacheManager manager = new DefaultCacheManager(globalConf, conf);

Send Mail Via Gmail

2012-10-02T14:29:00.001+02:00

I already spent some time few years ago to figure that out and now I spent this time again.
So here's the short code snippet with an example how to send mail using Java Mail and GMail.

First, pom.xml. Include the following dependencies:

<dependency>
    <groupid>javax.mail</groupId>
    <artifactid>javax.mail-api</artifactId>
    <version>1.4.5</version>
</dependency>
<dependency>
    <groupid>javax.mail</groupId>
    <artifactid>mail</artifactId>
    <version>1.4.5</version>
</dependency>

And now the code:

import java.util.Date;
import java.util.Properties;

import javax.mail.Message;
import javax.mail.Session;
import javax.mail.internet.InternetAddress;
import javax.mail.internet.MimeMessage;

import com.sun.mail.smtp.SMTPTransport;

public class SendMail {

 public static void main(String[] args) throws Exception {
  Properties props = System.getProperties();
  props.put("mail.smtps.host", "smtp.gmail.com");
  props.put("mail.smtps.auth", "true");
  Session session = Session.getInstance(props, null);
  Message msg = new MimeMessage(session);
  msg.setFrom(new InternetAddress("somebody@gmail.com"));
  msg.setRecipients(Message.RecipientType.TO,
    InternetAddress.parse("somebody_else@gmail.com", false));
  msg.setSubject("I'm the subject!");
  msg.setText("And I'm text of your message");
  msg.setHeader("X-Mailer", "Java Program");
  msg.setSentDate(new Date());
  SMTPTransport t = (SMTPTransport) session.getTransport("smtps");
  t.connect("smtp.gmail.com", "somebody@gmail.com",
    "here comes the password");
  t.sendMessage(msg, msg.getAllRecipients());
  System.out.println("Response: " + t.getLastServerResponse());
  t.close();
 }
}

And the thanks goes here.

Apache HTTP Client - Ignore SSL Problems

2012-07-01T14:47:00.001+03:00

Sometimes, especially when testing, it can be useful to make Apache HTTP Client ignore the SSL problems.
SSL problems may include ignoring of certificate trust (issuers) and host verification. The following snippet creates an Apache HttpClient with SingleClientConnManager that will ignore the SSL problems:

import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;
import org.apache.http.conn.params.ConnRoutePNames;
import org.apache.http.conn.scheme.PlainSocketFactory;
import org.apache.http.conn.scheme.Scheme;
import org.apache.http.conn.scheme.SchemeRegistry;
import org.apache.http.conn.ssl.SSLSocketFactory;

public HttpClient createHttpClient() {
  TrustManager[] trustAllCerts = new TrustManager[] { new X509TrustManager() {

    @Override
    public java.security.cert.X509Certificate[] getAcceptedIssuers() {
     return null;
    }

    @Override
    public void checkClientTrusted(java.security.cert.X509Certificate[] certs, String authType) {
    }

    @Override
    public void checkServerTrusted(java.security.cert.X509Certificate[] certs, String authType) {
    }
  } };
  SSLContext context = SSLContext.getInstance("TLS");
  context.init(null, trustAllCerts, null);

  SSLSocketFactory sf = new SSLSocketFactory(context, SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER);

  SchemeRegistry schemeRegistry = new SchemeRegistry();
  schemeRegistry.register(new Scheme("http", 80, PlainSocketFactory.getSocketFactory()));
  schemeRegistry.register(new Scheme("https", 443, sf));

  SingleClientConnManager cm = new SingleClientConnManager(schemeRegistry);

  return new DefaultHttpClient(cm);
}

Pay Attention! In production you must use a valid SSL! Use this code for testing purposes only!

Jetty - Basic Hardening

2012-06-18T18:01:00.002+03:00

Jetty by default is shipped with two annoying features that should be turned off in production.

The first one is contexts listing. If you access the root folder, and if there is no special context configured to be a root context, Jetty will display a list of all contexts installed. While it may be nice to see it during the development, it's unnecessary information during production.

The class responsible for displaying this list is org.eclipse.jetty.server.handler.DefaultHandler and it's configured in jetty.xml. setShowContexts to false to turn off the contexts listing.

Actually you may consider to provide your own Hanlder class instead of the DefaultHanlder. The org.eclipse.jetty.server.handler.DefaultHandler is also responsible to display the Jetty's default favicon. It may be configured not to server the favicon at all by setServeIcon(false), but it does not allow to customize the favicon. So if you want to do it, you'll need a custom class.

The second annoying feature is a directory content listing - when you access a directory, the Jetty will generate a page with a list of files located inside. This configuration can be turned off per context: under <Configure class="org.eclipse.jetty.webapp.WebAppContext"> put the following init parameter:

<Call name="setInitParameter">
  <Arg>org.eclipse.jetty.servlet.Default.dirAllowed</Arg>
  <Arg>false</Arg>
</Call>

REST Best Practices: Use HTTP Status Codes

2012-05-07T15:10:00.000+03:00

When implementing RESTful service, keep in mind that HTTP already provides you with an ability to send the status code as part of the protocol. Do NOT put the error code inside the message itself!

HTTP defines five types of status codes:
* 1xx - Informational
* 2xx - Successful
* 3xx - Redirection
* 4xx - Client Errors
* 5xx - Server Errors

See Full reference of HTTP/1.1 Status Code Definitions

I'm not going to describe all HTTP Status codes here, but to give some basic tips.

1. In case of a successful request, always return the status code from the 2xx group. It's highly recommended to use not only the basic 200 code, but also additional codes. For example 201 means a new resource was created and the LOCATION header contains its path. 204 means that the response contains no content, so the client can optimize its code and not even try to read the content of the response (saves some redundant objects creation).

2. In case of an error request, clearly distinguish between the client and server errors. Client errors mean that the client has sent a bad request: it may be incorrectly formatted, unauthorized, method not allowed (e.g. Server accepts GET requests, while POST has been sent), and more.
In case of a client error, the best is to detect it as soon as possible in the server's code, to reduce amount of logic that will run, until the request is rejected. For example it's redundant to parse request body, if it misses the security header.
2.1 Send clear response to client. It can be a good practice to supply text or html content inside the message telling what was wrong in the request.
2.2 Don't log requests with a client's error above the info level. After all, it's a client's error, not server's. The only reason to log client' errors at all, is to help clients by taking look at the server logs. Make sure not to create a misleading picture of a log full with exceptions, while there is no problem at server side at all.
2.3 By receiving the 4xx code, the client must understand that it did something wrong and should correct its request.

3. In case of a server error, return 5xx error code. Usually the best choice will be to return simply 500. Do NOT put the full stacktrace into the response body. Actually put nothing in the response body. Why should a client care about the reason of server's failure. Is the database down? Is it a code problem? Whatever it is, it's not the client's business.
3.1 Sometimes, it can be nice to return 501 Not Implemented code. Usually it will happen, if you agreed on some functionality, created a prototype for it but didn't implemented it yet.
3.2 By receiving the 5xx code, the client must understand that something went wrong and that it should retry the same request. Does it make sense to retry immediately, in 5 minutes or in 5 days depends on the client. The server can return the *Retry-After* header and client should respect it.

There are more status code that can and should be used with RESTful APIs. For example 304 Not Modified allows you to save traffic and skip the response if the resource was not modified. Many status codes are implemented by the frameworks and intermediaries, for example JAX-RS frameworks will automatically return some client errors, like 405 and 415.

Summary

1. In case of successful request, always return a request from 2xx group.
2. In case of unsuccessful request, never return 2xx with embedded status code in the message. Return appropriate 4xx or 5xx status code.
3. For client errors, return 4xx. Don't log these requests above the info level. These are client's problems, not server's.
4. For server errors, return 5xx. Log the error, but don't send it to client.

Promoting tarlog-plugins

2012-05-03T15:37:00.004+03:00

In the last four years, since I have first published the tarlog-plugins Eclipse plugin, it has 4600+ downloads.
I know that many readers of this blog are actually using it.
I won't ask you why none of you has never clicked the Donate button. After all I have never did it myself...
But why you won't you share it?
You can star and/or +1 it on the project's page.
You can favorite it on the marketplace.
You can... well, the previous two are enough. But if you really want to Donate, there is a Donate button on the project's page.

P.S. And there is also an Encoder Tool. It didn't become as popular as tarlog-plugins, but it still nice for small encoding/decoding tasks.

REST Best Practices: Using HTTP Verbs

2012-04-16T10:01:00.001+03:00

The common mistake about RESTful API that they MUST be CRUD-like: Create maps to POST. Read maps to GET. Update maps to PUT. Delete maps to DELETE.

In fact this is incorrect. They MAY be CRUD-like or MAY be something else. The rule is: each resource MUST have a set of predefined operation. But not necessary CRUD. Even not necessary GET-POST-PUT-DELETE should be used. In fact did you know that HTTP/1.1 defines 8 HTTP Verbs: HEAD, GET, POST, PUT, DELETE, OPTIONS, TRACE and CONNECT? And in addition you may define your own HTTP Verbs that will behave as you wish? For example WebDAV defines the following verbs in addition to the standard ones: PROPFIND, PROPPATCH, MKCOL, COPY, MOVE, LOCK, UNLOCK.

So now, when you know that you may not stick to the standard HTTP verbs, I still advocate you to do so. Why? It's relatively clear what action these methods will do with the resource. It's may be less clear what action your own verb will do.

So let's consider that 90% of RESTful APIs will use only the standard HTTP verbs. What's important to pay attention to? The most important rule is to use the HTTP verbs correctly. The most common example of an incorrect use of the GET verb can be taken from HTML forms: in HTML both GET and POST can be used to submit data to a server from a form. But actually GET is a safe method, it means that it SHOULD NOT have the significance of taking an action other than retrieval.

So remember, Rule 1: HEAD and GET are safe methods.

Now about the Idempotent Methods. Methods are idempotent if the side-effects of N > 0 identical requests is the same as for a single request. The idempotent methods are GET, HEAD, PUT and DELETE.
Some examples what does it mean:
1. PUT can be used to create a resource. Yes, I know that the best practice say that it is POST, who creates resources and in a second I'll explain why. So let's say PUT creates a resource, but if a request is duplicated for some reason, the second request must not create a new resource and return exactly the same response as the first request. It's important to understand that various HTTP intermediaries know that PUT is idempotent and for example they may apply some caching, etc. So actually POST is much safer, especially for creating resources. But pay attention: POST may be safely used also for updates and even fetching the resource! Nobody said it is only for creation.
2. When DELETE-ing the resource, pay attention that the second DELETE request should return the same response. So if a first response returns "200 OK", the second (third, fourth, etc) DELETE requests to the same resource also should return "200 OK". Pay attention that once the resource was deleted, the GET request to this resource should return "404 NOT FOUND", but DELETE should continue returning "200 OK".

So Rule 2 will be: use idempotent methods correctly. If you cannot ensure that you use them correctly, use other methods and POST can be a good choice.

Now a little about the methods that you are unlikely to implement yourself. The behavior of the HEAD, TRACE, CONNECT and OPTIONS is well defined and they are usually implemented by the infrastructure (CONNECT and TRACE by the web servers, HEAD and OPTIONS by the frameworks, like Servlets or JAX-RS). If, for some reason, you decide to implement one of these methods, make sure your do it correctly.

Summary

1. Don't try to map CRUD to HTTP verbs. There is no need to do it.
2. Create new HTTP verbs, if needed.
3. Use existing HTTP verbs correctly, pay special attention to the safe methods (HEAD and GET) and to the idempotent methods (HEAD, GET, PUT and DELETE).
4. If you are now sure which HTTP verb to choose, use POST.

REST Best Practices: Use JSON

2012-04-11T14:35:00.000+03:00

Something that RESTful design does not care about, is the actual representation. The concept say that the resource can be represented in one representation or more. It does not affect the resource itself. The JAX-RS solved it very beautifully with a clear separation between the Resources and Providers.

And then happens a real life, and you need to implement a resource and represent it somehow. The native choice of the programmers who come to the RESTful world from SOAP is XML.
There are several reasons for it, and the main one: they were used to xml and why change something that works? Or at least worked?
Wrong! You don't need to stick to something you have used to use. You can use something better. And JSON is better. Why? The main reason: it's more simple - it has no namespaces and no attributes. It's less to write, less places to make a mistake in.
The compatibility issues with JSON are solved better, since you don't have xsds and you cannot fail with the validations. The new fields will be silently ignored by the old version and that's all.
You can always create XML out of JSON in case your client prefer to get XML. But it's not so easy to create JSON out of XML (yes, I heard of Badgerfish, but if you start your design from JSON and then add XML, you don't need Badgerfish).
JSON is also less verbose, which saves you the traffic, but really it's not the main reason to use it. The main reason is SIMPLICITY!

Now a little implementation note for those of you, who use Java: Create the classes that you want to be sent/received by the resources as POJOs. Keep them simple and prefer using String as a major data type. If you need something complex, like Date, do the formatting yourself and describe it in your documentation (SimpleDateFormat will help you). Once you have a set of simple POJOs, use Jackson to create/parse JSON and JAXB (integrated in Java 6+) to create/parse XML. Jackson comes with JAX-RS Providers. JAXB Providers are already included by the JAX-RS implementations. Thus in one shot you'll get both JSON and XML representations for your resources.

Back to the representations. And what about if you cannot use JSON? Of course such a thing can happen. For example you may want to upload image and there is no need to try to fit it in JSON. It's fine. Use whatever representation you want, but AVOID CUSTOM REPRESENTATIONS! If you feel that you must invent some new serialization mechanism that was not invented before, think twice. Are you really so unique? Really? Remember, you may have different clients. Each of them will have to know your serialization mechanism, will it be simple for them? Or you'll need to create a custom library. Remember: there are a lot of different languages, technologies and platforms, and you may need to support all of them.

Summary

1. Design your APIs to send/receive JSON.
2. So if you need XML, it can be easily added without writing a single line of code (at least with JAX-RS)
3. Need to send/receive something binary? Fine. But make your best to use standard formats and don't reinvent the wheel (until you feel that it's a must)
4. Never use Java standard serialization mechanism. If you need to serialize the Java class, read again #1 and use JSON ;)

REST Best Practices: Create a good URL

2012-04-11T10:56:00.001+03:00

REST is not a standard, therefore you are free to choose how to use it. It's very hard to say, if you do something "100% right" or "100% wrong". And still there are good RESTful APIs and bad RESTful APIs. Probably the most important part of your RESTful API are the URLs. They identify your resources. Good designed URLs make your API look good.

And here come some practices that I personally believe are best (or just good):

1. URL must uniquely identify the resource - it should be impossible using the same URL to access two different resources based on something else (e.g. header).
Example: suppose we design a RESTful API for a library. Let's say the url /books/ABC returns Winnie the Pooh for a registered user. A very BAD practice would be, if the unregistered user will get a different book for the same url. It should not matter who is the user, the same url should lead to the same book.
Now, of course our application may have security implications, so for example registered user can see the book and unregistered cannot. It's ok, return 404 NOT FOUND for unregistered user. Or eliminate the book from the book search. But NEVER return a different resource.

2. URL should be designed for further API changes - this one is kinda tricky. Suppose a registered user can add some books to a favorite list. What URL will be used?
Option 1: "/user/{userid}/favorites/" will return favorites list of a user. Sounds reasonable. But what happens if we decide to extend this feature and allow a user have multiple list, what will return this url now?
Option 2: "/user/{userid}/favorites/" will return a list of the favorites lists and "/user/{userid}/favorites/{id}" will return the specific list. Sounds reasonable. But what happens if we add a feature that allows user to share lists? Actually here we got to the point I'll discuss in the next bullet, but it's quite clear here that "/user/{userid}" should not be part of the url, right?
Option 3: "/favorites/" will return a list of the favorites lists and "/favorites/{id}" will return the specific list. Both "/favorites/" and "/favorites/{id}" resources will return the lists based on the user's privileges: the system administrator will see all lists, the user will see his lists the ones that were shared with him.

3. Security is not part of the resource identification and therefore should not be a part of the URL - in the previous bullet I have already described why adding a userid to the url is problematic. Let's talk about it a little bit more. First, you cannot rely "userid" present in the url in order to identify the user. You need a different form of authentication (unless you use username+password authentication and you put a password also in url, it's possible, but it's really a VERY BAD practice). Furthermore, the good api should not contain the definition of an authentication method within it at all. It should be possible to change the authentication method without changing the API (for example it quite common in the last few years to move from username+password to OAuth authentication). Basically the API should expect to receive the userid always, but not as an integral part of an API! With HTTP it's quite easy to put the security related stuff in the headers, and to keep the url clean.

4. If you must put additional metadata on URL, put it as a query parameter - In general all metadata you have (like security, content-type, accept content-type) should be in headers. However, sometimes it for some technical reasons it become impossible to put it in a header, so you must use URL. It's ok, but put it after the question mark in the query parameters part. Thus it becomes "optional" part and can be easily changed or removed if not needed later.

Why REST? Or "Simple vs. Easy"

2012-04-10T17:28:00.002+03:00

This is a post that I want to write for years and cannot find the correct words. I'm fully aware that hundreds of blog posts have already fully covered this subject and still I want to write my opinion in my blog. It is my blog after all, right?

So "Why REST?"
But first, I'd like to say two words. One is "simple" and the second one is "easy". Although these words sound like synonyms, actually they are not. Some simple things can be not easy. And some easy things can be not simple.
For example: breathing is easy for most people, you don't need to think about it. But is it simple? Can you make someone breathe, if he doesn't?
On contrary cleaning an apartment is simple. But is it easy? Not for me at least...

The key point here is that simple things just work. They may not be easy to achieve, but once you do it, they work. Once you clean an apartment, you have a clean apartment. For a while at least.
But when something is not simple, its quite problematic even to understand how it should work. Sometimes it's easy to start (breathe for example), but what happen if something goes wrong (stop breathe for example)?

And now back to REST.
REST is simple. Not easy, but simple. Correct RESTful APIs are very clear, very simple to understand. They may not be easy to implement, but once implemented they work. It's quite easy to troubleshoot the RESTful APIs, you can do with with a simple HTTP Proxy (e.g. Fiddler). It ensures quite a decoupling between server and client.
Yes it takes time to develop a RESTful API. Yes there is no much automation around and suddenly a developer needs to develop a layer that he got generated with SOAP.

And now to SOAP (and other APIs that are generated from wslds/custom xmls/whatever): this one is easy. Usually I can get some working API very fast. There are a lot of tools that help you. BUT, it is not simple: the tools generate a lot of code that you don't know and sometimes not very readable. The integrations become very complex (did you ever install ESB?) The versioning becomes a nightmare. And the standards don't work very well (did you ever tried to integrate Java to .NET using some complex types?)

So the bottom line is: REST is simple, but not easy. SOAP is easy at the beginning, but not simple.
Simple is good, specially if you continue keeping it simple.
And forget about easy. Nobody said that a software development should be easy.

Changing Putty Default Settings

2012-04-04T17:49:00.000+03:00

It's trivial, I should have figure it out myself, but I suffered for many years until finally I googled for it, and voila: To save a default setting in putty, open putty, change the setting, in "Saved Sessions" choose "Default Settings" and click "Save".

Integration Testing of RESTful Application

2011-12-08T17:26:00.002+02:00

In the previous posts I have described how to start Jetty from code in the beginning of the unit tests and how to initialize the in-memory HSQL database once Jetty is started.

If you have completed the steps from these posts, you should have a running application in the beginning of your tests. And now you are ready to start the actual testing of the application.

I believe that the best way to test RESTful API is to issue actual HTTP requests and since we have a Jetty server running, it becomes possible.

There are a lot of HTTP Clients available in Java. The examples below use Apache HTTP Client.
But as always first let's add a maven dependency:

<dependency>
         <groupId>org.apache.httpcomponents</groupId>
         <artifactId>httpclient</artifactId>
         <scope>test</scope>
        </dependency>

Now some convenient static methods that can be used:

import org.apache.http.HttpResponse;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpDelete;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.methods.HttpPut;
import org.apache.http.entity.ByteArrayEntity;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.message.AbstractHttpMessage;
import org.apache.http.message.BasicHeader;
...
   static HttpResponse executeGet(String url) throws IOException {
        HttpClient httpclient = new DefaultHttpClient();
        HttpGet get = new HttpGet(url);
        return httpclient.execute(get);
    }

    static HttpResponse executeDelete(String url) throws IOException {
        HttpClient httpclient = new DefaultHttpClient();
        HttpDelete delete = new HttpDelete(url);
        return httpclient.execute(delete);
    }

    static HttpResponse executePost(String url, byte[] body) throws IOException {
        HttpClient httpclient = new DefaultHttpClient();
        HttpPost post = new HttpPost(url);
        post.setEntity(new ByteArrayEntity(body));
        return httpclient.execute(post);
    }

    static HttpResponse executePut(String url, byte[] body) throws IOException {
        HttpClient httpclient = new DefaultHttpClient();
        HttpPut put = new HttpPut(url);
        put.setEntity(new ByteArrayEntity(body));
        return httpclient.execute(put);
    }

So your test can be something like:

@Test
    public void testGet() throws Exception {
        HttpResponse response = executeGet(url, null, null, null);
        assertEquals(response.getStatusLine().getStatusCode(), 200);
    }

Automated Integration Tests Using with Jetty, Maven and Other Neat Freameworks - 2

2011-12-08T10:48:00.006+02:00

In the previous post, I have started a Jetty server in the beginning of a unit test with a configured external data-source.

Let's talk about it a little bit more. The assumption here was that the application on a regular basis uses an external data-source that is accessed via JNDI. In general it's a good practice to keep a data-source external to the application:
1. It's always possible to change the data-source without touching the application - let's say a bug was found in a data-source you are using. Or it should be configured differently. If a data-source embedded into an application and such a change is required, you will probably need to release patch. If the datasource is external, it would be enough just to change/reconfigure it.
2. In some deployments several applications can use the same datasource. Consider a Tomcat running with several wars: quite a common case, right? If a datasoure is embedded, each war has its own data-source. Usually this would be a waste of resources and duplication of a configuration.
3. And finally it would be possible to change a data-source for unit tests, which is exactly our use case.

For unit tests it's really convenient to use in-memory database. There are few databases implemented in java that provides this capability. My favorite are H2 and HSQL. In this example I have used HSQL, so here comes the jetty-ds-test.xml:

<Configure id="Server" class="org.eclipse.jetty.server.Server">
    <New id="DSTest" class="org.eclipse.jetty.plus.jndi.Resource">
        <Arg></Arg>
        <Arg>jdbc/my_ds</Arg>
        <Arg>
            <New class="org.hsqldb.jdbc.JDBCDataSource">
                <Set name="Url">jdbc:hsqldb:mem:test;sql.syntax_ora=true</Set>
                <Set name="User">sa</Set>
                <Set name="Password">sa</Set>
            </New>
        </Arg>
    </New>
</Configure>

This is configuration of HSQL in memory. Notice sql.syntax_ora=true, which makes HSQL to use Oracle syntax. This is useful if you are using Oracle in production, but you want to use HSQL for unit testing or development.

So now you have at the beginning of a unit-test you have a Jetty server running with your application connected to in-memory HSQL database via JNDI. But something is still missing. This something is a database schema: to remind you, we have just started a new database in memory and it's empty. You probably already have a script that generates schema, tables, indexes and may be some data. Now you need to run it.

Actually there are several options how do it. One of the easiest is to use Springs's SimpleJdbcTestUtils. But first we'll need to add a dependency on spring-test in our pom.xml:

<dependency>
         <groupId>org.springframework</groupId>
         <artifactId>spring-test</artifactId>
         <scope>test</scope>
         <version>${spring-version}</version>
</dependency>

And here comes the code that runs the sql script:

import javax.naming.InitialContext;
import javax.naming.NamingException;
import javax.sql.DataSource;
import org.springframework.core.io.ClassPathResource;
import org.springframework.core.io.Resource;
import org.springframework.jdbc.core.simple.SimpleJdbcTemplate;
import org.springframework.test.jdbc.SimpleJdbcTestUtils;
...
    @BeforeClass(dependsOnMethods = { "startJetty" })
    public void initiateDatabase() throws Exception {       
        InitialContext initialContext = new InitialContext();
        DataSource ds = (DataSource) initialContext.lookup("jdbc/my_ds");
        SimpleJdbcTemplate simpleJdbcTemplate = new SimpleJdbcTemplate(ds);
        Resource resource = new ClassPathResource(sqlScriptFileName);
        SimpleJdbcTestUtils.executeSqlScript(simpleJdbcTemplate, resource, false);
    }

In this snippet I load the sqlScriptFileName from the classpath. Usually it's convenient to place the script in src/test/resources, but if you don't like it, you can always load it from a different place by using other Resource implementations (e.g. URLResource is quite convenient).

As I have already said in this snippet I used Spring. If you are familiar with Spring, it is probably natural to you. If you don't - don't be afraid. Only the unit tests become dependent on Spring, but the actual application did not.

And now you are ready to start the testing.

Automated Integration Tests Using with Jetty, Maven and Other Neat Freameworks

2011-12-08T10:02:00.008+02:00

Let's say that you have a web application (aka war) that exposes RESTful API and connects to a database. Now you want to do some automation tests (and I'm not going to describe here why actually you must have automation tests that run regularly on your API).

One of the best solutions to do it, in my opinion, is running your application on a Jetty server that is embedded in your unit test with some in-memory database.

But let's do it step-by-step:

Step1: Start Jetty from a Unit Test

(In this guide I have used TestNG, but I see no reason why the same functionality cannot be achieved in JUnit)

So first we need to start Jetty in our test. But before we actually do that, we need to make sure that our Maven project contains the relevant dependencies. So first we need Jetty:

<dependency>
            <groupId>org.eclipse.jetty</groupId>
            <artifactId>jetty-server</artifactId>
            <version>${jetty-version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.eclipse.jetty</groupId>
            <artifactId>jetty-webapp</artifactId>
            <version>${jetty-version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.eclipse.jetty</groupId>
            <artifactId>jetty-jndi</artifactId>
            <version>${jetty-version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.eclipse.jetty</groupId>
            <artifactId>jetty-plus</artifactId>
            <version>${jetty-version}</version>
            <scope>test</scope>
        </dependency>

The Jetty dependencies in this guide already contains the JNDI support. It will be needed later. But if JNDI support is not required, they can be omitted.

And then let's start it before the tests start:

import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.webapp.WebAppContext;
import org.eclipse.jetty.xml.XmlConfiguration;
import org.testng.annotations.BeforeClass;

public class MyTest {

    private static final String RESOURCES_URL = "/rs";
    private static final String CONTEXT       = "/app_context";
    private static final String DS_CONFIG     = "/jetty-ds-test.xml";
    private String              baseResourceUrl;

    @BeforeClass
    public void startJetty() throws Exception {
        Server server = new Server(0);   // see notice 1
        server.setHandler(new WebAppContext("src/main/webapp", CONTEXT)); // see notice 2

        // see notice 3
        InputStream jettyConfFile = InboxTest.class.getResourceAsStream(DS_CONFIG);
        if (jettyConfFile == null) {
            throw new FileNotFoundException(DS_CONFIG);
        }
        XmlConfiguration config = new XmlConfiguration(jettyConfFile);
        config.configure(server);

        server.start();
        
        // see notice 1
        int actualPort = server.getConnectors()[0].getLocalPort();
        baseResourceUrl = "http://localhost:" + actualPort + CONTEXT + RESOURCES_URL;
    }

Please notice that:
1. Jetty is started on a random port. The actual url with the actual port is saved to baseResourceUrl to be used later by tests.
2. Web application context points to maven's src/main/webapp.
3. Jetty is started with a data source configuration. (See Runnig Jetty from Maven using JNDI Data Source)

Part 2

Runnig Jetty from Maven with JNDI Data Source

2011-12-05T09:50:00.005+02:00

During the development of a java web component (aka war) it can be very useful to run the application as quick as possible. Jetty provides a Maven plugin that allows running it directly from maven build or explicitly using "mvn jetty:run" from the command line.

But what happens if the war uses external database? Especially when the datasource is defined externally to the war and accessed vie the JNDI?

Actually the solution if quite simple:

Step 1 - Define the Data-source

Create the file defining the data-source:
(The example below is for Oracle. See this page for examples of the other databases.

<Configure id="Server" class="org.eclipse.jetty.server.Server">
    <New id="DSTest" class="org.eclipse.jetty.plus.jndi.Resource">
        <Arg></Arg>
        <Arg>jdbc/my_ds</Arg>
        <Arg>
            <New class="oracle.jdbc.pool.OracleDataSource">
                <Set name="DriverType">thin</Set>
                <Set name="URL">jdbc:oracle:thin:@(DESCRIPTION=(ENABLE=BROKEN)(ADDRESS=(PROTOCOL=TCP)(HOST=127.0.0.1)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=something_to_replace)))
                </Set>
                <Set name="User">my_user</Set>
                <Set name="Password">my_password</Set>
                <Set name="connectionCachingEnabled">true</Set>
                <Set name="connectionCacheProperties">
                    <New class="java.util.Properties">
                        <Call name="setProperty">
                            <Arg>MinLimit</Arg>
                            <Arg>10</Arg>
                        </Call>
                        <Call name="setProperty">
                            <Arg>InactivityTimeout</Arg>
                            <Arg>600</Arg>
                        </Call>
                    </New>
                </Set>
            </New>
        </Arg>
    </New>
</Configure>

Pay attention to "jdbc/my_ds". It's the datasource JNDI name. Make sure to put there the actual JNDI name used by your application.

Place this file under your maven project. In my opinion the best is to place this file under src/dev/resources, but basically it can be anywhere.

To continue the example, I'll name the file: jetty-ds-dev.xml

Step 2 - Define the Jetty Maven Plugin

<build>
...
        <plugins>
...
            <plugin>
                <groupId>org.mortbay.jetty</groupId>
                <artifactId>jetty-maven-plugin</artifactId>
                <version>${jetty-version}</version>
                <configuration>
                    <jettyConfig>src/dev/resources/jetty-ds-dev.xml</jettyConfig>
                </configuration>
                <dependencies>
                    <dependency>
                        <groupId>com.oracle</groupId>
                        <artifactId>ojdbc14</artifactId>
                        <version>${oracle-ojdbc-version}</version>
                    </dependency>
                </dependencies>
            </plugin>
        </plugins>
    </build>

Pay attention to the configuration of jetty-ds-dev.xml.
Also pay attention that Jetty must be able to find the relevant JDBC driver in its classpath!

And basically that's all. Run "mvn jetty:run" and your application should work with the provided database.

Mount external path using cifs

2011-09-27T17:47:00.003+03:00

How to mount Windows server from Unix using Cifs:

mount -t cifs //host:port/path /mnt -o user=username,pass=password

Don't forget to replace host, port, path, mnt, username and password.

Simple Batch Script to Run Tomcat

2011-09-26T18:25:00.006+03:00

Sometimes it's a good idea to separate the actual Tomcat installation from the running configuration. Actually Tomcat comes with this ability built-in:

catalina.base

system property defines the location of the configuration, while

catalina.home

system property defines the location of the Tomcat installation.
And here comes the basic script that runs the Tomcat:

@echo off
setlocal

SET TOMCAT_BASE=c:\work\tomcats\tomcat1
SET TOMCAT_HOME=C:\tps\apache-tomcat-7.0.16
SET TOMCAT_CLASSPATH=%TOMCAT_HOME%\bin\bootstrap.jar;%TOMCAT_HOME%\bin\tomcat-juli.jar;%JAVA_HOME%\lib\tools.jar
SET JAVA_ENDORSED_DIRS=%TOMCAT_HOME%\endorsed
SET JAVA_OPTS=
SET JAVA_OPTS=%JAVA_OPTS% -Djava.util.logging.config.file="%TOMCAT_BASE%\logging.properties" -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
SET JAVA_OPTS=%JAVA_OPTS% -Djava.io.tmpdir="%TOMCAT_BASE%\temp"
SET JAVA_OPTS=%JAVA_OPTS% -Dcatalina.base="%TOMCAT_BASE%" -Dcatalina.home="%TOMCAT_HOME%"
SET JAVA_OPTS=%JAVA_OPTS% -Djava.endorsed.dirs="%JAVA_ENDORSED_DIRS%"

REM SET JAVA_OPTS=%JAVA_OPTS% -Xdebug -Xrunjdwp:transport=dt_socket,address=7878,server=y,suspend=n

%JAVA_HOME%\bin\java %JAVA_OPTS% -classpath %TOMCAT_CLASSPATH% org.apache.catalina.startup.Bootstrap start

endlocal

Kindle with Special Offers

2011-06-24T22:28:00.003+03:00

Amazon now sells the Kindle with Special Offers for just $114 without 3G ($164 with 3G). It's $25 less then the original price.
The idea that it shows sponsored screen savers, when a Kindle is not in use.

I'll probably buy another one.

Cookiejacking

2011-05-30T14:43:00.004+03:00

Pretty nice vulnerability in IE (could reproduce it easily in IE 7)

In short: stealing cookies in IE is pretty easy.

Method has the same erasure bug in javac

2011-03-31T17:38:00.001+02:00

Interesting bug in javac. Consider the two cases:
Case 1:

class Case1 {
    public String foo(Set<String> str) {
        return null;
    }

    public String foo(Set<Integer> str) {
        return null;
    }
}

Case 2:

class Case2 {
    public String foo(Set<String> str) {
        return null;
    }

    public Integer foo(Set<Integer> str) {
        return null;
    }
}

What do you think about them? Should Class1 compile? What about Class2?
In my opinion both of classes should not compile, since methods have the same erasure. In this is exactly what happens in Eclipse.
But, in IntelliJ the Class2 compiles just fine. And Maven also compiles the Class2 without problem.
So how does it happen?
It appears that Eclipse uses it own compiler, while Maven and IntelliJ use javac.
So where is the bug, in Eclipse or javac?
Here the bug that was opened in Eclipse community.
And here the bug in Sun (or should I say Oracle?)
So as you can see, both of them agree with me: the bug is in javac and not in Eclipse. Actually Sun marked it as fixed at 7(b55). Does it mean Java 7?

Maven - Hotswap Plugin

2011-03-28T21:50:00.000+02:00

For many years I was missing the following feature of Maven: In a multi-module project, when I build a single module in a jar, I want it to be installed into the running environment (app server) seamlessly. During these years I created various scripts for this tasks, why none of them lasted too long: it's very hard to create a generic script, while nongeneric scripts need to be maintained and actually each module must have had its own script. Duh.

Thinking about it, I decided that it should be a Maven plugin, which is very simple - during the install phase, search some predefined location and hotswap the found jars. That's it. So I thought that somebody probably already thought about it and asked a question at Stackoverflow. I received few interesting answers, but not a simple plugin I needed.

So I decided to implement it myself. It took me few hours, since it was my first Maven plugin, so I had some "doing it for the first time" troubles. But finally I did it.

You are welcome to use it, and as always any feedback is appreciated.

How I didn't get even a job interview in Google

2011-03-23T16:55:00.002+02:00

This is going to be a very different post and it is not going to be technical.
I wish to register a complaint:

But first some history: Once upon a time I have sent my resume to Google. I did it via the Google web site, got an automatic reply and was never contacted. You may say that sending the resume via the company web site is not effective way to find a job, and you are right. But actually I was not really looking for a job, I saw a Google advertisement on the web, and decided to apply.

About a month ago, I got the following mail:

Subject: Hello from Google/Google Opportunities

Hi Michael,

My name is ****** ***** and I am an Engineering Recruiter for Google. Your name was passed to me confidentially by an engineer at our Mountain View headquarters.

I see that you have applied, via our website, to opportunities in our offices in Israel, but nothing materialized. If you are interested in positions in those offices I can pass your information to someone who supports that office and can make sure that you are given serious consideration for those positions. Otherwise, I support our HQ and other offices in the US and if you are interested in a major relocation I can assist. If you are interested in opportunities with us please send me an updated copy of your resume along with your reply.

Best regards,

******

To tell the truth my first idea was to say "Yes! I want to come to an interview to Mountain View. Please, make sure that I fly business/first and live in a nice hotel."
But I'm probably too shy to ask for traveling, while I'm not actually interested in a relocation. While position in Tel-Aviv could be interesting, so I replied:

Hi ******,

Thank you for your mail.
I'm really curious to know, who is the person who passed you my name.
I'm asking, since I don't think that I know anyone, who works at Mountain View...

Anyway, I'm interested in the positions at Google offices in Israel. However, if it does not succeed, I won't be available for the relocation, since *********************************.

My updated resume is attached.
Thank you,

Michael.

And very quickly I received the following two mails:

Mail 1:

Michael,

Thanks for your response and for sending me your resume. We currently have Software Engineering openings in Haifa and Tel Aviv. I can get your resume and information over to a colleague of mine that supports those locations. I will forward your information over today.

The person that referred you definitely works in Mountain View, but I can't disclose their name.

Let me know if you have any questions and hopefully something with us in Israel will work out for you.

Best,

******

Mail 2:

Hi Tal,

My name is ****** and I work for the Google recruitment team.

I got your info from *********** from the USA Google Staffing.

I would like to ask if you would be interested in having an open conversation about current job opportunities to see if we have anything available that may interest you?

If you would like to discuss this further then please send me a convenient time/date and your telephone number.

I look forward to hearing from you.

Best Regards,

--
***************

My name is not Tal, but who cares :) So I replied with a phone number and few days later I got a phone call from the Google recruiter in Tel-Aviv. She asked me few questions about what I'm doing and what I'm interested to do and then said the following (this is not the exact quote, but this is how I remember it): "We don't invite to an interview the candidates who have such a low degree and such a low GPA, and I'm calling you only since a colleague from the USA team has recommended you. If you are interested I will send you an online programming exam and if you successfully complete it, I'll invite you to an interview."

Few points here: I own a BA degree in a Computer Science with GPA 85. I graduated in 2004 and have 7 years of an experience. I can hardly understand the recruiter who is interested in GPA of a person, who has graduated 7 years ago.
Anyway, she didn't reply and didn't send me any exam. I believe she forgot about me the second the phone was disconnected. May be even a second before.

So I do understand that she called me only because she got my resume from her colleague. The thing I don't understand: Is there such a difference between the requirements for Google in US and in Israel? I mean, I don't believe that the US recruited wrote me mail knowing that my degree is not high enough.

Anyway, I'm pretty sure that even if I was getting to an interview, my chances to pass it probably are not high, so I didn't have any real expectations. It's just a nasty feeling that I have after this a very short recruitment process that wasn't even initiated by me.

Hibernate - How to Insert Data During the Create/Create-drop

2011-03-17T15:31:00.001+02:00

And here comes something I was missing for years.
It appears that it's possible to execute additional SQLs during the Hibernates create/create-drop. It's very useful for the unit testing.

In short: Hibernate will run "import.sql" file from the classpath.

I found it here.

Tarlog on Java

DevOps: Making Fast Deployments of Java Servers using Maven and Nexus

Recommended Reading

Deadlock in Jetty or Be Careful while Synchronizing

Summary

Initiate Infinispan with a Custom JChannel

Send Mail Via Gmail

Apache HTTP Client - Ignore SSL Problems

Jetty - Basic Hardening

REST Best Practices: Use HTTP Status Codes

Summary

Promoting tarlog-plugins

REST Best Practices: Using HTTP Verbs

Summary

REST Best Practices: Use JSON

Summary

REST Best Practices: Create a good URL

Why REST? Or "Simple vs. Easy"

Changing Putty Default Settings

Integration Testing of RESTful Application

Recommended Reading

Automated Integration Tests Using with Jetty, Maven and Other Neat Freameworks - 2

Recommended Reading

Automated Integration Tests Using with Jetty, Maven and Other Neat Freameworks

Step1: Start Jetty from a Unit Test

Recommended Reading

Runnig Jetty from Maven with JNDI Data Source

Step 1 - Define the Data-source

Step 2 - Define the Jetty Maven Plugin

Recommended Reading

Mount external path using cifs

Simple Batch Script to Run Tomcat

Kindle with Special Offers

Cookiejacking

Method has the same erasure bug in javac

Maven - Hotswap Plugin

How I didn't get even a job interview in Google

Hibernate - How to Insert Data During the Create/Create-drop