Thought Bytes

Using the Neo4j browser with Embedded Neo4j

2014-11-10T10:39:00.002+05:30

There are times when you have an application using Neo4j in embedded mode but also need to play around with the graph using the Neo4j web browser. Since the database can be accessed from at most one process at a time, trying to start up the Neo4j server when your embedded Neo4j application is running won't work.
The WrappingNeoServerBootstrapper, although deprecated, comes to the rescue.
Here's how to set it up.

1. Make sure you have these maven dependencies-


<dependency>
    <groupId>org.neo4j</groupId>
    <artifactId>neo4j</artifactId>
    <version>2.1.5</version>
</dependency>
<dependency>
    <groupId>org.neo4j.app</groupId>
    <artifactId>neo4j-server</artifactId>
    <version>2.1.5</version>
</dependency>
<dependency>
    <groupId>org.neo4j.app</groupId>
    <artifactId>neo4j-server</artifactId>
    <version>2.1.5</version>
    <classifier>static-web</classifier>
</dependency>

2. Start the WrappingNeoServerBootstrapper

public static void connectAndStartBootstrapper() {

    WrappingNeoServerBootstrapper neoServerBootstrapper;

    GraphDatabaseService db = new GraphDatabaseFactory()

            .newEmbeddedDatabaseBuilder("/path/to/db").newGraphDatabase();

    registerShutdownHook(db);

    try {

        GraphDatabaseAPI api = (GraphDatabaseAPI) db;

        ServerConfigurator config = new ServerConfigurator(api);

        config.configuration()

            .addProperty(Configurator.WEBSERVER_ADDRESS_PROPERTY_KEY, "127.0.0.1");

        config.configuration()

            .addProperty(Configurator.WEBSERVER_PORT_PROPERTY_KEY, "7575");

        neoServerBootstrapper = new WrappingNeoServerBootstrapper(api, config);

        neoServerBootstrapper.start();

    catch(Exception e) {

       //handle appropriately

}

}

Two things happen here- the GraphDatabaseService is ready to use in embedded mode, and the Neo4j web browser is available for use on http://127.0.0.1:7575/

You need not start them together but instead start and stop the WrappingNeoServerBootstrapper on demand, you just need to have a handle to the GraphDatabaseService.

Again, note that the WrappingNeoServerBootstrapper is deprecated. At the time of writing, this code works on 2.1.5 but does not offer any guarantees for future releases of Neo4j.

-Luanne

Integrate apps with Neo4j using Zapier

2014-07-18T00:00:00.000+05:30

Recently, I was directed to Zapier to get some lightweight integration done between systems for a quick proof of concept. Initially skeptical, I found that it really could save time and tie together all those pieces of your system you never got around to integrating.

Moreover, it is a way for people to integrate the applications they use without having to code or pay a developer to do it for you.

Going through the Zapbook, I found MongoDB, MySQL, Postgresql, SQL Server and gasp! no Neo4j. Sad :-(

I already had a potential use case which was to collect data via a form and get it into Neo4j ASAP i.e. no coding. Google Forms is available on Zapier, so I went about making Neo4j available as well. I’ve now got a first version zap ready for Neo4j which allows one to collect data triggered by another zap, and save it to Neo4j via a Cypher statement.

Here’s what it looks like. Using the Google Forms example, I’ve set up a form to capture feedback about a product and I want to push this data into Neo4j every time the form is submitted.

Step 1: Log into Zapier, click on Make a Zap!

Step 2: The triggering app is Google Docs, where we want to save data to Neo4j every time a form is filled i.e. the spreadsheet backing the form has a new row inserted.

The Neo4j zap currently supports only one action- Update the graph.

Step 3: Follow the instructions to make sure Zapier can access your Google Docs account

Step 4: Set up a Neo4j account. Call it whatever you like, supply the username, password and URL. Note that in this version, the assumption is that your Neo4j database is not left open to the world. I used the Authentication extension to set mine up.

Make sure you don't have a trailing / in the Neo4j Server URL. http://:/db/data is valid, http://:/db/data/ is not!

Click on Continue and make sure Zapier confirms that it can indeed access your Neo4j database

Step 5: Select your spreadsheet and the Worksheet that contains the data. Here’s what my spreadsheet looks like-

Step 6: Write a Cypher query to convert that row into nodes and relationships. You must write a parameterized Cypher query in the Cypher Query field.

The Cypher Parameters must contain a comma separated list of the parameter names used in the query and the field selected from the triggering app (use the Insert Fields button).

Important! The Cypher Query is just that- a parameterised Cypher query with no parameter value substitution from your google doc or other trigger.
The place that links the parameters with their values from the trigger is the Cypher Parameters field, where you list all your parameters in the form of a comma separated list- key:value, key:value etc. and the value is chosen from your trigger.
The screenshot below represents what this should look like.

Step 7: See what the trigger and action samples look like- then test it out and celebrate when it says Success!

I checked what my database looked like at this point and sure enough:

That’s all there is to it. Zapier will poll the triggering app every 15 minutes so by the time all your forms are filled, you have a Neo4j database filled with data!

I tried out the MongoDB->Neo4j and Trello->Neo4j integration and they worked well.

Whether you need a quick and dirty integration with Neo4j, or you want to collect data from other applications into Neo4j for later analysis, or you’re building a serious application, Zapier could be of use.

If you’d like to try it out, send @luannem a message and I’ll send you a beta invite.

And if you think this is useful, I’d be happy to hear about it and add more features to the Neo4j zap!

-Luanne
@luannem

Neo4j Unit Testing with GraphUnit

2014-06-16T10:32:00.000+05:30

Testing the state of an Embedded Neo4j database is now much easier if you use GraphUnit, a component of the GraphAware Neo4j Framework.

I tried replacing an existing Flavorwocky unit test with GraphUnit to check out the benefits.
Let's walk through a before-after case study.

The unit test in question is the one to test that a Pairing is saved correctly. A pairing must have exactly two ingredients. Each Ingredient node has a name; a Pairing has an affinity and an array of allAffinities (all affinities ever assigned to the pairing).

Before: Pseudocode to test that my code saved a pairing correctly in Neo4j

 Execute Cypher statement to find the first Ingredient by name (Chicken)  
   Fail if no results returned  
 Use the core Java API to check that a pairsWith relation exists to another node with label Ingredient and name Garlic  
   Fail if no relation exists  
   Fail if the other Ingredient node has the wrong/missing name  
 Use the core Java API to find the Pairing node from the two ingredient nodes and check that the properties on Pairing are set correctly  
   Fail if the Pairing node was missing or the properties were incorrect  
 Make sure that the graph contains ONLY these nodes and relations and no incorrectly created nodes floating around

That's quite a bit of code to write just to test a simple structure. And it's difficult to remember what it does and modify it when you come back to it after a considerably long period of time.

GraphUnit allows you to compare the entire state of your graph to a subgraph represented in the form of a data-creating Cypher statement.

After: Writing a Cypher statement to represent a subgraph with two ingredients and a pairing is very easy, and my test code reduces to

 String ingredientSubgraph =  
         "create (i1:Ingredient {name: 'Chicken'}), (i2:Ingredient {name: 'Garlic'}), " +  
         "(i1)<-[:hasIngredient]-(p:Pairing {affinity: 0.45, allAffinities: [0.45]}), " +  
         "(p)-[:hasIngredient]->(i2) merge (i1)-[:pairsWith]-(i2)";  
 GraphUnit.assertSameGraph(getGraphDb(), ingredientSubgraph);

This will make sure that my graph contains nothing but the nodes and relations created by the Cypher statement, and that every property matches exactly. Pretty cool.

GraphUnit exposes another method to match the existence of a subgraph in the entire graph. This will verify only the subgraph specified and ignore other nodes and relations.

 GraphUnit.assertSubgraph(getGraphDb(), ingredientSubgraph);

Check out the entire test in a simplified version of the flavorwocky project.

Note at this point in time, GraphUnit requires a handle to org.neo4j.graphdb.GraphDatabaseService hence it currently supports testing against an Embedded or Impermanent graph database.

Update: Testing against a Neo4j server is now possible using GraphAware's RestTest

An ImpermanentGraphDatabase is great to unit test against. Depending on your testing preferences, you can decide to create a new empty graph for every test or build up on the same one for a series of tests. BaseTest uses standard jUnit fixtures to do this.

I inject the desired flavour of the GraphDatabaseService into a factory class that produces the right database to be used either in production mode or testing, and that's all there is to it.

Apart from testing code, I found another use for GraphUnit this weekend- verifying that data imported into a graph is correct. I'm creating a 2014 World Cup graph using LOAD CSV to import a variety of data from scattered csv files. Visually inspecting the graph or querying it after an import run isn't quite a productive activity, so now I've got a single unit test that uses GraphUnit to verify subgraphs in my world cup database.

 public class ImportTest extends BaseTest {  
   @Test  
   public void testGoal() {  
     String goalSubgraph = "Match (country:Country {name: 'Australia'})<-[:HomeCountry]-(player {name: 'Tim CAHILL'}), (m:Match {id: 4}), (min:Minute {id: 35}) create (player)-[:Scored]->(goal:Goal{type: 'regular'})-[:In]->(m) create (goal)-[:AtMinute]->(min)";  
     GraphUnit.assertSubgraph(getGraphDb(), goalSubgraph);  
   }  
   @Test  
   public void testReferee() {  
     String refSubgraph = "match (m:Match {id:4}) match (c:Country {name:'Ivory Coast'}) create (m)-[:Referee]->(r:Referee {name:'Noumandiez DOUE'})-[:HomeCountry]->(c)";  
     GraphUnit.assertSubgraph(getGraphDb(), refSubgraph);  
   }  
 }

It's simple to get started- include the maven dependency and you're away!

-Luanne

Migrating the Flavorwocky database using the Neo4j Cypher JDBC Driver

2014-02-08T01:19:00.000+05:30

Time to upgrade the Flavorwocky graph from 1.8 to the new, shiny 2.0!

I’m used the Neo4j JDBC Driver to help with my migration process.

Since Flavorwocky was built with the Grails Neo4j plugin, there are a bunch of reference and type nodes that we don’t want any more (which implies that Flavorwocky will be moving off Grails soon)

There are 5 types of entities- each of them has a __type__ property on the node:

com.herokuapp.flavorwocky.Ingredient
com.herokuapp.flavorwocky.User
com.herokuapp.flavorwocky.Pairing
com.herokuapp.flavorwocky.LatestPairing
com.herokuapp.flavorwocky.Category

Then we’ve got the root node, connected via a SUBREFERENCE relation to a “Type” node, and each instance of that type is connected via the INSTANCE relation.

I want to get rid of these and move to labels.

Here are two views of the data as it exists currently:

Upgrade the database from 1.8 to 2.0

I followed the instructions from http://www.neo4j.org/download/upgrade-faq

In particular-

The upgrade basically happens by starting your Neo4j data store (after making the backup) with Neo4j 2.0 and the configurationallow_store_upgrade=true enabled in the configuration file conf/neo4j.properties. To avoid running into server startup issues as part of the upgrade, you can also use the neo4j-shell for that purpose. The command line call would then be:bin/neo4j-shell -path data/graph.db -config conf/neo4j.properties.

Use the Neo4j JDBC Driver

Get the Neo4j JDBC Driver binary from http://www.neo4j.org/develop/tools/jdbc or build it from the 2.0 branch at https://github.com/neo4j-contrib/neo4j-jdbc

I built it from source and dropped it into the lib directory at https://github.com/luanne/flavorwocky-jdbc-migrate

You can use the driver both with an embedded database or with the Neo4j server.

I chose to use it with the embedded database because that supports turning auto commit off, which enables me to verify that every entity is migrated correctly before applying the changes on the database.

Using the driver is straightforward.

The connection URL for an embedded file path is jdbc:neo4j:file:/path/to/db while the url for the server looks like jdbc:neo4j://:/ (authentication parameters are passed as properties) [See ConnectionFactory.java and the properties defined in db.properties]

Statements and ResultSets work as you’d expect them to- just swap out SQL for Cypher.

With a PreparedStatement, you need to remember to parameterize the Cypher query using {1},{2}, {3} etc. Named parameters are not supported (PreparedStatement itself does not support named parameters). [See MigrateDb.java]

Statement.executeUpdate will return 0 at the time of writing this- it is not yet implemented.

Pretty easy!

The source is available at https://github.com/luanne/flavorwocky-jdbc-migrate

Flavor of the month- Neo4j add-on for Heroku

Creating a JDBC Driver for Neo4j

Cypher Neo4j JDBC Driver

-Luanne

Neo4j Java REST Binding- Part 2 (Batching)

2013-07-25T18:40:00.001+05:30

In Part 1, we talked about setting up a connection to the Neo4j Server using the Java REST Binding.

Let’s now go into some details about transactions, batching, and what the REST requests really look like.

Make sure you turn on logging (set system property org.neo4j.rest.logging_filter to true) as described in Part 1.

We will change our code to execute these Neo4j API calls.

Example 1:

Transaction tx = graphDb.beginTx();
    Map props=new HashMap();
    props.put("id", 100);
    props.put("name","firstNode");
    Node node=graphDb.createNode(props);

    props.put("id",200);
    props.put("name","secondNode");
    Node node2=graphDb.createNode(props);

    node.createRelationshipTo(node2, DynamicRelationshipType.withName("KNOWS"));

    tx.success();
    tx.finish();
    
    result=engine.query("start n=node(*) return count(n) as total", Collections.EMPTY_MAP);
    Iterator iterator=result.iterator();
    if(iterator.hasNext()) {
        Map row= iterator.next();
        out.print("Total nodes: " + row.get("total"));
    }

Check the logs (for me, they appeared by default on the Tomcat console) and look for the REST calls. The code above produced:


INFO: 1 * Client out-bound request
1 >  POST http://localhost:7474/db/data/node
1 >  Accept: application/json; stream=true
1 >  X-Stream: true
1 >  Content-Type: application/json
1 >
{"id":100,"name":"firstNode"}

INFO: 1 * Client in-bound response
1 < 201
1 < Access-Control-Allow-Origin: *
1 < Transfer-Encoding: chunked
1 < Content-Encoding: UTF-8
1 < Location: http://localhost:7474/db/data/node/1
1 < Content-Type: application/json; stream=true
1 < Server: Jetty(6.1.25)
1 < 
{"extensions":{},"paged_traverse":"http://localhost:7474/db/data/node/1/paged/traverse/{returnType}{?pageSize,leaseTime}","outgoing_relationships":"http://localhost:7474/db/data/node/1/relationships/out","traverse":"http://localhost:7474/db/data/node/1/traverse/{returnType}","all_typed_relationships":"http://localhost:7474/db/data/node/1/relationships/all/{-list|&|types}","property":"http://localhost:7474/db/data/node/1/properties/{key}","all_relationships":"http://localhost:7474/db/data/node/1/relationships/all","self":"http://localhost:7474/db/data/node/1","properties":"http://localhost:7474/db/data/node/1/properties","outgoing_typed_relationships":"http://localhost:7474/db/data/node/1/relationships/out/{-list|&|types}","incoming_relationships":"http://localhost:7474/db/data/node/1/relationships/in","incoming_typed_relationships":"http://localhost:7474/db/data/node/1/relationships/in/{-list|&|types}","create_relationship":"http://localhost:7474/db/data/node/1/relationships","data":{"name":"firstNode","id":100}}

INFO: 2 * Client out-bound request
2 > POST http://localhost:7474/db/data/node
2 > Accept: application/json; stream=true
2 > X-Stream: true
2 > Content-Type: application/json
2 > 
{"id":200,"name":"secondNode"}

INFO: 2 * Client in-bound response
2 < 201
2 < Access-Control-Allow-Origin: *
2 < Transfer-Encoding: chunked
2 < Content-Encoding: UTF-8
2 < Location: http://localhost:7474/db/data/node/2
2 < Content-Type: application/json; stream=true
2 < Server: Jetty(6.1.25)
2 < 
{"extensions":{},"paged_traverse":"http://localhost:7474/db/data/node/2/paged/traverse/{returnType}{?pageSize,leaseTime}","outgoing_relationships":"http://localhost:7474/db/data/node/2/relationships/out","traverse":"http://localhost:7474/db/data/node/2/traverse/{returnType}","all_typed_relationships":"http://localhost:7474/db/data/node/2/relationships/all/{-list|&|types}","property":"http://localhost:7474/db/data/node/2/properties/{key}","all_relationships":"http://localhost:7474/db/data/node/2/relationships/all","self":"http://localhost:7474/db/data/node/2","properties":"http://localhost:7474/db/data/node/2/properties","outgoing_typed_relationships":"http://localhost:7474/db/data/node/2/relationships/out/{-list|&|types}","incoming_relationships":"http://localhost:7474/db/data/node/2/relationships/in","incoming_typed_relationships":"http://localhost:7474/db/data/node/2/relationships/in/{-list|&|types}","create_relationship":"http://localhost:7474/db/data/node/2/relationships","data":{"name":"secondNode","id":200}}

INFO: 3 * Client out-bound request
3 > POST http://localhost:7474/db/data/node/1/relationships
3 > Accept: application/json; stream=true
3 > X-Stream: true
3 > Content-Type: application/json
3 > 
{"to":"http://localhost:7474/db/data/node/2","type":"KNOWS"}

INFO: 3 * Client in-bound response
3 < 201
3 < Access-Control-Allow-Origin: *
3 < Transfer-Encoding: chunked
3 < Content-Encoding: UTF-8
3 < Location: http://localhost:7474/db/data/relationship/0
3 < Content-Type: application/json; stream=true
3 < Server: Jetty(6.1.25)
3 < 
{"extensions":{},"start":"http://localhost:7474/db/data/node/1","property":"http://localhost:7474/db/data/relationship/0/properties/{key}","self":"http://localhost:7474/db/data/relationship/0","properties":"http://localhost:7474/db/data/relationship/0/properties","type":"KNOWS","end":"http://localhost:7474/db/data/node/2","data":{}}

INFO: 4 * Client out-bound request
4 > POST http://localhost:7474/db/data/cypher
4 > Accept: application/json; stream=true
4 > X-Stream: true
4 > Content-Type: application/json
4 > 
{"query":"start n=node(*) return count(n) as total","params":{}}

INFO: 4 * Client in-bound response
4 < 200
4 < Access-Control-Allow-Origin: *
4 < Transfer-Encoding: chunked
4 < Content-Encoding: UTF-8
4 < Content-Type: application/json; stream=true
4 < Server: Jetty(6.1.25)
4 < 
{"columns":["total"],"data":[[3]]}

A total of 4 REST calls over the wire for that tiny piece of code. You definitely want to avoid this as far as possible.

Option 1 is to use Cypher as far as possible.

We could convert the first three REST calls into one by not using the embedded style API and switching to Cypher.

Example 2:

 Map<String,Object> props=new HashMap<String, Object>();
 props.put("id", 100);
 props.put("name","firstNode");
     
 Map<String,Object> props2=new HashMap<String, Object>();
 props2.put("id",200);
 props2.put("name","secondNode");
    
  Map<String,Object> params=new HashMap<String, Object>();
  params.put("props1",props);
  params.put("props2",props2);
        
 engine.query("create (n1 {props1})-[:KNOWS]->(n2 {props2})", params);

This produces:

1 > POST http://localhost:7474/db/data/cypher
{"query":"create (n1 {props1})-[:KNOWS]->(n2 {props2})","params":{"props1":{"id":100,"name":"firstNode"},"props2":{"id":100,"name":"firstNode"}}}

Jul 24, 2013 10:38:47 PM com.sun.jersey.api.client.filter.LoggingFilter log
INFO: 1 * Client in-bound response
1 < 200
1 < Access-Control-Allow-Origin: *
1 < Transfer-Encoding: chunked
1 < Content-Encoding: UTF-8
1 < Content-Type: application/json; stream=true
1 < Server: Jetty(6.1.25)
1 < 
{"columns":[],"data":[]}

Batching all operations within a transaction

The documentation at https://github.com/neo4j/java-rest-binding states:

“In 1.8 it tries to collect all operations within a tx as a batch-operation which will then be executed on the server. This has the implication that the results retrieved within that "tx" are not immediately available but only after you called tx.success and tx.finish“

However, note that this is NOT the default behavior as you can see from Example 1. To turn this on, you need to set the following system property:

org.neo4j.rest.batch_transaction=true

Once the system property is set and Example 1 is re-run, the REST calls look like(requests only):

INFO: 1 * Client out-bound request 1 > POST http://localhost:7474/db/data/batch 1 > Accept: application/json; stream=true 1 > X-Stream: true 1 > Content-Type: application/json 1 > [{"id":1,"to":"node","body":{"id":200,"name":"secondNode"},"method":"POST"},{"id":2,"to":"node","body":{"id":200,"name":"secondNode"},"method":"POST"},{"id":3,"to":"{1}/relationships","body":{"to":"{2}","type":"KNOWS"},"method":"POST"}] INFO: 2 * Client out-bound request 2 > POST http://localhost:7474/db/data/cypher 2 > Accept: application/json; stream=true 2 > X-Stream: true 2 > Content-Type: application/json 2 > {"query":"start n=node(*) return count(n) as total","params":{}}

You may also explicitly create batch operations like so:

List<Node> response =graphDb.executeBatch(new BatchCallback<List<Node>>() { @Override public List<Node> recordBatch(RestAPI batchRestApi) { List<Node> nodes=new ArrayList<Node>(); Map props=new HashMap<String, Object>(); props.put("id",600); nodes.add(batchRestApi.createNode(props)); Map props2=new HashMap<String, Object>(); props2.put("id",500); nodes.add(batchRestApi.createNode(props2)); return nodes; } });

This translates into:

INFO: 1 * Client out-bound request 1 > POST http://localhost:7474/db/data/batch 1 > Accept: application/json; stream=true 1 > X-Stream: true 1 > Content-Type: application/json 1 > [{"id":1,"to":"node","body":{"id":600},"method":"POST"},{"id":2,"to":"node","body":{"id":500},"method":"POST"}]

Any of the Cypher/Batching approaches are highly recommended over the finer grained Neo4j Java API.

In the final post, we will take a look how transactions behave in the context of the REST binding.
-Luanne

Getting started with the Neo4j Java Rest Binding (Heroku deployment)

2013-07-20T14:43:00.001+05:30

Believe it or not, the last couple of days were my first attempt at using Neo4j’s Java Rest Binding. My earlier application on Heroku, Flavorwocky, used the Grails Neo4j plugin and for other stuff including work, embedded Neo4j rules.

However, embedded won’t do when deploying to Heroku, so the Java Rest Binding it is. Much better than dealing with raw REST, especially if you are very used to the embedded api’s.

Since it took me a short while to figure things out, mostly due to my shameful lack of Maven knowledge, thought I’d post a simple “getting started” guide.

At the end of this, your application and Neo4j should be talking to each other.

Note 1: This article deals with the rest binding for Neo4j 1.8.1, which is the version used by my Heroku application.

Note 2: Add the Neo4j add-on to your Heroku application if you do plan on trying this out

Step 1: Get/Include the library

If you are not deploying to Heroku and/or want to just include the jars, pick them up from here: http://m2.neo4j.org/content/groups/everything/org/neo4j/neo4j-rest-graphdb/1.8.1/

If you like/have to use maven, include the following dependency in your pom.xml

  <dependency>  
    <groupId>org.neo4j</groupId>  
    <artifactId>neo4j-rest-graphdb</artifactId>  
    <version>1.8.1</version>  
  </dependency>

The dependency refused to resolve until I added the following:

 <repositories>  
     <repository>  
       <id>mvn-neo4j</id>  
       <url>http://m2.neo4j.org/content/groups/everything</url>  
     </repository>  
 </repositories>

Step 2: Connect to your local Neo4j server

I created a simple servlet to test code and figure out the library, you might add this code to whatever you like.

 RestAPI graphDb = new RestAPIFacade("http://localhost:7474/db/data");

Note that you can also use

 GraphDatabaseService graphDb=new RestGraphDatabase(“http://localhost:7474/db/data”);

which is what I attempted to use first, and it does work, but doesn’t seem to be a valid parameter when creating a RestCypherQueryEngine (see step 3)

Step 3: Does it work?

Execute a Cypher query to return the number of nodes in the database.

 QueryEngine engine=new RestCypherQueryEngine(graphDb);  
 QueryResult<Map<String,Object>> result=  
     engine.query("start n=node(*) return count(n) as total", Collections.EMPTY_MAP);  
 Iterator<Map<String, Object>> iterator=result.iterator();  
 if(iterator.hasNext()) {  
   Map<String,Object> row= iterator.next();  
   out.print("Total nodes: " + row.get("total"));  
 }

Step 4: Let’s find out

Now is a good time to start your Neo4j server if you haven’t already.

If you’re using Tomcat and following the guide here https://devcenter.heroku.com/articles/create-a-java-web-application-using-embedded-tomcat then build and run:

 mvn package  
 sh target/bin/webapp

or launch your web application otherwise.

Access your servlet/jsp/anything else and you should see the count of nodes in your Neo4j database. A new database will return 1 node (the reference node)

Step 5: Deploy to Heroku

The URL to your Neo4j database on Heroku can be found in the NEO4J_URL environment variable so you don’t need to hardcode this in your application.

The variable value looks like this:

 http://username:password@db.hosted.neo4j.org:port

Pick up the variable in your application code using

 String neoUrl=System.getenv("NEO4J_URL");

and parse it to extract the username, password and db URL (http://db.hosted.neo4j.org:port)

Modify your code to now accept a username and password:

 GraphDatabaseService graphDb=new RestGraphDatabase(dbUrl + “/db/data”,username,password);

Push changes to heroku, access your application, and you should see the count of nodes in your remote instance.

Step 6: Add a node just to be really sure

 Transaction tx = graphDb.beginTx();  
 Map<String,Object> props=new HashMap<String, Object>();  
 props.put("id",100);  
 props.put("name","firstNode");  
 Node node=graphDb.createNode(props);  
 tx.success();  
 tx.finish();

Execute the same Cypher query to fetch the count of nodes and you should see it increment by 1.

If you can’t wait to see the REST calls that the binding is actually producing, you can turn on logging.

As per the documentation, you need to set the system property org.neo4j.rest.logging_filter to true.

I saw the REST calls being logged on my Tomcat console.

Further googling told me that I could set this property on Heroku by adding it as an extraJvmArgument in the appassembler-maven-plugin in pom.xml:

  <configuration>  
    <assembleDirectory>target</assembleDirectory>  
    <programs>  
      <program>  
        <mainClass>launch.Main</mainClass>  
        <name>webapp</name>  
      </program>  
    </programs>  
    <extraJvmArguments>-Dorg.neo4j.rest.logging_filter=true</extraJvmArguments>  
 </configuration>

That’s it! You should now be able to manipulate your graph using Cypher or the core API, but more on transactions and batching in Part 2.

-Luanne

Optimizing Neo4j Cypher queries

2013-01-02T23:15:00.002+05:30

Last week, I spent a good number of hours trying to optimize around 20 Cypher queries that were performing disastrously(36866ms to 155575ms) with data from a live system. After some trial and error, and a lot of input from Michael, I was able to figure out generally what needed to be done to the queries to make them perform better- at the end of this, the worst performing query dropped to 521ms on a cold graph and 1GB heap space (and that query has optional paths- not quite sure how to improve that), and the remaining were all under 50ms- quite a large improvement from original numbers.

Hoping that this might help someone else, here is what I did (mostly guess work and largely unscientific)- perhaps Michael can help explain the internals and correct me on some assumptions.

The first thing I did was make sure that every cypher query is using parameters- as stated in the Neo4j documentation, this helps in caching of execution plans.

Second, I came across a post in the Neo4j mailing list, where Michael mentioned NOT re-instantiating the ExecutionEngine so that the above said parameterized queries actually do cache. This might seem obvious to many, but it was a fact that easily slipped by, given I have a class called QueryExecutor that contains a method to execute a query with map of parameters, and the method created a new ExecutionEngine for every query. Once this method was written and used many, many times over, it was very easy to forget about. However, this was a very important factor in the overall performance (a mention in the docs would be extremely helpful), and it explained why my queries generally took the same time to execute even when parameterized. Changing this to use a cached ExecutionEngine saw the bottom half of my query time worksheet drop off...0 to 1 ms after being cached- excellent progress.

Now, to get to each query, starting with the worst. I decided to optimize on my local machine, with only 1GB heap space allocated, and on a cold graph. So I ignore the improvement in the query execution after caching- I feel that this is a better way to ensure progress- if the first query hit isn't improving, then you really haven't optimized it. That way, if it works really well with limited hardware, I am more confident of it working better in production.
Apart from timing the query in the code, I optimized using the webadmin console. The indicator that the query was awful is that it just would not return and the console would hang. Optimizing so that it didn't hang was a major improvement in itself. Highly unscientific, but I recommend it.

My first query averaged about 76558ms (the time was obtained by putting in a start and end time around the engine execute method). After the first optimization, it was down to 466ms.
Here is the original query: https://gist.github.com/4436272


And this is the optimized one: https://gist.github.com/4436281
There was no need to do this huge match only to filter out results based on the alertDate property on a, so I reduced the match to return the smallest set of data that could be filtered first ie. the path to a.
If you were to execute the bad query up to the first match, you would see something like 20K rows returned. After filtering, they are a mere 450 odd rows.
So as you can imagine, the second query quickly trims down the number of results you are potentially working with.

The second change was something I learnt from Michael on the day, when I asked whether it makes sense to do a giant match or keep pruning with subqueries. His response was "The question is, are you using match to actually expand the result set by describing patterns or do you just want to make certain that some relationships are there (or not) aka FILTERING. If the latter, you might want to use the path-expression syntax in a where clause.

WHERE a-[:FOO]->b
or WHERE NOT(a-[:FOO]->b)"

This took a little getting used to as the way I wrote the MATCH clause was exactly how I'd think of it in my head- but now I am able to differentiate between a match for results required versus match to filter. In the query above, I need (ir) in my results, so there is no need to include (a)-[:alert_for_inspection]->(i) in the match; I can just use it in the WHERE to make sure that a does indeed relate to i.

Here is another example: https://gist.github.com/4436293

   Straight away, you can see that we filter the cm relations by date- there is no need for us to first go and match all sorts of stuff if they don't fall in the date range. So this part of the query can be rewritten to:

   start c=node:companies(id={id})
match c <-[parent_company*0..]-(n)
with n
match n-[:defines_business_process]->(bp)-[:has_cycle]->(cm)
where cm.measureDate>={measureStartDate} and cm.measureDate<={measureEndDate}

After that, the next filter is on a- same principle applies:

with cm
    match (cm)-[:cycle_metric] ->m-[:metric_activity] ->ma-[:metric_unit] -> (u)-[:alert_for_unit]-(a) where a.alertDate=cm.measureDate and a.fromEntityType={type}

This further prunes our result set. Finally add the connections to r (for our results), making sure that paths that do not lead to r but are necessary go into the WHERE clause:

with a,ma
    match (r) < - [:for_inspection_result]-a-[:alert_for_inspection]- > i
    where (i) < -[:metric_inspection]-(ma)
    return a.id as alertId, r.value as resultValue

Here is the full query: https://gist.github.com/4436328
Original time- 33360ms, after optimization- 246ms.

At least for me, most of my queries fell into this pattern, so by day 2, I was able to refactor them very quickly.
The interesting thing is, after this, I still felt the response sluggish but the query times printed out in my log were tiny. After elimination, I found that my code actually stuck for a very long time after the query had executed (by executionEngine.execute) but within the first iteration over the resultset.
I assume that the results are not collected necessarily during the execute() method, but lazily fetched upon iteration of the resultset- I do not know Cypher internals so I might be completely wrong. But timing the iteration itself serves to point out even more badly written queries.

The other bits and pieces are- ORDER BY adds on a lot of time. If you can do without it, this is the first thing you should drop.
DISTINCT also adds time, but in many of my cases, it was difficult to drop.

If you want to check the absence of optional paths, where I typically do MATCH (u)-[r?:has_a]- (a) WHERE NOT (r is null), instead, re-write as MATCH (u)-[other_stuff]-.. WHERE NOT(u-[:has_a]-a) and it performs much better. However where I have optional paths like MATCH X-[o?:optional]-Y WHERE (o is present, match Y to A and B) OR (o is absent, match X to c and d), I was unable to simplify and these queries still take some time as compared to others without optional paths.

Finally- the problem was discovered so late because test data is never quite close to live data. The structure of the graph played a big role- some nodes were heavily connected, others, not so much- and the queries involving those heavily connected nodes were the ones that hurt us most. Try and use production quality data as far as possible to performance test or try and create test data as similar to it.

So, to summarize:
1. Always parameterize your queries
2. Cache the ExecutionEngine
3. Figure out what you're filtering on- try and apply that filter as early as possible with the least possible matches involved, so that your resultset gets progressively smaller as you move further into the query. Keep measuring both timing and results returned in each subquery so you can decide what goes first when the filter is not obvious
4. Examing your MATCH and RETURN clauses- include in the MATCH only those pieces that are required in RETURN. The remaining which would be to filter the results can go into the WHERE
5. If you don't need ORDER BY, drop it yesterday
6. If you don't need DISTINCT, get rid of it too
7. Checking the absence/presence of optional paths can be moved from the MATCH into WHERE if you don't need fancy filtering stuff based on it
8. Time not only the execute() on the query, but also the time to iterate through the results
9. If your webadmin console hangs, you have done a Bad Thing. Drop various parts of your query to figure out the offending one.
10. Try to use live data as far as possible
11. Test on a cold graph with miserly resources- you feel so much better when you see it zip past you on production!

-Luanne

Getting started with the Grails + Neo4j plugin + Heroku

2012-03-04T22:53:00.000+05:30

As promised in a previous post, here is a brief update with the Grails Neo4j 1.0.0.M2 plugin.

This largely follows the guide at http://devcenter.heroku.com/articles/grails

Create the grails app

>grails create-app heroku-neo4j
>cd heroku-neo4j

Check into Git
Create a .gitignore file for ignored files and check into git

*.iws
*Db.properties
*Db.script
.settings
.classpath
.project
eclipse
stacktrace.log
target
/plugins
/web-app/plugins
/web-app/WEB-INF/classes
web-app/WEB-INF/tld/c.tld
web-app/WEB-INF/tld/fmt.tld

>git init
>git add .
>git commit -m init

Create a Heroku app

>heroku create --stack cedar

...and deploy

>git push heroku master

Check it out in the browser

>heroku open

Add the Neo4J addon

>heroku addons:add neo4j:test

Install the Grails Neo4j plugin

>grails install-plugin neo4j

Update BuildConfig.groovy
Add the neo4j repo,
repositories {
...
mavenRepo 'http://m2.neo4j.org/releases'
}

and a compile dependency,
dependencies {
...
compile "org.neo4j:neo4j-rest-graphdb:1.6"
}

Update DataSource.groovy
Configure your database location through DataSource. Since the Heroku Neo4j addon is only accessible via REST, we use type="rest". On Heroku, the system environment variable NEO4J_REST_URL will be set to your own Neo4j instance

development {
grails {
neo4j {
type = "rest"
location = "http://localhost:7474/db/data/"
}
}
}

production {
grails {
neo4j {
type = "rest"
location = System.getenv('NEO4J_REST_URL') ?: "http://localhost:7474/db/data/"
}
}
}

Now you should be able to use GORM to maintain your Neo4J datastore.
What's great with this setup is, you can switch to an embedded or HA data setup by modifying the type and location parameters in DataSource, without changing any other code.

One more thing...
Relationship properties are not yet supported. See https://groups.google.com/d/topic/neo4j/6IiFN_RG19I/discussion

Similarity based recommendations with Cypher

2012-02-25T22:17:00.000+05:30

Josh Adell recently published a blog post on the similarity based recommendation engine he is building for rating and recommending beer- always a welcome service! His post shares his experience with Gremlin, a graph traversal language. I'm going to take his example to show you how it can be done using Cypher, Neo4j's query language.

So basically, we want to be able to recommend beers. But we just don't want the highly rated beers- it means much more to us if they were rated highly by people who have similar tastes to us.

The goal is to answer two questions:

1. For a beer that I have not rated, what is the average rating given to it by people similar to me?

2. Which beers should I try and then rate? We want to recommend beers that have been rated 7 or higher by people similar to me.

To get to both these questions, we first need to determine people with similar tastes. For our purposes, Josh defines a similar user as one whos ratings on an average are within 2 points of my ratings for the same items.

We'll use this simple graph to test our Cypher queries:

(user1) -[:RATED 3]->(itemA)

(user2) -[:RATED 2]->(itemA)

(user3) -[:RATED 7]->(itemA)

(user1) -[:RATED 8]->(itemB)

(user2) -[:RATED 7]->(itemB)

(user3) -[:RATED 4]->(itemB)

(user2) -[:RATED 5]->(itemC)

(user3) -[:RATED 9]->(itemC)

(user2) -[:RATED 8]->(itemD)

(user3) -[:RATED 4]->(itemD)

Considering user1, the user with similar tastes is user2. Let's take a look at the Cypher query:

start me = node(user1) //Look up user1 via an index or some other means

match (me)-[myRating:RATED]->(i)<-[otherRating:RATED]-(u)

where abs(myRating.rating-otherRating.rating)<=2

return u

The match clause finds all items that I rated which have also been rated by other users, represented by u.

The where clause filters out ratings that differ by more than 2 points. The result of this query would be user2.

Let's try to answer question 1: What is the average rating(by similar people) of a beer not rated by me?

start item=node(x), //Look up item via an index or other means

similarUsers=node(u) //similarUsers here is the result received in the first query above

match (similarUsers)-[r:RATED]->(item)

return AVG(r.rating)

And finally, question 2: Which beers(that I haven't tried) have been rated 7 or higher by users similar to me?

start me=node(user1), //Look up item via an index or other means

similarUsers=node(3) /similarUsers here is the result received in the first query above

match (similarUsers)-[r:RATED]->(item)

where r.rating > 7 and not((me)-[:RATED]->(item))

return item

There you go! Of course there's much more to recommending beers- read more about it in Josh' post.

-Luanne

Flavor of the month- Neo4j and Heroku, part 2

2012-02-25T10:19:00.002+05:30

Continuing from the previous post, here are the features we used for the first version of Flavorwocky:

Set up:
When the application starts up, we check if the database is empty. To do this, we perform a traversal from the reference node looking for any categories connected to it. If there are none, then we go ahead and create the set of categories by first creating their nodes, and then a relationship to the reference node. We also create the index "ingredients", which is used to index ingredient names. Source code: https://github.com/luanne/flavorwocky/blob/master/grails-app/conf/BootStrap.groovy

Add pairing:
Adding a pairing involves

Checking whether either node already exists to avoid recreating it
Creating both nodes and linking them to their categories in a single transaction
Creating a relationship between them in the same transaction above

To accomplish this, we used the Batch operation (note, this is still experimental).
Source: https://github.com/luanne/flavorwocky/blob/master/grails-app/controllers/com/herokuapp/flavorwocky/FlavorwockyController.groovy (fetchOrCreateNodes())

Auto complete: This was just an index lookup matching the partially entered ingredient by name. Source: https://github.com/luanne/flavorwocky/blob/master/grails-app/controllers/com/herokuapp/flavorwocky/FlavorwockyController.groovy (autosearch())

Visualization:
We used d3.js to provide two visualizations for the search results. The "Explore" visualization is based on the Node-Link tree; we used a Cypher query to find all ingredients that pair with the searched ingredient up to 3 levels deep, transform it into the appropriate data structure, and render it as JSON. Note, although the visualization interactive, the fetching of data is not. The entire set of data for 3 levels is grabbed at once- a future enhancement would be fetching children only when you expand a node.
Source: https://github.com/luanne/flavorwocky/blob/master/grails-app/controllers/com/herokuapp/flavorwocky/FlavorwockyController.groovy (getSearchVisualizationAsTreeJson())

Although the tree is pretty slick, one shortcoming is that it is a tree- so if children are linked to each other, then you see multiple instances of that ingredient in the tree. Hence we tried another visualization to capture the interconnections between ingredients and also surface interesting facts such as flavor trios- there's a pretty high chance that if you see a triangle in the network visualization, those three ingredients can be combined together well.
We used the Force directed graph for this. Again, a Cypher query came to the rescue (this time 5 levels deep to produce a richer model).
Source: https://github.com/luanne/flavorwocky/blob/master/grails-app/controllers/com/herokuapp/flavorwocky/FlavorwockyController.groovy (getSearchVisualizationAsNetworkJson())

In both examples, the affinity of the pairing is used to calculate the length of the connector between ingredients, indicating shorter connections have ingredients that pair much better than those with longer connections.

That's about it! There's so much more than can be done with this application, but it's going to have to wait for a bit.
Again, please about Flavorwocky if you like it or just want me to win. Voting also helps!

-Luanne

Flavor of the month- Neo4j add-on for Heroku

2012-02-25T10:13:00.001+05:30

Neo4j launched a challenge earlier this year called "Seed the Cloud" to get folks to create templates or demo applications on Heroku using the Neo4j add-on. After much internal debate, I decided to enter, only to be thrown into despair for lack of an idea. The idea came to me while I was doing nothing in particular- to build a simple app that would help one find ingredients whose flavors complement one another.

Basically, you have these ingredients that pair really well together- knowing which ingredients have flavor affinities can produce some amazing new dishes.

The app allows you to add pairings with an 'affinity'- how well they pair together- and search for an ingredient to find others that pair with it.

The app is built using Grails 2.0 for the front end, visualizations are the result of the very neat d3.js library, and interaction with Neo4j is done using the Neo4j REST Api.
The entire thing is deployed on Heroku (with the Neo4j add-on), while the source is available on github
To get started with Grails 2.0, Neo4j add-on and Heroku, read Aldrins post on the topic: http://thought-bytes.blogspot.in/2012/02/grails-20-heroku-and-neo4j-addon.html

How to deploy Flavorwocky locally as well as on Heroku is documented in the Readme.

The model lends itself very nicely to a graph- as you can see, it is very simple:

Every category is color coded- for convenience, it is stored as a property on the category node, but of course, it doesn't have to be.

Flavorwocky was also picked as the basis for this challenge because it is a real world use case for a graph, and I wanted this entry to really focus on Neo4j primarily rather than supporting bells and whistles. The features of the Neo4j REST api used in this app will be the topic of the next post.

Gentle reminder: Please about Flavorwocky if you like it or just want me to win. Voting also helps!

-Luanne

Grails 2.0, Heroku and Neo4j addon

2012-02-25T10:07:00.000+05:30

This post will help you get started with Grails 2.0 on Heroku with the Neo4j addon.
This largely follows the guide at http://devcenter.heroku.com/articles/grails
Note: This does not use the Grails Neo4j plugin (http://grails.org/plugin/neo4j), since I couldn't get it working with Grails 2.0 (Will post an update as and when I can)
(Update: here's the one with the neo4j plugin http://thought-bytes.blogspot.in/2012/03/getting-started-with-grails-neo4j.html)

Create the grails app

>grails create-app heroku-neo4j
>cd heroku-neo4j

Delete the DataSource.groovy

>del grails-app\conf\DataSource.groovy

Check into Git
Create a .gitignore file for ignored files and check into git

*.iws
*Db.properties
*Db.script
.settings
.classpath
.project
eclipse
stacktrace.log
target
/plugins
/web-app/plugins
/web-app/WEB-INF/classes
web-app/WEB-INF/tld/c.tld
web-app/WEB-INF/tld/fmt.tld

>git init
>git add .
>git commit -m init

Create a Heroku app

>heroku create --stack cedar

...and deploy

>git push heroku master

Check it out in the browser

>heroku open

Add the Neo4J addon

>heroku addons:add neo4j:test

Install the Grails Rest plugin

>grails install-plugin rest

Now you can access the Neo4J datastore on Heroku via its REST interface. For example, to create new node from a controller,

import static groovyx.net.http.ContentType.JSON
withRest(id: "neo4j", uri: "<REST-URL>") {
auth.basic '<login> ', '<password> '
def response = post(contentType:JSON, requestContentType:JSON, body: ['name': 'Artichoke'] )
}
render response.status
render response.data

Replace <REST-URL> , <login> and <password> with your Neo4J instance's settings.

-Aldrin

Getting Maven to just run - tips, workarounds

2012-02-07T15:43:00.002+05:30

Recently, I was trying to run an inherited project, but couldn't get maven to compile. Some local libs were not being picked up by maven. I had to add it manually to the local maven repo. So, when you have a library your project is dependent on and you cannot find it on a public maven repository, then you can add it to your local repository as below:

mvn install:install-file -Dfile=C:\my-awesome-api-1.0.jar -DgroupId=com.awesome -DartifactId=awesome-api -Dversion=1.0 -Dpackaging=jar

Then add a <dependency> to your pom.xml,

<dependency>
< groupId>com.awesome < /groupId>
< artifactId>awesome-api < /artifactId>
< version>1.0 < /version>
< /dependency>

Once the compilation went through successfully, the test started failing. If you don't particularly care about the tests, then you can skip it at any time by adding -Dmaven.test.skip=true or -DskipTests=true to the command line,

ex. > mvn:package -DskipTests=true

Graph visualization with Neo4J and Arbor.js

2011-02-03T13:24:00.011+05:30

As you might have gathered from my previous post, I have been tinkering with Neo4J lately and I am beginning to appreciate the power and elegance of graphs.
I also came across Arbor.js and was bowled over by it. Loved the force directed layouts and the fact that if you are patient, you will watch your nodes traveling far and wide to settle in their final destination. No need to drag, drop or re-organize the graph- what emerges is a visual pattern, obvious to the eye. I can think of tons of usages for such a visualization, but thought I'd start with something simple just to get the feel of it.

I decided to model a simple organization chart or graph. So off I went to create that model in Neo4J, fed data into it, and my graph was ready very quickly.
On to Arbor.js- I must admit, I was rather stumped, even with the reference doc they publish. I did not know where to start and where to end- of course, my lack of knowledge around jQuery syntax contributed a lot.

So, I did the next best thing which was to get the sample project working locally. Tweak main.js to play with the layout and understand what's really happening. Scroll right down to the $(document).ready function to see the 'ParticleSystem' getting initialized. The sample project initializes with three parameters: repulsion, stiffness and friction. There are more available like frames per second and precision, but I have not tried them yet. Play with those values to see how they affect the graph. I settled on: var sys = arbor.ParticleSystem(500, 600, 0.5)
I also left gravity=true since I definitely want the settling graph.

You can then add edges and nodes to the ParticleSystem object you just initialized.
Adding a node is as simple as sys.addNode(name,data).
The name you supply functions as an identifier of the node and the data is an object you can use to associate any set of parameters with this node. These are especially useful when you need additional information about the node to render it- for example, set its color.
Adding an edge is just as simple: sys.addEdge(source,target,data). Source and target nodes are identified by their node names, and data again is an object of parameters to associate with the edge.

Now the sample project displays little black boxes for nodes. I was looking for something more like circular nodes with their labels in them- much like the arbor.js home page.

Just adding nodes and edges does not do anything- the ParticleSystem needs a renderer to well, render the graph. The arbor.js reference will tell you that you need to create an object with two methods- init and redraw, and set your ParticleSystem renderer to this object.
If you look at main.js, this is exactly what it does (viewport is the canvas object in which to render the graph)-

sys.renderer = Renderer("#viewport")

The init method in this case simply sets up screen dimensions.
The redraw method is the one responsible for painting nodes and edges. You will see those in the
particleSystem.eachEdge and particleSystem.eachNode loops.

So then this is the place to tweak the actual representation of nodes and edges.
Borrowing from the source of the arbor.js home page, I changed the code to:
particleSystem.eachNode(function(node, pt){
        
var w = Math.max(20, 20+gfx.textWidth(node.name) )
if (node.data.alpha===0) return
  if (node.data.shape=='dot'){
      gfx.oval(pt.x-w/2, pt.y-w/2, w, w, {fill:node.data.color, alpha:node.data.alpha})
      gfx.text(node.name, pt.x, pt.y+7, {color:"white", align:"center", font:"Arial", size:12})
      gfx.text(node.name, pt.x, pt.y+7, {color:"white", align:"center", font:"Arial", size:12})
  }else{
      gfx.rect(pt.x-w/2, pt.y-8, w, 20, 4, {fill:node.data.color, alpha:node.data.alpha})
      gfx.text(node.name, pt.x, pt.y+9, {color:"white", align:"center", font:"Arial", size:12})
      gfx.text(node.name, pt.x, pt.y+9, {color:"white", align:"center", font:"Arial", size:12})
  }

})
You will note that the color of the node is pulled from the data object attached to the node.
The font, size, width etc. can all be parametrized by setting them in the data object when you create the node.
This example uses the graphics library, so be sure to include graphics.js in your html.


For the edges, I changed the color and also parametrized the thickness of the connector line by the width parameter.
particleSystem.eachEdge(function(edge, pt1, pt2){
if (edge.source.data.alpha * edge.target.data.alpha == 0) return
      gfx.line(pt1, pt2, {stroke:"#151B8D", width:(edge.data.weight/2), alpha:edge.target.data.alpha})

       })


That is sufficient for a basic graph representation. Now to get the data out of Neo4J and into the graph.
The ParticleSystem has a graft method which accepts an object in the form {nodes:{}, edges:{}}.
These nodes and edges are added to the particle system.
I navigated through my Neo4J graph and spat out the data in this json like format.

Then, in the JS, you simply do a
var data = $.getJSON("orgChart.json",function(data){
  sys.graft({nodes:data.nodes, edges:data.edges})
})
and you're done.
Just make sure you enclose each attribute in quotes- for example, color and shape in the code below.
And it also freaks if you supply a .8 instead of a 0.8 for the length attribute in this example.


The final graph rendered in my browser looks like this(click to enlarge)-

This really just scratches the surface of what you can do with arbor.js
Add to it the various types of graphs or models you can extract out of Neo4J, and you could build some great stuff!


orgChart.json contains the following-
{"nodes":{
   "Aidan":{"color":"green", "shape":"dot", "alpha":1},
   "Sofia":{"color":"green", "shape":"dot", "alpha":1},
   "Liam":{"color":"GoldenRod", "shape":"dot", "alpha":1},
   "Charlotte":{"color":"green", "shape":"dot", "alpha":1},
   "Noah":{"color":"GoldenRod", "shape":"dot", "alpha":1},
   "Ava":{"color":"green", "shape":"dot", "alpha":1},
   "Jackson":{"color":"lightblue", "shape":"rectangle", "alpha":1},
   "Addison":{"color":"green", "shape":"dot", "alpha":1},
   "Ethan":{"color":"green", "shape":"dot", "alpha":1},
   "Olivia":{"color":"green", "shape":"dot", "alpha":1},
   "Kayden":{"color":"lightblue", "shape":"rectangle", "alpha":1},
   "Amelia":{"color":"green", "shape":"dot", "alpha":1},
   "Mason":{"color":"lightblue", "shape":"rectangle", "alpha":1},
   "Lily":{"color":"green", "shape":"dot", "alpha":1},
   "Logan":{"color":"green", "shape":"dot", "alpha":1},
   "Isabella":{"color":"green", "shape":"dot", "alpha":1},
   "Jacob":{"color":"lightblue", "shape":"rectangle", "alpha":1},
   "Emma":{"color":"green", "shape":"dot", "alpha":1},
   "Benjamin":{"color":"lightblue", "shape":"rectangle", "alpha":1},
   "Abigail":{"color":"lightblue", "shape":"rectangle", "alpha":1},
   "Caleb":{"color":"lightblue", "shape":"rectangle", "alpha":1},
   "Scarlet":{"color":"lightblue", "shape":"rectangle", "alpha":1},
   "Owen":{"color":"lightblue", "shape":"rectangle", "alpha":1},
   "Audrey":{"color":"GoldenRod", "shape":"dot", "alpha":1},
   "Gavin":{"color":"lightblue", "shape":"rectangle", "alpha":1},
   "Grace":{"color":"lightblue", "shape":"rectangle", "alpha":1},
   "Connor":{"color":"green", "shape":"dot", "alpha":1},
   "Claire":{"color":"GoldenRod", "shape":"dot", "alpha":1},
   "Riley":{"color":"lightblue", "shape":"rectangle", "alpha":1},
   "Emily":{"color":"lightblue", "shape":"rectangle", "alpha":1},
   "Nolan":{"color":"lightblue", "shape":"rectangle", "alpha":1},
   "Quinn":{"color":"red", "shape":"dot", "alpha":1}
},
"edges":{
   "Quinn":{"Liam":{"length":2.5,"weight":2},
           "Audrey":{"length":2.5,"weight":2},
           "Noah":{"length":2.5,"weight":2},
           "Claire":{"length":2.5,"weight":2}},
   "Liam":{"Sofia":{"length":2.5,"weight":2},
           "Ethan":{"length":2.5,"weight":2},
           "Amelia":{"length":2.5,"weight":2}},
   "Audrey":{"Aidan":{"length":2.5,"weight":2},
               "Ava":{"length":2.5,"weight":2},
           "Jackson":{"length":2.5,"weight":2}},
   "Claire":{"Charlotte":{"length":2.5,"weight":2},
           "Olivia":{"length":2.5,"weight":2}},
   "Noah":{"Addison":{"length":2.5,"weight":2},
           "Kayden":{"length":2.5,"weight":2}},
   "Olivia":{"Mason":{"length":2.5,"weight":2},
           "Lily":{"length":2.5,"weight":2}},
   "Ethan":{"Logan":{"length":2.5,"weight":2},
           "Isabella":{"length":2.5,"weight":2}},
   "Addison":{"Jacob":{"length":2.5,"weight":2},
           "Emma":{"length":2.5,"weight":2},
           "Benjamin":{"length":2.5,"weight":2}},
   "Emma":{"Abigail":{"length":2.5,"weight":2},
           "Caleb":{"length":2.5,"weight":2}},
    "Logan":{"Scarlet":{"length":2.5,"weight":2},
           "Owen":{"length":2.5,"weight":2},
           "Gavin":{"length":2.5,"weight":2}},
    "Lily":{"Connor":{"length":2.5,"weight":2},
           "Grace":{"length":2.5,"weight":2},
           "Nolan":{"length":2.5,"weight":2}},
    "Connor":{"Riley":{"length":2.5,"weight":2},
           "Emily":{"length":2.5,"weight":2}},
   }
}

RuntimeExceptions and Neo4J

2010-12-13T16:52:00.006+05:30

I'm currently working with Neo4J(http://www.neo4j.org) - in the evaluation stage really, trying out various use cases. TestNG is my test framework of choice, so most of my work is driven by it.
What really irritated me is the fact that some tests failed. When I looked at the TestNG report, I found that in some cases my test data was bad, so in effect, I was creating a self relationship (which Neo4J frowns upon). There were also cases where, during trial and error, I did not clean the db so I ended up doing a node.getSingleRelationship and instead of one relationship, there were more.

So why was I irritated? My test cases did fail due to an exception. However, they were Runtime exceptions. When I wrote the code, I did not think they may be thrown- I did not look at the JavaDoc in detail- just used IntelliJ to show me what I could possibly do with Node, and winged it from there. So, when my test failed, I was a bit disbelieving. First checked the exception- and it was a Runtime exception (duh). Then pulled up the JavaDoc and saw that Neo4J does indeed indicate an exception can be thrown in these cases.
This caused me to tweet about Neo4J and their runtime exception philosophy. (Pleasantly surprised to see @emileifrem respond to it)

Anyway, somewhere in one of many open windows, I happened to search for any such issues related to Neo4J. I came across Stephen Colebourne's blog instead (http://www.jroller.com/scolebourne/entry/checked_exceptions_bijava)

So really, did it matter whether Neo4J threw RuntimeExceptions or not? Apart from the test results, which were surprising because I neglected to read the JavaDoc in detail, I went through some exceptions that Neo4J throws (leaving out the very obvious Runtime exceptions).
Now if a DeadlockDetectedException, HaCommunicationException, NotFoundException(thrown if a request is made to a node, relationship or property that does not exist), NotInTransactionException, ReadOnlyIndexException or TransactionFailureException were thrown, what would I really do about it? Nothing much, unfortunately. There is no way I could be smart and recover from these. At the most I'd log it and re-throw it.

Now on to the ones I was disturbed with- methods mostly on the Node.

getSingleRelationship() which (as the name indicated) returns the only relationship of a given type and direction. Well, the fact that I'm using this method does mean that I really expect there to be just one relationship, so when a runtime exception is thrown, there is no way to recover from it.

delete() throws a RuntimeException if it has relationships attached to it. Can I do anything about it? Maybe- depending on functionality, I might attempt to do a cascade delete and re-try the operation. Not sure how many use cases would really want a cascade delete though.

createRelationshipTo() throws a RuntimeException if you try to create a self relationship. Note here, that the JavaDoc does not specify this in the Throws, but rather in the description. This arguably could be dealt with by creating an intermediate node- but at the same time, this situation can easily be avoided by defensively coding for it.

So did it really matter that Neo4J threw Runtime exceptions? After a lot of thought, not really. Why some exceptions are true classes of RuntimeException and why some (the Node exceptions) are just RuntimeExceptions in themselves, I am not sure. The source does not indicate that they do anything over and above being a RuntimeException.
Does it really matter that there is a split convention? I'd think if I needed to recover from a delete exception, then it does matter.
It will be interesting to see as I develop more with Neo4J, how much these Runtime exceptions come into play and whether they are really as disturbing as I initially thought.

Disclaimer: I still do not believe that we should do away with checked exceptions altogether :-)

If you aint going to use it, don't write it

2010-04-16T08:47:00.003+05:30

It's interesting to see how engineers "follow processes" that they don't agree with. They might not agree with it because maybe they haven't understood the value it provides to them (after all, they are typically not dumb guys), or they are not provided with adequate time to follow them, or they are just plain stupid (the processes- not the engineers).

I've written about testing before. Once I realized its value, I don't go anywhere without my tests (so to speak). Now you can preach about it all you want but unless engineers try it, do it properly, and realize the value they lend to our complicated lives, they're not going to write them. Now take those engineers and tell them that they must write tests. First, everyone ignores the new law. Then they're reminded and now it's an exit criteria with one more requirement- coverage numbers.
So, what happens? They produce tests. They run them and get them to pass and management is happy. But what are the value of those tests? Are they maintained? Are they used to help verify that they didn't break anything when the next feature comes along? There are even interesting cases where tests haven't been run as part of builds for months on end and nobody noticed. Which says to me "delete those tests". If nobody cares about those poor things, it is more merciful to end their lives.

If you're not going to get value out of code you've written, then it is better not to have written it in the first place. I'm sure you can think of better ways to use your time (even if it's taking a nap).

-Luanne

Marker Interfaces- what are they marking?

2010-04-01T22:42:00.003+05:30

This might be a controversial statement, but I dislike marker interfaces. Yes, I get that the Serializable marker interface, for example, has a purpose. Let me amend that statement to 'I dislike marker interfaces with no clear intent'.

An interface specifies a contract. If you implement me, then you must provide these services.
An empty interface cannot specify a contract because, well, it doesn't have one to offer. It is more of a tagging mechanism. It says, if you implement me, then you're tagged in this manner. But, in most of the cases I've seen, nothing is done with this information. OK, so the class is tagged- but what the heck does that mean? Does it mean the class behaves in a certain manner? Does it mean the class can offer these services? Who knows? More often than not, marker interfaces are used as a handy mechanism to refer to classes that implement it, in a uniform manner, without the hassle of inheritance. That's about it. Which is why you'll find code riddled with if-then-elses that do an instanceOf. Kind of defeats the purpose.

What is worse about these badly designed marker interfaces, is that it's one large, no-invite-needed party; anyone and their pet elephant can tag along. Not that having the elephant would break up the party. But it isn't appropriate for the elephant to be there- the party wasn't really meant for him and passers-by would have a hard time figuring out what exactly was going on.

Let's take this very example. Your event planning application has people that are invited to parties. Just as you finish designing your Person class, along comes someone and says, we also invite entertainers to the party. The entire group is to be a single guest. Of course, they're paid and they're not expected to eat the snacks, but they're on the invite list. All your nice API's that returned collections of Person to represent the entire guest list is now left wondering what to do.

So what is usually done? Simple, lets introduce this marker interface:
public interface Invitee {} and have Person and EntertainmentGroup implement it. Then all we do is change our API to return Invitees and we're done.
What does it mean to be an invitee? Nothing.
Now you need to find out how many snacks to order assuming that each person will eat 5. The code I see more often than not is, get a collection of Invitees from the guest list, iterate through them, check if the Invitee is an instanceOf Person- and increment the snack counter by 5.
What did it mean to be an invitee? Nothing.
Tomorrow when the zoo keeper is invited to the party and insists on bringing his pet elephant, what do you do? Simple. Have Elephant implement Invitee. Then when you calculate the amount of snacks to order, you add yet another check to see if the Invitee is an instanceOf Elephant and you add 50 kilos of bananas to the list.
Ideally the Invitee class would specify what it means to be an invitee. It's behavior is predictable. When the event manager gets rich and decides that only movie stars can attend his parties, then you can no longer have Invitee extend MovieStar, unless you only know elephants that are movie stars.

Unless you are doing something very concrete with the marker interface, unless you are processing that tag information in some manner, I consider it a Bad Thing.
Tagging can be accomplished using annotations. You can selectively tag classes/methods and actually process them to do or mean something useful.

What do you think? Do you see marker interfaces being used as convenience mechanisms?

-Luanne

DeadlineExceededException on GAE with Grails

2010-02-10T23:45:00.003+05:30

Grails on the Google App engine takes substantial time to startup and will probably exceed the hard deadline limit set by GAE. If you see a lot of "DeadlineExceededException" errors in your logs, you should be aware of this. There is a jira for this - http://jira.codehaus.org/browse/GRAILSPLUGINS-1736 . Vote this up if it matters to you.

Notes on Google AppEngine 1.3, Grails 1.2.1, and app engine plugin 0.8.8.

2010-02-03T17:57:00.007+05:30

Currently in the process of upgrading a grails project to grails 1.2.1. Noted down some observations (and will keep updating as i test stuff)

1.
There doesn't seem to be a need to add the following to the Config log4j anymore (this was needed in grails 1.x see http://jira.codehaus.org/browse/GRAILSPLUGINS-1184)

'null' name:'stacktrace'

2.
Had some trouble with the sessions and kept getting this error

[java] WARNING: Nested in org.springframework.web.util.NestedServletException: Request processing failed; nested exception is java.lang.RuntimeException: Session support is not enabled in appengine-web.xml. To enable sessions, put

true

in that file. Without it, getSession() is allowed, but manipulation of sessionattributes is not.:

As advised in the plugin documentation, added the following to Config, but it still didn't work!

google.appengine.sessionEnabled = true

Had to eventually update the /web-app/WEB-INF/appengine-web.xml file and uncomment the line below


3.
Can now user grails run-app to run the development server instead of grails app-engine

The Refactoring Quandary

2007-09-06T14:13:00.000+05:30

We all know by now that refactoring is important. The problem is- when do you refactor?
In my opinion, the time to refactor is when the thought first crosses your mind.

Let's say you're implementing a new feature or fixing a bug. You think 'hmm, I seem to be duplicating code' or 'looks like we could reorganize a bit of this'. Instead of pushing the thought 'this could be refactored later' to the back of your head where it will stay there for a very long time and come back to haunt you once your code is so messy that even you don't know how it got that way, the answer is to refactor immediately. Do what you can to improve your code. If you don't, then you and everybody else just pass the buck on and say "X will do that someday", and soon you are in programming hell. Moreover, if you refactor regularly, you most often need to just refactor small areas at a time. It's much better then saying we'll refactor this entire application- more often than not, throwing it out and restarting would be more appealing.

Of course there are reasons why people are frightened of refactoring. Acknowledging the need to refactor does not mean saying to the world "I have bad code". You do not have to have glaring bugs to think of refactoring. Think of it as general housekeeping. Just because the hotel housekeeping staff cleans your room every morning does not mean you left it looking like a pig sty the night before.
Another important reason is the fear of breaking code. Which is why to refactor confidently, you need to be backed by a solid suite of test cases. If you don't, then this is a very real risk you run.

Refactoring is easier said than done. You rarely find that you've been told to spend the next week refactoring. There are always more pressing tangible deliverables. However, it is extremely important to realize that the intangible ones will actually affect your tangible deliverables in the near future. Just as we need to consciously budget time to write automated tests, refactoring does not happen magically either. A small bit of effort regularly will go a long way to contributing to better manageable, extensible and flexible code.

-Luanne

At the local Udipi...

2007-08-24T15:10:00.000+05:30

So I'm sitting in this udipi fast food restaurant waiting for my lunch (rawa idli - highly recommended), and I find myself observing the waiters going about their jobs. This is what typically happens:

1. Customer enters, sits, decides what he wants and tells the waiter.

2. Customer asks the waiter how long it will takes and the waiter gives an approximate waiting time.

3. Waiter heads over to the kitchen and bellows the order to the cooks.

4. If a customer gets impatient, the waiter goes to the kitchen to check and assures the customer 'it's almost done!'.

5. Once the order is ready, the waiter takes it to the customer's table and serves it to him.

6. If the customer didn't like the meal, the waiter gets a earful. He smiles and goes back and abuses the cook.

7. Waiter handles many customers at the same time.

8. Finally the waiter hands over the bill.

With a start I realise - it's no different from my current job. I'm sort of a Project Lead. This is what I do:

1. The client sends me the requirements.

2. Client asks me how long it would take to deliver it. I estimate and return a figure.

3. I hand the requirements over to my team, who then start working on it.

4. Client gets impatient and eager to see 'anything', so I tell them 'it's 90% done!'

5. When it's actually done then it's released to the client

6. If it's a poor quality release, then I get a earful from the client and I pass it on to my team.

7. Many such requirements are being handled simultaneously.

8. End of the month, I total up the time sheets for the billing.

!!!

I ate quickly, tipped the waiter handsomely and left.

Someday, I want to go back into the kitchen and be the cook...

Cheers,
Aldrin

How much can we test?

2007-06-11T09:22:00.000+05:30

A couple of years back, I was introduced to unit testing. At the time, I found it rather tedious. Though it did expose bugs in our code early in the cycle, maintenance of test cases were becoming a nightmare. So the question is, what should you test, and how much should you test it? I've formed some opinions along the way- tell me what your experience has been...

First, every automated test is NOT a unit test. A unit test is simply a test that validates the smallest "unit" in your application. It could be a class, and you would be validating each operation of that class by testing its methods, for example. A test which tries to validate a business operation that spans more than one class would probably be a functional test.
A simple example would be a Shopping Cart class. A unit test could verify that deleting items from an empty cart is not possible. A functional test would be one that validates the shopping process- add items to cart, checkout, verify bill amount.

Which kinds of tests should you write? Only unit? Functional? Both?

Well, unit tests are no doubt good. You should ideally write them before you write your actual code. Ideally. In the real world of the services industry at least, there are very many projects whose scope and functionality change so fast that programmers are always playing catch up with unit tests. It is simply not possible to maintain a vast number of test cases. So writing unit tests become an afterthought. And they are the very first thing axed from the budget :-)
If we were to write comprehensive unit tests that really add value, teams would require time to do so. They would also require time to maintain them. That time is rarely granted.

So, what is the next option? What about writing functional tests? Agreed, they will not provide 100% coverage, but let us assume that we could test major business flows/processes. The amount of test cases decrease, but the ROI is visible earlier on because major flaws can be uncovered before moving into system testing. There will always be bugs. Subtle bugs that even unit tests will fail to pick up. So is it worth spending time trying to think of many, many ways to break the code when the code will certainly break somewhere in a place you did not expect it to?
I personally would write functional tests, and enhance the suite over time. For example, when a bug is reported, a test could be written for it. It could very well be a unit test. But with bugs and change requests, the suite will grow in sensible proportions.

What do you think?

Preventing multiple user sessions with the same user account

2007-05-22T08:30:00.000+05:30

Often you want to prevent the same user account being used simultaneously in different sessions (business license restrictions being one reason, complex technical synchronisation problems being another)। There are two typical strategies to prevent थिस :-
#1. Prevent any new sessions while a session with the same user account exists.
#2. Allow a new session, but disconnect any previous sessions if they exists.

Imho, I think #1 can be very problematic for the user. If the user faces a system crash during his session, his subsequent attempts to login after recovery will be thwarted. Until his previous session times out, he will be left twiddling his thumbs. Using cookies to maintain the session information would be a good idea here. If you are supporting auto-login, then you are already doing something like this. A session id stored in the cookie, would be enough clue to server to reconnect to the same session. You need to ensure however, that the user does a proper 'logout' - even going to the extent of hooking your 'logout' into the body unload event if the user chooses to shut the browser window, without clicking on your 'logout' link.
Now what happens when a user logs into your site, shuts his browser, deletes his cookies and tries to login again, before his session times out ?? Let him suffer, i say ! :)

#2 is much easier to implement. All you need to do is maintain a Map of user accounts with their sessions in application scope. Add a listener for session timeouts, to remove these entries from the map when the session times out. When a new session if being created, simply look up the map for previous sessions by the user and invalidate them if present and also remove them from the map.

I'd like to hear your comments on how you achieved similar requirements।

Cheers
Aldrin

You minus IDE=helpless?

2007-04-16T11:07:00.000+05:30

Nowadays, the IDE is your workbench. You can generate your UML, debug, run embedded servers, inspect your code, read the Dilbert strip...all from within a single application.

With each release boasting nifty features, there is very little reason to leave the comfort of your IDE.

Or is there?

In the good old days, we had no such pleasures. I remember writing code in Notepad, and was thrilled when I discovered EditPlus. Debugging meant compiling on the command line, fixing error by error till you had a successful compile, starting tomcat and testing your web app. Today, you can do all this and more at the click of a button.
This is certainly great news for the developer...and a very bad habit as well.
No doubt IDE's save you a lot of trouble and time. IntelliJ IDEA has increased my productivity tremendously and helps me with a whole lot of things- including reading the Dilbert strip ;-)
But, if I had no IntelliJ for a day, then I would still be able to code in EditPlus and look up the JavaDoc from wherever if need be, deploy my application and debug it. And that is the difference.

Too often I see IDE whizzes who can do almost anything with CTRL, SHIFT or ALT and one or two keys. But take the IDE away...and they are helpless. This is Bad Thing in my opinion. The fallout of not knowing the basics is terrible. And this doesn't just apply to software.

I have seen cases where for some reason, the embedded web server plugin fails for whatever reason. Everyone is examining their code and can find nothing wrong...and they're re-examining it for hours and hours. I say shut it down and deploy manually...but few know how to :-)
In fact, very few know how to run javac !!! Turns out that the code base is fine...it was the embedded web server....hours and hours of valuable time wasted because we refused to abandon our beloved IDE.

Consider this scenario- you've developed this fantastic application, deployed it using your IDE, debugged it and it runs like a dream. Then you go to install it at the clients site. And horror of horrors, they don't have your IDE. And won't let you install it. Now what???
You should be able to say so what? You'd roll up your sleeves, check your code out of CVS or wherever, set your build properties, run ANT, deploy, start your webserver and go home in a couple of hours to have a nice drink.
But if you know no such thing...then you'll be pleading with IT to install your IDE and when they refuse you'll have to call someone in your team and have them guide you step by step. And you'll be drowning your sorrows in many drinks after many, many tense hours. Which is a better option??

I am not condemning IDE's. On the contrary. I have used a couple (and IntelliJ by far is THE best...though beware, after using it, like me, you will hate the likes of Eclipse). They are huge help. But use them after you know how to get up and running without their help. If you are developing a simple webapp, can you code a servlet for example, without the IDE? Can you compile it? Can you deploy it? Do you know how to run Tomcat? Where to find the log files?
If yes, then go ahead...open up your IDE and be happy :-)

-Luanne

Hello World with standalone Java Persistence, Hibernate and HSQLDB

2007-04-03T17:53:00.000+05:30

Lately just started reading (& learning) EJB3. Because of a slow PC at home, I needed to be able to run code for the Java Persistence API in the standalone mode, without the overhead of a huge database or app server. So, I rustled up this guide to get up and running with Java Persistence. Basically all you need is a database and an implementation of the persistence API. We will use HSQL and Hibernate Entity Manager respectively. Hibernate Entity Manager runs on the Hibernate core, so will need that too.

Step 1: You will need Java 5. Get it from http://java.sun.com

Step 2: Download HQSLDB from http://hsqldb.org. Extract into some folder (eg. D:\hsqldb)

Step 3: Download the Hibernate Entity Manager from http://hibernate.org. Extract into some folder. (eg. D:\hibernate-entitymanager-3.2.1.GA)

Step 4: Download the Hibernate Core from http://hibernate.org. Extract into some folder. (eg. D:\hibernate-3.2)

Step 5: Set up the database

Create a new folder for all database related files we will create in this step. Lets call it 'db_stuff'.

Create a server.properties file. We will name our database 'think'. Configuration parameters should be as below:

server.database.0 file:./think
server.dbname.0 think

Start the database in the server mode by running the following from the command line (Better still put this in a batch file)

java -cp D:\hsqldb\lib\hsqldb.jar org.hsqldb.Server

You can use the HSQLDB DatabaseManager to view this database. Run the following from the command line to invoke the tool.

java -cp D:\hsqldb\lib\hsqldb.jar org.hsqldb.util.DatabaseManager

In the "Connect" dialog, select the following options:
Type: HSQL Database Engine Server
Driver: org.hsqldb.jdbcDriver
URL: jdbc:hsqldb:hsql://localhost/think
User: sa
Password:

Step 6: Now the standalone. Add the following jars to your project classpath:

hsqldb.jar from D:\hsqldb\lib\hsqldb.jar
hibernate3.jar from D:\hibernate-3.2
all jars from D:\hibernate-3.2\lib\
hibernate-entitymanager.jar from D:\hibernate-entitymanager-3.2.1.GA
all jars from D:\hibernate-entitymanager-3.2.1.GA
Whew!

Step 7: Create the persistence configuration

Create a file named persistence.xml, the persistence configuration file. This file needs to be within a META-INF folder in your classpath. If you are compiling all your classes into a folder, say 'classes', create a META-INF folder under 'classes', then create the persistence.xml file in META-INF. The file should look like below:

<persistence xmlns="http://java.sun.com/xml/ns/persistence"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence" version="1.0">

<persistence-unit name="think" transaction-type="RESOURCE_LOCAL">
<properties>
<property name="hibernate.dialect" value="org.hibernate.dialect.HSQLDialect"/>
<property name="hibernate.hbm2ddl.auto" value="update"/>
<property name="hibernate.connection.driver_class" value="org.hsqldb.jdbcDriver"/>
<property name="hibernate.connection.username" value="sa"/>
<property name="hibernate.connection.password" value=""/>
<property name="hibernate.connection.url" value="jdbc:hsqldb:hsql://localhost/think"/>
</properties>
</persistence-unit>
</persistence>

Note that the transaction-type is RESOURCE_LOCAL. This is required for standalone environments. Also note the "hibernate.connection.url" matches the URL you use to connect from the HSQL DB Manager to the database, jdbc:hsqldb:hsql://localhost/think

Step 8: Create an entity class, BroadcastMessage.java

package com.blogspot.thought_bytes;

import javax.persistence.Entity;
import javax.persistence.Id;

@Entity
public class BroadcastMessage {

@Id
private long id;

private String message;

public long getId() {
return id;
}

public void setId(long id) {
this.id = id;
}

public String getMessage() {
return message;
}

public void setMessage(String message) {
this.message = message;
}
}

Step 9: Create a harness to interact with the persistence layer, ClientHarness.java

package client;

import com.blogspot.thought_bytes.BroadcastMessage;

import javax.persistence.EntityManager;
import javax.persistence.EntityManagerFactory;
import javax.persistence.Persistence;
import javax.persistence.EntityTransaction;

public class ClientHarness {

public static void main(String[] args) {
//Use the persistence.xml
EntityManagerFactory emf = Persistence.createEntityManagerFactory("think");
EntityManager em = emf.createEntityManager();

//
EntityTransaction tx = em.getTransaction();
tx.begin();

//Create and initialize an entity instance
BroadcastMessage mb = new BroadcastMessage();
mb.setId(1);
mb.setMessage("Hello World");

//Persist using the entity manager
em.persist(mb);

//Save
tx.commit();

em.close();
emf.close();

}
}

Run the above.

Go back to the database manager and refresh the view. You should be able to see a new tabel "broadcastmessage". If you query this table you should get the following:

ID MESSAGE
-- -------
1 Hello World

That's it! And now the real hard work begins... :)

Cheers,
Aldrin