Stephen Mallette

Using Gremlin Java in TinkerPop 2.x

2015-04-02T00:00:00+00:00

Some organizations are single-minded in their choice of programming language. They are “Java shops” or “Python shops” and there is no room for anything in between. This choice often places developers in a position where they find new technology that would be useful in achieving their goals, but they are unable to use it because it uses a non-approved technology. Gremlin, as it stands in TinkerPop 2.x, is best used in its Groovy form, as the syntatic sugar of Groovy helps form a simple and concise DSL for traversing graphs. Unfortunately, Groovy is not always on the list of approved technology at these organizations that define themselves as a “Java Shop”. As such, developers in these organizations look to use Gremlin in its Java form.

Unfortunately, Gremlin Java is quite verbose, which is largely a function of the way Java itself works. Developers tend to have a hard time understanding the method in which their Gremlin Groovy converts to Java and, as a result, there tend to be many questions about the best way to go about doing this.

I recently provided an answer to this question on StackOverflow where someone was trying to understand just how to do this. I went into some detail on this topic and hopefully provided a nice tutorial for those who have this problem. I think that in many ways, this answer will become the reference that we point people to who have this problem.

Article - Powers of Ten Series

2014-06-02T00:00:00+00:00

One of the most common questions that people seem to have when getting started with graph databases is how to best load their data. As is the case with most technological things, a one size fits all answer isn’t really applicable. While there are some general patterns that can be followed, there are definitely different strategies to apply depending on the size of the data.

To identify the common patterns and address the different strategies at different sizes issue, I worked with Daniel Kuppitz to author the Powers of Ten series, a two part article on bulk loading data into the Titan graph database. Part I of the series focused on datasets that contained millions to tens of millions of edges and Part II focused on hundreds of millions of edges to billions.

I think that these posts are very useful resources for those getting started with Titan.

Deconstruct Gremlin Paths

2014-04-09T00:00:00+00:00

I mentioned to someone recently that to be great at Gremlin, you have to be great at Groovy. There is too much syntactic sugar and shorthand in Groovy to just ignore. In my previous post, I demonstrated the use of collate in batching Gremlin results. In this post, I present another bit of Groovy goodness in relation to Gremlin and it lies in Groovy’s ability to destructure lists into variables:

gremlin> (x,y,z) = [10,20,30]
==>10
==>20
==>30
gremlin> y
==>20
gremlin> x + y + z
==>60

This ability to do multiple assignment ends up presenting opportunity for some nice Gremlin usage with respect to the use of the path step:

gremlin> g = TinkerGraphFactory.createTinkerGraph()
==>tinkergraph[vertices:6 edges:6]
gremlin> g.v(1).outE.inV.path.collect{it[0].name + ":" + it[1].weight + ":" + it[2].name} 
==>marko:0.5:vadas
==>marko:1.0:josh
==>marko:0.4:lop

In the above code, we traverse out from g.v(1) and then grab the path. The path is a List of the elements that made up the traversal. In this case, the item in the first postion is g.v(1), the item in the second position is the out edge and item in the third position is the in-vertex. As the path is a List, we must reference the items in it by index. It works, but is a bit difficult to read. The alternative is to destructure the list into variables:

gremlin> g.v(1).outE.inV.path.collect{marko,outE,inV->"${marko.name}:${outE.weight}:${inV.name}"}        
==>marko:0.5:vadas
==>marko:1.0:josh
==>marko:0.4:lop

With named variables for the items in the path, the intent of the collect closure can be more clearly understood at a glance. While “multiple assignment” has a nice use case with the Gremlin path step, it is quite useful for other tasks where accessing objects in a list by position can get overly-complicated.

Use Collate to Batch Gremlin Results

2014-02-28T00:00:00+00:00

Groovy has many great functions and syntax that can expand the capabilities and flexibility of the Gremlin programming language. A recent polyglot query I came across, highlighted how important it can be to know Groovy as well as you know Gremlin.

Consider a situation where you want to write some Gremlin to pull some data from the TinkerPop toy graph to then pass to a query sent to a MySQL database. It might look something like this:

gremlin> g  = TinkerGraphFactory.createTinkerGraph()
==>tinkergraph[vertices:6 edges:6]
gremlin> g.V.name.gather.transform{
             sql.rows("SELECT * FROM toy WHERE n IN ('" + it.join("','") + "')")}.next()
==> ...

So the code above gets the name property from each vertex, uses gather to pull them all into a List and then transforms that List by using it as parameters to an IN clause of a SQL statement. That works nicely with the toy graph because it is small (only six vertices). In a larger graph, it’s possible that the set of returned vertices might exceed the capability of the IN expression or otherwise cause the query to not perform well. In this case, it might be better to batch the results to the IN clause so that they could be executed in smaller bits.

Groovy provides a very succinct way to do this in the form of the collate function. Consider the following Gremlin session, which still uses the toy graph for demonstration purposes:

gremlin> g.V.name.gather                                             
==>[lop, vadas, marko, peter, ripple, josh]

The output above shows that a List of names is being passed to the transform function shown in the first block of code. To batch that list, replace gather with collate as follows:

gremlin> g.V.name.toList().collate(2)
==>[lop, vadas]
==>[marko, peter]
==>[ripple, josh]

Using collate, the names are placed in batches of two, where each batch will be iterated into the transform effectively creating a batched approach to issuing the SQL query. In this case, three separate queries would be executed, each representing one small chunk of the total data to be retrieved. The following code shows the modified code using collate:

gremlin> g.V.name.toList().collate(2).transform{
             sql.rows("SELECT * FROM toy WHERE n IN ('" + it.join("','") + "')")}.scatter.toList()
==> ...

Note that the end of the statement changed as well, where the use of scatter unrolls the rows returned from the database into a List. This part effectively reassembles the result data to a single List, just as would have been done in the single SQL request in the original traversal.

Groovy has many useful functions. Some are better known than others, but every once in a while a lesser known one might be just what is needed to get the job done.

Use has() Over filter()

2013-11-20T00:00:00+00:00

There’s been some discussion on the Aurelius mailing list regarding the advantages of using a single `filter` with a complex closure over usage of multiple `has` steps. You can read more about the discussion here. The core of the discussion is about realizing that Gremlin will compile `has` steps down to query objects that can be used by the underlying graph store to optimize the traversal, whereas a `filter` is merely a function that evaluates an item in the pipeline. Therefore, using `has` is generally better than using `filter`.

In midst of the discussion, I decided to try a small experiment. I wanted to show that two `has` steps were no slower than a single `filter` operation that performed the same function, even when no query optimizations were performed by the underlying store. Titan does perform these optimizations, so I decided to utilize a TinkerGraph instead. TinkerGraph utilizes DefaultQuery to process `has` operations. `DefaultQuery` is used by all Blueprints implementations that do not implement their own optimizations, so the results of this experiment should carry across all Blueprints implementations that don’t optimize.

To do the experiment, I opened a Gremlin REPL and constructed a graph with a half-million vertices with some random property values.

gremlin> g = new TinkerGraph()                                                                        
==>tinkergraph[vertices:0 edges:0]
gremlin> r = new Random()                                                                             
==>java.util.Random@4841a34
gremlin> (0..<500000).each{g.addVertex([i:it,r:r.nextLong().toString().padLeft(21,"0")])};null      
==>null
gremlin> g.v(0).map
==>{r=001842268616820437679, i=0}

I then wrote two queries. The first used `filter` with two property evaluations to determine whether they met the restriction or not and the second used two `has` operations one for each property being evaluated. I ensured that both queries returned the same number of vertices.

gremlin> g.V.filter{it.r>="00000000000" && it.i>=200000}.count()                               
==>150211
gremlin> g.V.has("r",T.gte,"00000000000").has("i",T.gte,200000).count()                        
==>150211

Once assured that I was returning the same data for both traversals, I timed the results.

gremlin> s=System.currentTimeMillis();g.V.has("r",T.gte,"00000000000").has("i",T.gte,200000)
         .iterate();System.currentTimeMillis()-s
==>95
gremlin> s=System.currentTimeMillis();g.V.filter{it.r>="00000000000" && it.i>=200000}
         .iterate();System.currentTimeMillis()-s       
==>5521

Of course, I was quickly reminded that I wasn’t using `getProperty` in my `filter` which balanced the numbers considerably:

gremlin> s=System.currentTimeMillis();g.V.has("r",T.gte,"00000000000").has("i",T.gte,200000)
         .iterate();System.currentTimeMillis()-s
==>95
gremlin> s=System.currentTimeMillis();g.V.filter{
            it.getProperty("r")>="00000000000" && it.getProperty("i")>=200000
          }.iterate();System.currentTimeMillis()-s       
==>150

The use of reflection to get the property value is very expensive and something I don’t use in production code (I shouldn’t have missed that).

The results show a traversal with two `has` steps performs somewhat better than a single `filter` step. What’s interesting about this finding, is the fact that TinkerGraph is not optimizing the query when processing the `has`. The results basically mean that finding a way to use `has` instead of `filter` will generally result in better traversal performance even for Blueprints implementations that don’t support query optimizations.

Lein Uberjar with Titan Dependency

2013-11-14T00:00:00+00:00

If you’re working with Titan in Clojure there’s a good chance you’re using Lein and if you are there’s also a chance that you are building a “jar with dependencies” via lein uberjar. I was doing just that yesterday, when I was suddenly tossed into a point of deep frustration. I watched happily as my code ran perfectly in my local environment with lein run, but after doing a lein uberjar and pushing my fat jar to a server instance in EC2 I got this:

Caused by: java.lang.IllegalArgumentException: A SPI class of type org.apache.lucene.codecs.
  PostingsFormat with name 'Lucene41' does not exist. You need to add the corresponding JAR 
  file supporting this SPI to your classpath.The current classpath supports the 
  following names: [Pulsing41, SimpleText, Memory, BloomFilter, Direct]
	at org.apache.lucene.util.NamedSPILoader.lookup(NamedSPILoader.java:109)
	at org.apache.lucene.codecs.PostingsFormat.forName(PostingsFormat.java:100)
	at org.elasticsearch.index.codec.postingsformat.PostingFormats.<clinit>(
	  PostingFormats.java:76)

I ended up unlocking the solution to the problem by noticing an issue in the ElasticSearch GitHub Repo named “Conflicting META-INF between elasticsearch & lucene-core/lucene-codecs for packaging into standalone binary”. At that point, I realized that Lein was overwriting these files over and over again in the uberjar process:

META-INF/services/org.apache.lucene.codecs.Codec
META-INF/services/org.apache.lucene.codecs.DocValuesFormat
META-INF/services/org.apache.lucene.codecs.PostingsFormat

For my “jar with dependencies” to be functional, I would need those files to merge with one another as opposed to simply performing an overwrite. In Maven, you would likely use the Shade Plugin configured to do a resource trasformation. Up until yesterday, I wasn’t aware of how to do that with Lein, but after some research realized that a recently added feature to Lein (as of 2.3.3) provided a similar feature set. I simply added the following snippet to my project.clj:

:uberjar-merge-with {#"org\.apache\.lucene\.codecs\.*" [slurp str spit]}

which merged those various files together instead of writing one over the other. At that point, my “jar with dependencies” was good to go. You can read more about uberjar-merge-with here.

Article - Developing a Domain Specific Language in Gremlin

2013-08-26T00:00:00+00:00

The gremlin-users mailing list always has questions asking how to productionalize Gremlin. In a recent blog post for Aurelius, entitled Developing a Domain Specific Language in Gremlin, I describe some common patterns for working Gremlin into a production environment. The patterns all relate to the concept of writing a Domain Specific Language (DSL) on top of Gremlin to allow the language of the domain to traverse the graph. The DSL approach to Gremlin yields many benefits and should make development and maintenance of Gremlin code much more straightforward.

New GremlinDocs.com Logo

2013-07-05T00:00:00+00:00

GremlinDocs is a learn-by-example tutorial for the Gremlin graph traversal language. Ketrina Yim who produces the wonderful and fantastical characters of TinkerPop has brought some new life to the GremlinDocs web site. She has illustrated a new logo that brings a unique look to the site and brings some character to something that would otherwise just be a bunch of words and code.

I see a very simple story in that picture:

When not traversing graphs, Gremlin takes some of his free moments to write his autobiography. Quill and scroll are his writing implements of choice, producing characters of sweeping grace that whimsically flow together to tell stories of graph exploration.

These stories are GremlinDocs.

Shortest Path With Gremlin

2013-06-07T00:00:00+00:00

There has been a fair amount of discussion on finding shortests paths using Gremlin both in the Gremlin Users mailing list and on StackOverflow. The questions asked sent me into a fair bit of depth on the subject, allowing for a better understanding of how best to implement such traversals in Gremlin.

The result of this effort are as follows:

An answer to Gremlin: How to get more than 1 shortest path with the same number of edges? on StackOverflow
A Gremlin-based answer to Is it possible to get the shortest path with traversal cost by using Cypher or Gremlin on StackOverflow.
An update to the GremlinDocs Shortest Path Recipe that provides much greater details on the topic (received some help from Daniel Kuppitz in this regard).

Article - Polyglot Persistence and Query with Gremlin

2013-02-06T00:00:00+00:00

On a recent trip, where I hung around with Marko Rodriguez, we came to a realization that Gremlin has a wider applicability than often thought of. Gremlin can be used in a variety of ways to work with disparate data sources that center around graphs. From those thoughts, this blog post, Polyglot Persistence and Query with Gremlin, was written.

GremlinDocs - Gremlin by Example

2012-09-23T00:00:00+00:00

I’ve been working with clojure for some time now, mostly using it for small, fast development efforts so that I could become more familiar with it. In that time, I came to really like the format of ClojureDocs. It provides code snippets as samples for each function which clearly demonstrate how it works.

I started thinking that Gremlin would benefit from that kind of documentation. With that in mind, I created GremlinDocs.com. GremlinDocs is meant to be a companion to the Gremlin wiki, a reference for both beginners and advanced Gremlin users alike. It is a quick reference to the functions that Gremlin has with examples taken from the Gremlin REPL to demonstrate those functions.

GremlinDocs is actually a GitHub repository where all the content for it is contained within a single README.md file, with the idea that the repository could be forked and community contributions would help further improve the documentation.

.NET Client for Kafka Checkpoint

2012-07-28T00:00:00+00:00

I’d written several posts in the past for some work I did for Kafka, which is a distributed publish/subscribe messaging system. Here’s a conclusive list of those posts from some months back:

After getting a message, which asked about my specific work with the C# client and mentioned some dead links to its location, I realized that I should probably close out this part of the story. I no longer maintain the C# client for Kafka which is available here. Maintenance was picked up by someone external to the Kafka project, who made a number of enhancements to my original work and was then contributed back to Kafka.

Custom Graph Configuration In Rexster

2012-05-03T00:00:00+00:00

Rexster is a graph server that is capable of exposing any Blueprints enabled graph through different interfaces, such as REST. Rexster is “packaged” with the common Blueprints implementations that are maintained by TinkerPop. At this time, given Rexster 0.8, those implementations include:

TinkerGraph in-memory graph
Neo4j graph database
OrientDB graph database
DEX graph database
Sesame 2.0 compliant RDF stores

Outside of these implementations, there are others that are not maintained by TinkerPop, such as the one from InfiniteGraph. There is also the recently submitted pull request that implements Blueprints over Datomic, which, as of today, has not yet been merged. In addition, one might find themselves in a scenario where they have some sort of “private” Blueprints implementation over some other data store. Even though Rexster doesn’t have explicit, native understanding of these externalized implementations, it is capable of serving them if configured properly to do so.

Rexster is configured via its rexster.xml file. A typical configuration of a TinkerGraph looks like this:

<graphs>
  <graph>
    <graph-name>mygraph</graph-name>
    <graph-type>tinkergraph</graph-type>
  </graph>
</graphs>

This configuration could also be written as:

<graphs>
  <graph>
    <graph-name>mygraph</graph-name>
    <graph-type>com.tinkerpop.rexster.config.TinkerGraphGraphConfiguration</graph-type>
  </graph>
</graphs>

Note that the <graph-type> is now pointing to a specific com.tinkerpop.rexster.config.GraphConfiguration implementation. The GraphConfiguration interface looks like this:

public interface GraphConfiguration {
    Graph configureGraphInstance(Configuration properties) throws GraphConfigurationException;
}

Classes that implement this interface are responsible for constructing a Blueprints graph for Rexster, thereby allowing Rexster to extend itself to serve any Blueprints graph. To implement the configureGraphInstance method, simply construct a new graph using the properties passed to it. The properties are just the root level key/values within the <graph> element. For example, TinkerGraph configuration in Rexster allows for a parameter, as in:

<graph>
  <graph-name>gratefulgraph</graph-name>
  <graph-type>com.tinkerpop.rexster.config.TinkerGraphGraphConfiguration</graph-type>
  <graph-location>data/graph-example-2</graph-location>
</graph>

The following code extracts that value from the properties, returning null if it does not exist in the XML configuration:

String graphFile = properties.getString("graph-location", null);

Key/value pairs are great, but properties can also handle more complex parameters. A good example of that is shown in the Neo4jGraphConfiguration where an arbitrary set of sub-properties are passed to the GraphConfiguration implementation, which in turn is passed to the Neo4jGraph instance.

To expose the custom configuration to Rexster, the process is the same as a Rexster Extensions. Simple copy the compiled JAR containing the custom configuration in the REXSTER_HOME/ext directory. It will then be possible to reference the class within rexster.xml. While the built-in configurations for Rexster have short-hand configuration options, such as tinkergraph or neo4jgraph for the <graph-type>, it is not possible to alias a custom implementation.

Blog Post - Multitenant Graph Applications

2012-04-09T00:00:00+00:00

I co-authored this blog post on multitenancy in systems backed by graph databases. Multitenancy is actually easily achieved using TinkerPop Blueprints (as of version 1.2) and its PartitionGraph graph wrapper. The running example we used follows the healthcare industry, specifically a multitenant EHR.

IIS Configuration Behind Elastic Load Balancer With SSL

2012-01-01T00:00:00+00:00

I wanted to set up an Amazon Elastic Load Balancer(ELB) with SSL in front of a set of IIS servers, where the IIS servers simply served over port 80. I further wanted it to work such that requests to port 80 forwarded to the SSL port on 443. Basically, the ELB port forwarding looks like:

ELB forwards requests on port 443 to port 80 of IIS.
ELB forwards requests on port 80 to port 80 of IIS.

I found that this could be peformed pretty easily with the URL Rewrite Module with the following configuration:

<system.webServer>
    ...
    <rewrite xdt:Transform="Insert">
        <rules>
            <rule name="HTTPs Redirect" stopProcessing="true">
                <match url="(.*)" />
                <conditions>
                    <add input="{ALL_HTTP}" matchType="Pattern" 
                         pattern="HTTP_X_FORWARDED_PROTO:https" ignoreCase="true" negate="true" />
                </conditions>
                <action type="Redirect" url="https://{HTTP_HOST}/{R:1}" />
            </rule>
        </rules>
    </rewrite>
</system.webServer>

The key is knowing that the ELB passes the X_FORWARDED_PROTO request header. By checking if the request did not originate as https, the client can be redirected to their requested URI with SSL.

But the Tests Ran...

2011-09-24T00:00:00+00:00

I write this as a bit of a penance and a reminder after a difficult day as a software engineer. A great set of automated tests are critical to building great software. For a test suite to be considered “great” it should likely extend beyond just some quantitative metric like the total number of tests or code coverage percentages. Greatness is additionally defined by the quality of the test set as determined by those who know the code and the features.

Greatness can breed confidence. Confidence to refactor. Confidence to release and deploy.

With all this confidence around, it’s easy to make a different kind of mistake that all that greatness can’t protect you from: a poor or misunderstood design choice. We sometimes choose the less than optimal path purposely and stow it away as accepted technical debt. Other times, the sub-optimal choice occurs without intention and it’s a problem than lies in wait for someone to uncover it. In either case, these kinds of choices can produce scenarios that poke holes in the “great” test suite and the confidence you have because of the “great” test suite won’t allow you to see the side effects coming. The end results are bugs in the next release and your wondering how the test suite passed without you uncovering the problem before the release date.

The general warning here is to not become so reliant on your tests that you use them as the ultimate truth as to the quality of what you are doing.

Jersey - "charset" in Content-Type

2011-05-29T00:00:00+00:00

In working with Jersey for Rexster, the issue arose some time ago that the character set was not being established in the Content-Type of the response. Therefore, the Content-Type was being set this way:

Content-Type:application/json

as opposed to

Content-Type:application/json;charset=ISO-8859-1

While the omission of the charset did not prevent JSON from being served from Rexster, it did create some problems with certain clients that expected that value to be set. Moreover, the default charset of ISO-8859-1 created problems for some users who had unicode in their data and needed the charset established to UTF-8.

One way to solve the problem within Jersey is to append the charset programmatically into the header:

return Response.ok(resultList).header("Content-Type", "application/json;charset=UTF-8").build()

Aside from the fact that the inline coding of the Content-Type left no option for multiple output formats (as described in this Jersey Issue and the subsequentally created JAX-RS Specification ), it also isn’t a terribly elegant solution. It means that you have to ensure that every service method established has that bit of code appended to it when the response object is returned. I thought that it would be nice to centralize that functionality a bit within Rexster, attempting several solutions, until I came up with one that worked.

Rexster implements a Jersey ContainerResponseFilter. The simplified code for this filter is shown in the following:

import com.sun.jersey.spi.container.ContainerRequest;
import com.sun.jersey.spi.container.ContainerResponse;
import com.sun.jersey.spi.container.ContainerResponseFilter;

import javax.ws.rs.core.MediaType;

public class CharsetResponseFilter implements ContainerResponseFilter {

    public ContainerResponse filter(ContainerRequest request, ContainerResponse response) {

        MediaType contentType = response.getMediaType();
        response.getHttpHeaders().putSingle("Content-Type", contentType.toString() + ";charset=UTF-8");

        return response;
    }
}

This filter is then plugged into Jersey through rexster.xml. Of course, other applications using Jersey would simply add it through the configuration of the Jersey servlet or programmatically as in:

ServletAdapter jerseyAdapter = new ServletAdapter();
jerseyAdapter.addInitParameter("com.sun.jersey.spi.container.ContainerResponseFilters", "com.my.package.MyResponseFilter");

Obviously the CharsetResponseFilter shown above is overly simplified. Rexster does some basic checks to see if the charset is already established prior to overriding the settings. This allows individual service methods to be ultimately in charge of setting that value. This is useful for Rexster Extensions where returned media types can be disparate.

Avoid Preemptive Branching

2011-05-01T00:00:00+00:00

I really like this blog post on the dangers of preemptive branching. It basically points out that the act of branching for a release candidate should be performed as late as possible (if at all) and that the hardening process should utilize the entire team. This approach helps to avoid a series of problems which will hamper the project and the team with complexity and sub-optimal conditions. In the end, rather than getting value from splitting team effort, a preemptive branch actually diminishes the ability of the team to deliver. Some of the ugly results will be decreases in code quality, over-specialization and knowledge centralization of individual team members, general failure to meet the goals that the branch was supposed to help solve in the first place, increased technical debt, and overall reduction in agility.

The author lists a number of reasons why preemptive branching is problematic. I find this one to be quite damaging:

Increases the work-in-progress and creates a lot of planning, management, version-control, testing, etc. overheads.

The daily challenge of playing traffic cop to a team of committers under pressure to be “done” can be quite time consuming and frustrating. Consider some of the annoyances:

Every individual on the team needs to understand where they need to do their work. You might have a hardening team working in the release branch, but there is a good chance that these team members are also the “most trusted and knowledgeable” members of your team. As such, there will be cases where they will be needed to split their time between the release branch and mainline. While what needs to be communicated is typically simple (eg. Bug Fix A goes in mainline and Bug Fix B goes in the release branch), doing it in an effective way requires process and takes time.
Even if the team has overcome the hardship above, there is always the inevitable realization that a change that went to mainline was something that the release branch depended on for hardening. Those changes then need to be recreated manually in the release branch, which inevitably create merge conflicts when going back to mainline.
The longer a release branch lives, the greater the chance that the merge path among branches can become complex or in the worst case impossible.

In addition to the reasoning of the author against a preemptive branch, I also see the following risks:

All the merging and branching strategy that I describe above can easily become overly centralized such that only one person knows what is going on in source control.
The goal of just hardening a release is sometimes lost the longer the release branch exists. As the release remains in hardening mode past its expected delivery date, the demand for additional features promised can pressure the team into sneaking some of those features into the release branch.
The release branch may fail to ever truly harden. This failure could be a result of the previously described feature-sneak, but may also be the result of too much technical debt.
The sum negative impact of a preemptive branch chips away at the team’s agility.

I can’t seem to think of any easy ways out of a release branch started too early, especially once the project is weakened by several cycles of this type of branching. Preventing future preemptive release branches is likely the best thing to do, though that may only be possible after accepting delays in the release schedule.

.NET Client for Kafka Multi-Request

2011-03-28T00:00:00+00:00

I made some enhancements to the .NET client for Kafka that were recently accepted into the project mainline. Aside from some basic namespacing reorganization and some improved error handling, I added two major bits: Multi-produce and Multi-fetch batch operations.

The Multi-produce and Multi-fetch operations provide for a way to group up multiple ProducerRequest or FetchRequest objects and send them to Kafka in batch. The Multi-produce works like this:

List<ProducerRequest> producerRequests = new List<ProducerRequest>
{  
    new ProducerRequest("topic-a", 0, new List<Message> { new Message(Encoding.UTF8.GetBytes("1: " + DateTime.UtcNow)) }),
    new ProducerRequest("topic-b", 0, new List<Message> { new Message(Encoding.UTF8.GetBytes("2: " + DateTime.UtcNow)) }),
    new ProducerRequest("topic-z", 0, new List<Message> { new Message(Encoding.UTF8.GetBytes("3: " + DateTime.UtcNow)) }),
    new ProducerRequest("zopic-z", 1, new List<Message> { new Message(Encoding.UTF8.GetBytes("4: " + DateTime.UtcNow)) })
};

MultiProducerRequest producerRequest = new MultiProducerRequest(producerRequests);
Producer producer = new Producer("localhost", 9092);
producer.Send(producerRequest);

Note that the list of producerRequests pushes messages to three different topics and within one of those topic (topic-z), pushes to two different partitions. While the example above pushes a different message to each topic, it would not be hard to imagine a scenario where one might utilize this batching approach to send the same message to multiple topics and/or partitions.

The Multi-fetch works in a similar fashion:

Consumer consumer = new Consumer("localhost", 9092);
MultiFetchRequest request = new MultiFetchRequest(new List<FetchRequest>
{
    new FetchRequest("topic-a", 0, 0),
    new FetchRequest("topic-b", 0, 0),
    new FetchRequest("tobic-z", 0, 0),
    new FetchRequest("tobic-z", 1, 0)
});

List<List<Message>> messages = consumer.Consume(request);

The important bit to note in this example is that the Consume method returns a List of a List of Message objects. I’ll refer to that “inner” list as the “message set”. In the example above, there would be four message sets (one for each request), where the index of the request sent to Kafka in the batch, matches to the position in the List of message sets.

.NET Client for Kafka Checkpoint

2011-03-11T00:00:00+00:00

I finished enough of the .NET client for Kafka that I wrote about in my last post to feel comfortable in sending out a pull request to the Kafka team. I just received notification moments ago that it was accepted into the project and merged into mainline.

The client covers basic Producer and Consumer operations. The Producer can send both synchronous and asynchronous messages. The Consumer can retrieve messages and offsets from Kafka, but at the moment, unlike other client consumers that were developed, it does not yet have a mechanism for polling. I hesitantly left that bit out as I was concerned with “completeness” of the client API, but my primary need was related to sending messages to Kafka through .NET environments and not necessarily consuming them, so it was lower priority for me. Perhaps that will be a later addition to the client. You can read more about the client API in the README.md for the client.

While the demand for a .NET client is arguably low, given the Kafka target domain and environment, there might be a few progressive .NET minds who could find some value in what I’ve contibuted.

.NET Client for Kafka

2011-03-01T00:00:00+00:00

It’s a bit early stage, but I forked the Kafka project on GitHub today to add a .NET client. Kafka is a distributed publish/subscribe messaging system and looks very promising. It is aimed at capturing activity stream data in the vein of Facebook’s Scribe and Cloudera’s Flume. I’ve never look to deeply into Scribe, but was very interested in Flume to the very close point of forking it to try to add some features. When I came across Kafka I made an immediate connection to its architecture (probably due to the simple publish/subscribe model) and thought of an area where I might make use of it.

The area where I might make use of it, involves a bit of .NET so I needed a .NET client that would communicate with Kafka for me. Since I couldn’t find anyone who was doing that, I figured I’d strike a few lines of code and get it done.

I executed the first commit to my fork which contained basic support for a .NET message producer. The basic approach to sending off one or more messages looks something like this:

string payload1 = "kafka 1.";
byte[] payloadData1 = Encoding.UTF8.GetBytes(payload1);
Message msg1 = new Message(payloadData1);

string payload2 = "kafka 2.";
byte[] payloadData2 = Encoding.UTF8.GetBytes(payload2);
Message msg2 = new Message(payloadData2);

Producer producer = new Producer("192.168.50.201", 9092);
producer.Send("test", 0, new List<Message> { msg1, msg2 });

I need to get a basic consumer working next and then think about some more advanced features for the library.

Beyond Arbitrary Versioning

2011-02-10T00:00:00+00:00

While reading more about Zombie.js and looking at javascript unit testing options, I came across Sinon.js. I liked the look of both and am quite interested to see how they might help on the various projects I’m associated with at the moment. Unit testing javascript and browser testing can be a missing link in even the most complete test suites and I’m quite interested in understanding how to fill that gap.

Before I got terribly far in my research, however, I noted a link at the bottom of the Sinon.js web site. It read:

Sinon uses Semantic versioning

The Semantic Versioning site is little more than a single page that describes a simple specification for how to increment version numbers of releases for purpose of more clearly conveying the meaning of a change to a public API. Attaching a defined meaning to the version number, takes away the somewhat arbitrary method in which version numbers can be assigned. Even when version numbering strategies are not necessarily arbitrary, and definitions exist within the project for version number increments, adhering to a documented public standard makes it easier for users to understand the meaning of change.

As the manifesto states quite clearly, Semantic Versioning “is not a new or revolutionary idea”. As developers, we think in these versioning terms all the time. I think that I found it so interesting because Semantic Versioning creates an opportunity to not only think in these terms all the time, but to think in the same terms all the time.

I Guess This is Me

2011-01-21T00:00:00+00:00

I was asked to create an avatar for use on the tinkerpop web site. It is debatable as to whether or not I’ve captured my likeness in any way, shape or form, but it’s close enough for purposes of the exercise.

A reasonable facsimile…I guess.