John Liao's Blog

Riak Links and Link Walking with F# and CorrugatedIron

2013-05-12T21:00:00.000-07:00

Continuing my journey through the book Seven Databases in Seven Weeks, I explore links and link walking in this blog post. Riak has the ability to establish one-way relationship between entries via Links, providing some of the capabilities of a graph database (Riak documentation calls it a lightweight graph database). Riak documentation hints that links should be kept low, on the order of dozens, not thousands. Using the Twitter example from Riak Handbook, you can probably easily find out who Don Syme follows and who does those people follows, etc. But it would be difficult to find all the people that follows Don Syme using Riak's link capability.

Adding Links

Here's how you would add links with CorrugatedIron library:

type Cage = { room : int }

(*
Linking cage 1 to polly via contains, equivalent to doing the following
curl -X PUT http://localhost:8098/riak/cages/1 \
-H "Content-Type: application/json" \
-H "Link: </riak/animals/polly>; riaktag=\"contains\"" \
-d '{"room" : 101}'
*)
client.Put(
  let cage = new RiakObject("cages","1",{room=101})
  cage.LinkTo("animals","polly","contains")
  cage)

(*
Putting ace in cage 2 and setting cage 2 next to cage 1.
Equivalent to the following sample code:
curl -X PUT http://localhost:8091/riak/cages/2 \
-H "Content-Type: application/json" \
-H "Link:</riak/animals/ace>;riaktag=\"contains\",</riak/cages/1>;riaktag=\"next_to\"" \
-d '{"room" : 101}'
*)
// Adding more than one link  
client.Put(
  let cage = new RiakObject("cages","2",{room=101})
  [("animals","ace","contains");("cages","1","next_to")]
  |> List.iter(fun (bucket,key,tag) -> cage.LinkTo(bucket,key,tag))
  cage)

Link Walking

With CorrugatedIron, I can use the API to perform link walking and as an extra bonus, I don't have to worry about extracting data from the multipart/mixed mime types. CorrugateIron library takes care of all of that for us.

(*
Link walking, equivalent to 
curl http://riakhost1:8098/riak/cages/1/_,_,_
or in the new version:
curl http://riakhost1:8098/buckets/cages/keys/1/_,_,_
*)
 
let results = client.WalkLinks(new RiakObject("cages","1"),
                               [|new RiakLink(null,null,null)|])
         
// Dump results, which returns a list of RiakObject(s)
results.Value

// Since in this particular case, we have only one result, we can do the following
// and get back polly
results.Value.[0].GetObject<Animal>()

Here is the output result for results.Value:

// results.Value
val it : IList<RiakObject> =
  seq
    [CorrugatedIron.Models.RiakObject
       {BinIndexes = dict [];
        Bucket = "animals";
        CharSet = null;
        ContentEncoding = null;
        ContentType = "application/json";
        HasChanged = false;
        IntIndexes = dict [];
        Key = "polly";
        LastModified = 1359758822u;
        LastModifiedUsec = 523420u;
        Links = seq [];
        Siblings = seq [];
        UserMetaData = dict [];
        VTag = "2BTveSKTYDNOZNCiOxyryw";
        VTags = seq ["2BTveSKTYDNOZNCiOxyryw"];
        Value = [|123uy; 32uy; 34uy; 110uy; 105uy; 99uy; 107uy; 110uy; 97uy;
                  109uy; 101uy; 34uy; 32uy; 58uy; 32uy; 34uy; 83uy; 119uy;
                  101uy; 101uy; 116uy; 32uy; 80uy; 111uy; 108uy; 108uy; 121uy;
                  32uy; 80uy; 117uy; 114uy; 101uy; 98uy; 114uy; 101uy; 100uy;
                  34uy; 32uy; 44uy; 32uy; 34uy; 98uy; 114uy; 101uy; 101uy;
                  100uy; 34uy; 32uy; 58uy; 32uy; 34uy; 80uy; 117uy; 114uy;
                  101uy; 98uy; 114uy; 101uy; 100uy; 34uy; 32uy; 125uy|];
        VectorClock = [|107uy; 206uy; 97uy; 96uy; 96uy; 96uy; 204uy; 96uy;
                        202uy; 5uy; 82uy; 28uy; 169uy; 111uy; 239uy; 241uy;
                        7uy; 114uy; 230uy; 184uy; 103uy; 48uy; 37uy; 50uy;
                        230uy; 177uy; 50uy; 60uy; 59uy; 216uy; 112uy; 138uy;
                        47uy; 11uy; 0uy|];}]
      
// results.Value.[0].GetObject<Animal>()
val it : Animal = {nickname = "Sweet Polly Purebred";
                   breed = "Purebred";}

There seems to be some limitations with link walking using CorrugatedIron library. There is a WalkLinks() method as part RiakClient, but the input is expecting RiakLink object which has bucket,key, tag as arguments. The link spec is expecting bucket, tag, and keep flag as arguments. I do notice that in the Riak documentation that Link Walking is not available as part of the Protocol Buffers Client (PBC) API, so I'm guessing the WalkLinks() method in CorrugatedIron is either using HTTP protocol or a modified usage of MapReduce. Since link walking is a special case of MapReduce querying, it may not matter much that CorrugatedIron has some limitations on link walking. One other issue with CorrugatedIron and link walking is that when I add multiple links with the same tag and try to link walking with CorrugatedIron, I do not get back all the links, I only get one of the links. In order to follow the examples in the book Seven Databases in Seven Weeks, I can fall back to using ASP.NET MVC Rest API.

Link walking using CorrugatedIron library:

// Link walking, equivalent to 
// curl http://localhost:8098/riak/cages/2/animals,_,_
client.WalkLinks(new RiakObject("cages","2"),
                 [|new RiakLink("animals",null,null)|])
     
// curl http://localhost:8098/riak/cages/2/_,next_to,0/animals,_,_
client.WalkLinks(new RiakObject("cages","2"),
                 [|new RiakLink(null,null,"next_to");
                   new RiakLink("animals",null,null);|])

// I can't seem to specify the keep flag with CorrugatedIron, which keeps
// intermediate results as you walk beyond primary links
// curl http://localhost:8091/riak/cages/2/_,next_to,1/_,_,_

The book Seven Databases in Seven Weeks is still using the old format for HTTP Link Walking. The new format is as follows:

GET /riak/bucket/key/[bucket],[tag],[keep]            # Old format
GET /buckets/bucket/keys/key/[bucket],[tag],[keep]    # New format

Link walking using REST API:


let riakurl = "http://myriakhost1:8098"
let restClient = new HttpClient()

type LinkWalkSpec = 
    { bucket: string; tag: string;  keep: string; }

    member x.Link = (sprintf "%s,%s,%s" x.bucket x.tag x.keep)

let linkWalker url bucket key (links:LinkWalkSpec list) =
    let baseurl = sprintf "%s/buckets/%s/keys/%s" url bucket key
    let rec buildLinkWalkUrl (linklist:LinkWalkSpec list) baseUrl =
      match linklist with
      | [] -> baseUrl
      | [x] -> sprintf "%s/%s" baseUrl (x.Link)
      | h::t -> let newUrl = sprintf "%s/%s" baseUrl (h.Link)
                buildLinkWalkUrl t newUrl

    buildLinkWalkUrl links baseurl
    |> restClient.GetStringAsync

// Equiv to : curl http://localhost:8091/riak/cages/2/_,next_to,1/_,_,_  
   
[{bucket="_";tag="next_to";keep="1"};{bucket="_";tag="_";keep="_"}] 
|> linkWalker riakurl "cages" "2"

Link walking results from using REST API:

val it : Task<string> =
  System.Threading.Tasks.Task`1[System.String]
    {AsyncState = null;
     CreationOptions = None;
     Exception = null;
     Id = 1;
     IsCanceled = false;
     IsCompleted = false;
     IsFaulted = false;
     Result = "
--CveXyss6PAqBxOOWxeWBCf6eXii
Content-Type: multipart/mixed; boundary=AvYXKrJYDlkeNxh1bQyqDvBAuBF

--AvYXKrJYDlkeNxh1bQyqDvBAuBF
X-Riak-Vclock: a85hYGBgzGDKBVIcqW/v8Qdy5rhnMCUy5rEy9Oi+P8WXBQA=
Location: /buckets/cages/keys/1
Content-Type: application/json
Link: </buckets/animals/keys/polly>; riaktag="contains", </buckets/cages>; rel="up"
Etag: 6gyXgkIgzvwBGRRotqHK3b
Last-Modified: Fri, 26 Apr 2013 16:55:40 GMT

{"room":101}
--AvYXKrJYDlkeNxh1bQyqDvBAuBF--

--CveXyss6PAqBxOOWxeWBCf6eXii
Content-Type: multipart/mixed; boundary=SaWqmmho48dzhMDmJy3BVcCWrzu

--SaWqmmho48dzhMDmJy3BVcCWrzu
X-Riak-Vclock: a85hYGBgzGDKBVIcqW/v8Qdy5rhnMCUy5rEyPDvYcIovCwA=
Location: /buckets/animals/keys/polly
Content-Type: application/json; charset=utf-8
Link: </buckets/animals>; rel="up"
Etag: 2BTveSKTYDNOZNCiOxyryw
Last-Modified: Fri, 01 Feb 2013 22:47:02 GMT

{ "nickname" : "Sweet Polly Purebred" , "breed" : "Purebred" }
--SaWqmmho48dzhMDmJy3BVcCWrzu--

--CveXyss6PAqBxOOWxeWBCf6eXii--
";
     Status = RanToCompletion;}

Exploring Riak with F# and CorrugatedIron

2013-04-26T06:00:00.000-07:00

Many thanks to David and OJ for recommending CorrugatedIron .Net Riak client library. The following blog entry is to document my experiments with Riak CRUD operations using CorrugatedIron.

Pinging RIAK

I tested CorrugatedIron with F# script and here is the setup code along with testing ping capability:

// Needed to load the following libraries to get F# script to work
#r @"c:\dev\FsRiak\packages\CorrugatedIron.1.3.0\lib\net40\CorrugatedIron.dll"
#r @"c:\dev\FsRiak\packages\protobuf-net.2.0.0.621\lib\net40\protobuf-net.dll"
#r @"c:\dev\FsRiak\packages\Newtonsoft.Json.4.5.11\lib\net40\Newtonsoft.Json.dll"

open CorrugatedIron
open CorrugatedIron.Models
open Newtonsoft.Json
open System

// Setup connections
let cluster = RiakCluster.FromConfig("riakConfig", @"c:\dev\FsRiak\App.config");
let client = cluster.CreateClient();

// Ping the Riak Cluster
client.Ping()

Here is my App.config file used by CorrugatedIron:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <configSections>
    <section name="riakConfig" type="CorrugatedIron.Config.RiakClusterConfiguration, CorrugatedIron" />
  </configSections>
  <riakConfig nodePollTime="5000" defaultRetryWaitTime="200" defaultRetryCount="3">
    <nodes>
      <node name="dev1"  hostAddress="mydevhost-a" pbcPort="8087" restScheme="http" restPort="8098" poolSize="10" />
      <node name="dev2" hostAddress="mydevhost-b" pbcPort="8087" restScheme="http" restPort="8098" poolSize="10" />
      <node name="dev3" hostAddress="mydevhost-c" pbcPort="8087" restScheme="http" restPort="8098" poolSize="10" />
    </nodes>
  </riakConfig>
</configuration>

Here's the result from running ping:

val it : RiakResult = CorrugatedIron.RiakResult {ErrorMessage = null;
                                                 IsSuccess = true;
                                                 ResultCode = Success;}

Get List of Buckets

The following method call gets you the list of buckets along with metadata for the call status:

client.ListBuckets()

This method returns the following RiakResult object:

val it : RiakResult<seq<string>> =
  CorrugatedIron.RiakResult`1[System.Collections.Generic.IEnumerable`1[System.String]]
    {ErrorMessage = null;
     IsSuccess = true;
     ResultCode = Success;
     Value = seq ["photos"; "favs"; "animals"; "cages"; ...];}

Get Bucket Keys

Getting a list of keys for a bucket is also simple:

client.ListKeys("animals")

For the novice, this library warns you to not to do this in production environments....

*** [CI] -> ListKeys is an expensive operation and should not be used in Production scenarios. ***
val it : RiakResult<seq<string>> =
  CorrugatedIron.RiakResult`1[System.Collections.Generic.IEnumerable`1[System.String]]
    {ErrorMessage = null;
     IsSuccess = true;
     ResultCode = Success;
     Value = seq ["ace"; "polly"];}

Retrieve Content from Riak

Getting a value from Riak is pretty easy with this library:

client.Get("animals","ace")

A dump of the return object shows the actual data plus metadata about the Get operation:

val it : RiakResult<Models.RiakObject> =
  CorrugatedIron.RiakResult`1[CorrugatedIron.Models.RiakObject]
    {ErrorMessage = null;
     IsSuccess = true;
     ResultCode = Success;
     Value = CorrugatedIron.Models.RiakObject;}

A deeper dive into the Value field of RiakResult object gives the following:

val it : Models.RiakObject =
  CorrugatedIron.Models.RiakObject
    {BinIndexes = dict [];
     Bucket = "animals";
     CharSet = null;
     ContentEncoding = null;
     ContentType = "application/json";
     HasChanged = false;
     IntIndexes = dict [];
     Key = "ace";
     LastModified = 1359744019u;
     LastModifiedUsec = 788127u;
     Links = seq [];
     Siblings = seq [];
     UserMetaData = dict [];
     VTag = "7aPFusRQHlQ36ZP6G6GSyE";
     VTags = seq ["7aPFusRQHlQ36ZP6G6GSyE"];
     Value = [|123uy; 32uy; 34uy; 110uy; 105uy; 99uy; 107uy; 110uy; 97uy;
               109uy; 101uy; 34uy; 32uy; 58uy; 32uy; 34uy; 84uy; 104uy; 101uy;
               32uy; 87uy; 111uy; 110uy; 100uy; 101uy; 114uy; 32uy; 68uy;
               111uy; 103uy; 34uy; 32uy; 44uy; 32uy; 34uy; 98uy; 114uy; 101uy;
               101uy; 100uy; 34uy; 32uy; 58uy; 32uy; 34uy; 71uy; 101uy; 114uy;
               109uy; 97uy; 110uy; 32uy; 83uy; 104uy; 101uy; 112uy; 104uy;
               101uy; 114uy; 100uy; 34uy; 32uy; 125uy|];
     VectorClock = [|107uy; 206uy; 97uy; 96uy; 96uy; 96uy; 204uy; 96uy; 202uy;
                     5uy; 82uy; 28uy; 172uy; 90uy; 225uy; 175uy; 2uy; 57uy;
                     59uy; 156uy; 51uy; 152uy; 18uy; 25uy; 243uy; 88uy; 25uy;
                     132uy; 59uy; 26uy; 78uy; 241uy; 101uy; 1uy; 0uy|];}

The data I really wanted is embedded in another Value field where it is represented as an array of bytes, which is not the final format I want. I wanted to get back the JSON representation of the data I put in. In my previous blog, I had rolled my own JSON serializer/deserializer, but now I wanted to leverage the other pieces of library bundle by CorrugateIron, namely Json.NET. To do so,I define the Animal type and use Json.NET serializer/deserializer to convert between the objects and it's corresponding JSON representation

type Animal =
    { nickname: string; breed: string}

// Getting ace from animals bucket 
client.Get("animals","ace").Value.GetObject<Animal>()

Adding Content to Riak

Adding content to Riak is pretty easy also, after you define a specific type for Json.NET to serialize the fields of that type:

new RiakObject("animals","delta",{nickname="Snoopy"; breed="Beagle"}) 
|> client.Put

If you dump the RiakResult object and the Value field of RiakResult you get the following:


val it : RiakResult =
  CorrugatedIron.RiakResult`1[CorrugatedIron.Models.RiakObject]
    {ErrorMessage = null;
     IsSuccess = true;
     ResultCode = Success;
     Value = CorrugatedIron.Models.RiakObject;}
  
val it : RiakObject =
  CorrugatedIron.Models.RiakObject
    {BinIndexes = dict [];
     Bucket = "animals";
     CharSet = null;
     ContentEncoding = null;
     ContentType = "application/json";
     HasChanged = false;
     IntIndexes = dict [];
     Key = "delta";
     LastModified = 1366930771u;
     LastModifiedUsec = 271921u;
     Links = seq [];
     Siblings = seq [];
     UserMetaData = dict [];
     VTag = "OvZlH7bsYKdO8zL76QdDY";
     VTags = seq ["OvZlH7bsYKdO8zL76QdDY"];
     Value = [|123uy; 34uy; 110uy; 105uy; 99uy; 107uy; 110uy; 97uy; 109uy;
               101uy; 34uy; 58uy; 34uy; 83uy; 110uy; 111uy; 111uy; 112uy;
               121uy; 34uy; 44uy; 34uy; 98uy; 114uy; 101uy; 101uy; 100uy; 34uy;
               58uy; 34uy; 66uy; 101uy; 97uy; 103uy; 108uy; 101uy; 34uy; 125uy|];
     VectorClock = [|107uy; 206uy; 97uy; 96uy; 96uy; 96uy; 204uy; 96uy; 202uy;
                     5uy; 82uy; 28uy; 169uy; 111uy; 239uy; 241uy; 7uy; 114uy;
                     230uy; 184uy; 103uy; 48uy; 37uy; 50uy; 230uy; 177uy; 50uy;
                     4uy; 27uy; 190uy; 59uy; 197uy; 151uy; 5uy; 0uy|];}

To verify interoperability, I would check the newly added data with curl:

$ curl -X GET http://192.168.56.1:8098/riak/animals/delta
$ {"nickname":"Snoopy","breed":"Beagle"}

Delete Riak Contents

Delete content by calling the intuitively named Delete method:

client.Delete("animals","delta")

One thing that I wasn't sure is how CorrugatedIron talks to the Riak clusters. In my simple REST API example, I know exactly which host I'm talking to since I explicitly specified the url. For CorrugatedIron, I configure a pool of connections in the app.config file. I wasn't sure which of the nodes CorrugatedIron was talking to. I fire up Wireshark and notice that I'm connected to the Riak cluster via port 8087, which is through the Protocol Buffers Client (PBC) interface...which explains the need to load the PBC libraries. This Protocol Buffers client is something new to me and the bundled protobuf-net library seemed to be the code developed by Google (Many thanks to OJ for correcting me on this, this is NOT a Google library but a library written by Marc Gravell). This, in turn led me to look at Google Protocol Buffers . This little side jaunt into CorrugatedIron library led to the discovery (for me) of a whole new set of network communication protocols. In any case, after checking Wireshark output, it seems that the requests are spread out to the different nodes and not restricted to a single node. I'm guessing that there are some builtin load balancer code in CorrugatedIron that sends my request to different nodes in the Riak cluster.

For the basic CRUD operations on Riak, CorrugatedIron has made it easier to work with Riak then having to come up with my own helper functions. This has been a good start and I hope work through more of the Riak examples from the book Seven Databases in Seven Weeks with CorrugatedIron in future blog posts.

Exploring Riak with F#

2013-04-15T06:00:00.000-07:00

I have embraced the Polyglot Programming for quite a while already. This year, I wanted to tackle Polyglot Persistence. RDBMS has dominated my world view of persistence layer with everything else as second class citizens. I thought it was time to expand my world view of persistence layers, especially with the burgeoning popularity of NoSQL movement. To begin my exploration, I picked up the book Seven Databases in Seven Weeks by Eric Redmond and Jim Wilson and started with the first NoSQL persistence layer in the book, which was Riak. According to the book:

Riak is a distributed key-value database where values can be anything-from plain text, JSON, or XML to images or video clips-all accessible through a simple HTTP interface.

Setting up Riak

I deployed riak on 3 servers for testing purposes. In setting up the Riak clusters, I ran into the following errors:

10:42:57.339 [error] gen_server riak_core_capability terminated with reason: no function clause matching orddict:fetch('riak@192.168.56.1', [
{'riak@127.0.0.1',[{{riak_core,staged_joins},[true,false]},{{riak_core,vnode_routing},[proxy,legacy]},...]}]) line 72
/users/domains/riak/lib/os_mon-2.2.9/priv/bin/memsup: Erlang has closed.

A quick Google search brought up the following link Googling the web, I got this link: http://blog.alwayshere.info/2012/11/riak-error-genserver-riakcorecapability.html I originally started riak with 127.0.0.1 address. Then I made the modification to the configuration as documented in http://docs.basho.com/riak/latest/cookbooks/Basic-Cluster-Setup/ in trying to setup my 3 server Riak into a cluster. To fix my error, I had to go to ./data/ring folder and delete everything in there, then everything works as expected.

Most of the examples in the book leveraged curl. However, I learn best if I tried to work the examples in another way. I tried the examples in Clojure, using ClojureWerkz's Welle. I liked ClojureWerkz's Welle wrappers to Riak and would probably use it if I had to develop on Java platform. I also wanted to work with Riak from .NET platform and hence I'm using F# to explore Riak. The following examples where done on Visual Studio 2010 with ASP.NET MVC 4 installed. This also gives me a chance to take the REST API in ASP.NET MVC4 for a spin.

Pinging RIAK

The very first example in the book is to ping the Riak cluster, here's how I implemented it in F#

open System.Net.Http
open System.Threading.Tasks

// My 3 instances of Riak
let riak1_url = "http://192.168.56.1:8098"
let riak2_url = "http://192.168.56.2:8098"
let riak3_url = "http://192.168.56.3:8098"

// Pick one to work with
let riakurl = riak1_url

let client = new HttpClient()

let ping () = client.GetAsync(sprintf "%s/ping" riakurl)

ping()

Running the ping, I would get the following response from F#:

val it : Task =
  System.Threading.Tasks.Task`1[System.Net.Http.HttpResponseMessage]
    {AsyncState = null;
     CreationOptions = None;
     Exception = null;
     Id = 4;
     IsCanceled = false;
     IsCompleted = false;
     IsFaulted = false;
     Result = StatusCode: 200, ReasonPhrase: 'OK', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Vary: Accept-Encoding
  Date: Wed, 30 Jan 2013 22:20:11 GMT
  Server: MochiWeb/1.1
  Server: WebMachine/1.9.0
  Server: (someone had painted it blue)
  Content-Length: 422
  Content-Type: application/json
};
     Status = RanToCompletion;}

Adding Content to RIAK

Let's start by putting some stuff into Riak with the following snippet of code

let put bucket key content =
    let put_url= sprintf "%s/riak/%s/%s" riakurl bucket key
    client.PutAsync(put_url,content)


let put_html bucket key html =
    let put_url= sprintf "%s/riak/%s/%s" riakurl bucket key
    let content = new StringContent(html)
    content.Headers.ContentType.MediaType <- "text/html"
    put bucket key content


"<html><body><h1>My latest favorite DB is RIAK</h1></body></html>"
|> put_html "favs" "db"

Running the above script gets the following response:

val it : Task =
  System.Threading.Tasks.Task`1[System.Net.Http.HttpResponseMessage]
    {AsyncState = null;
     CreationOptions = None;
     Exception = null;
     Id = 5;
     IsCanceled = false;
     IsCompleted = false;
     IsFaulted = false;
     Result = StatusCode: 204, ReasonPhrase: 'No Content', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Vary: Accept-Encoding
  Date: Fri, 01 Feb 2013 18:07:24 GMT
  Server: MochiWeb/1.1
  Server: WebMachine/1.9.0
  Server: (someone had painted it blue)
  Content-Length: 0
  Content-Type: text/html; charset=utf-8
};
     Status = RanToCompletion;}

We got the 204 code as explained in the book. To test that Riak has stored this new content, simply point to any of the Riak instances, (e.g. http://192.168.56.3:8098/riak/favs/db) with a browser and you should see the webpage that was put into the first Riak server.

Here's the sample code to put JSON data into Riak:

// Simple utility to generate JSON - should really use a real JSON library
let tojson data = 
    data |> Seq.map (fun (k,v) -> sprintf "\"%s\" : \"%s\"" k v)
         |> Seq.reduce (sprintf "%s , %s")
         |> sprintf "{ %s }" 


let put_json bucket key jsondata =
    let content = new StringContent(tojson jsondata)
    content.Headers.ContentType.MediaType <- "application/json"
    put bucket key content

[("nickname","The Wonder Dog"); ("breed","German Shepherd")]
|> put_json "animals" "ace"

Again, you can check that it's stored in Riak by pointing the browser to: http://192.168.56.3:8098/riak/animals/ace and you should get back:

{ "nickname" : "The Wonder Dog" , "breed" : "German Shepherd" }

Removing Content from RIAK

Here's a snippet of script to remove a content from Riak


let delete bucket key =
    let delete_url= sprintf "%s/riak/%s/%s" riakurl bucket key 
    client.DeleteAsync(delete_url)

delete "animals" "ace"

Getting Bucket Keys

To get all keys in a bucket

let get_keys bucket =
    let get_url = sprintf "%s/riak/%s?keys=true" riakurl bucket
    get_url |> client.GetStringAsync

let results = get_keys "animals"
printfn "%s" results.Result

The above script would return the following (reformatted for legibility purposes):

{"props":{"name":"animals",
          "allow_mult":false,
    "basic_quorum":false,
    "big_vclock":50,
    "chash_keyfun":{"mod":"riak_core_util",
                    "fun":"chash_std_keyfun"},
    "dw":"quorum",
    "last_write_wins":false,
    "linkfun":{"mod":"riak_kv_wm_link_walker",
               "fun":"mapreduce_linkfun"},
    "n_val":3,
    "notfound_ok":true,
    "old_vclock":86400,
    "postcommit":[],
    "pr":0,
    "precommit":[],
    "pw":0,
    "r":"quorum",
    "rw":"quorum",
    "small_vclock":50,
    "w":"quorum",
    "young_vclock":20},
    "keys":["ace","polly"]}

Retrieving Content from Riak

To retrieve content:

let get bucket key = 
    let get_url = sprintf "%s/riak/%s/%s/" riakurl bucket key
    get_url |> client.GetStringAsync

let results = get "animals" "ace"
printfn "%s" results.Result

Compare Tibco EMS Performance Under Load with Clojure

2013-03-25T06:00:00.000-07:00

We have migrated our Tibco EMS infrastructure from the older Sun V490 servers to the newer HP ProLiant DL380 G7 servers. One of the concerns was how does the new servers perform under load compared with the older servers. I needed to create a load test that compared the performance of the two different types of servers. I turned to Clojure to help me put together this load test and here is the load test code:

Load Test Script in Clojure

; Leveraging Apache Commons to generate the random message
(def msg64 (RandomStringUtils/randomAlphanumeric 64))
(def msg128 (RandomStringUtils/randomAlphanumeric 128))
(def msg256 (RandomStringUtils/randomAlphanumeric 256))
(def msg64k (RandomStringUtils/randomAlphanumeric (* 64 1024)))
(def msg128k (RandomStringUtils/randomAlphanumeric (* 128 1024)))
(def msg256k (RandomStringUtils/randomAlphanumeric (* 256 1024)))
(def msg512k (RandomStringUtils/randomAlphanumeric (* 512 1024)))
(def msg1mb (RandomStringUtils/randomAlphanumeric (* 1024 1024)))
(def persistent-delivery (.intValue (javax.jms.DeliveryMode/PERSISTENT)))


; Send messages
(defn send-messages [server-url user password queue-name data iterations]
  (with-open [connection (-> (TibjmsQueueConnectionFactory. server-url)
            (.createQueueConnection user password))]
    (let [session (.createQueueSession connection false Session/AUTO_ACKNOWLEDGE) 
         queue   (.createQueue session  queue-name)]
      (with-open [sender (.createSender session queue)]
  (dotimes [_ iterations]
          (let [message (.createTextMessage session)]
            (.setJMSDeliveryMode message persistent-delivery)
            (.setText message data)
            (.send sender message)))))))
   

(def test-scenarios (list
 {:label "64 bytes" :msg msg64 :n 5000 }
 {:label "128 bytes" :msg msg128 :n 5000 }
 {:label "256 bytes" :msg msg256 :n 5000 }
 {:label "64KB bytes" :msg msg64k :n 2000 }
 {:label "128KB bytes" :msg msg128k :n 1000 }
 {:label "256KB bytes" :msg msg256k :n 1000 }
 {:label "512KB bytes" :msg msg512k :n 1000 }
 {:label "1MB bytes" :msg msg1mb :n 1000 }))

(doseq [my-test test-scenarios]
  (let [label (:label my-test)
       message (:msg my-test)
       n (:n my-test)]
    (println "\n\n")
 (println (str "Testing for " label " sized messages with n = " n))
 (time (send-messages server-url username password test-queue message n))))

Here's the message consumer part of the load test:

Message Consumer in Clojure

                
; Consume Queue Text messages asynchronously
(defn get-queue-text-messages [server-url user password queue-name process-message]
    (future
        (with-open [connection (-> (TibjmsQueueConnectionFactory. server-url)
                                   (.createQueueConnection user password))]
            (let [session (.createQueueSession connection false Session/AUTO_ACKNOWLEDGE)
                  queue (.createQueue session  queue-name)]
                (with-open [receiver (.createReceiver session queue)]             
                    (.start connection)
                    (loop [acc 0]                      
                        (process-message (.receive receiver) acc)
                        (recur (+ acc 1))))))))

; Dump just the message id      
(defn dump-message-id [message n]
    (println (str n)))

     
; Create function aliases with connection information embedded                   
(defn consume-messages [queue-name message-processor]
    (get-queue-text-messages  server-url username password queue-name message-processor))
 
; Start consuming messages asynchronously
(consume-messages queueName dump-message-id)

Here are the test results:

Load Test Result on Sun V490

Message Size	Number of Messages	Process Time (ms)	Throughput (msg/s)	Throughput (MB/s)
64	5,000	2,772.84	1,803.21	0.11
128	5,000	1,943.23	2,573.04	0.31
256	5,000	1,899.24	2,632.63	0.64
64K	1,000	3,020.98	331.02	20.69
128K	1,000	4,414.00	226.55	28.32
256K	1,000	20,911.61	47.82	11.96
512K	1,000	39,213.23	25.50	12.75
1MB	1,000	80,337.55	12.45	12.45

Load Test Result on HP ProLiant DL380 G7

Message Size	Number of Messages	Process Time (ms)	Throughput (msg/s)	Throughput (MB/s)
64	5,000	1,340.22	3,730.74	0.23
128	5,000	1,345.18	3,716.99	0.45
256	5,000	1,040.59	4,804.95	1.17
64K	1,000	2,178.69	458.99	28.69
128K	1,000	3,553.02	281.45	35.18
256K	1,000	7,490.54	133.50	33.38
512K	1,000	35,078.26	28.51	14.25
1MB	1,000	77,076.51	12.97	12.97

From this test result, we can conclude that HP ProLiant DL380 G7 performed better than Sun V490 under load.

Retrieving VMware vSphere Performance Metrics Catalog with F#

2013-03-04T06:00:00.000-08:00

vSphere collects a lot of performance metrics data on a VMware cluster. Our problem was that we cannot find a complete documentation on what are all those performance metrics data collected and what are the units of the collected metric data. We have asked VMware professional services engineer about this and his response was that VMware does not have the metrics data documented.

However, in Chapter 16 of the vSphere Web Services Programming Guide, it talked about the fact that you can retrieve the metrics catalog from vSphere itself. Although this document talks about retrieving the metrics catalog via vSphere Web Services SDK, you can also retrieve the metrics catalog with PowerCLI. Below is a F# script that retrieves vSphere metrics catalog:

#r @"C:\pkg\VMware\Infrastructure\vSphere PowerCLI\VMware.Vim.dll"

open System
open VMware.Vim
open System.Collections.Specialized

let client = new VimClient()
let service = client.Connect(serviceUrl)

// Must login to do anything - if you are getting null values, it means the session automatically timed out
client.Login(userId,password)

let printMetricCatalog () =
 // counterId max is arbitrarily set, I tried other values
 // and still produced the same number of returned metrics
    let counterIds = seq {1 .. 1000}
    let perfMgr = new PerformanceManager(client,client.ServiceContent.PerfManager)
    let metrics = perfMgr.QueryPerfCounter(counterIds |> Seq.toArray)

    let dumpAllMetricData (metric:PerfCounterInfo) = 
        printfn "-------------------------------------------------------------------\n"
        printfn "Key             : %i" metric.Key
        printfn "Level           : %A" metric.Level
        printfn "PerDeviceLevel  : %A" metric.PerDeviceLevel
        printfn "RollupType      : %A" metric.RollupType
        printfn "StatsType       : %A" metric.StatsType
        printfn "Group.Label     : %s" metric.GroupInfo.Label
        printfn "Group.Key       : %s" metric.GroupInfo.Key
        printfn "Group.Summary   : %s" metric.GroupInfo.Summary
        printfn "NameInfo.Label  : %s" metric.NameInfo.Label
        printfn "NameInfo.Key    : %s" metric.NameInfo.Key
        printfn "NameInfo.Summary: %s" metric.NameInfo.Summary
        printfn "UnitInfo.Label  : %s" metric.UnitInfo.Label
        printfn "UnitInfo.Key    : %s" metric.UnitInfo.Key
        printfn "UnitInfo.Summary: %s" metric.UnitInfo.Summary


    let dumpMetricData (metric:PerfCounterInfo) = printfn "\t%s (%s) - %A stat,  rollup: %A" 
                                                          (metric.NameInfo.Label) 
                                                          (metric.UnitInfo.Label) 
                                                          (metric.StatsType)
                                                          (metric.RollupType)

    // Break metrics catalog by metric groups
    let metricGroups = metrics |> Seq.map (fun metric -> metric.GroupInfo.Label) |> Set.ofSeq |> Set.toSeq

    // Dump short version of metrics catalog
    metricGroups
    |> Seq.iter (fun group ->  printfn "\n--------------------------------------------------"
                               printfn " Metric Group : %s" group
                               printfn "--------------------------------------------------"
                               metrics
                               |> Seq.filter (fun metric -> metric.GroupInfo.Label = group)
                               |> Seq.iter dumpMetricData)

    // Dump complete version of metrics catalog
    metrics |> Seq.iter dumpAllMetricData

printMetricCatalog()

Here's a dump of the metrics catalog reformatted to display as HTML tables:

VMware vSphere Metrics Catalog

CPU

Metric	Unit	Stats Type	Rollup	Description
CPU Capacity Contention	Percent	rate	average	Percent of time the VMs on this host are unable to run because they are contending for access to the physical CPU(s)
CPU Capacity Demand	MHz	absolute	average	The amount of CPU resources VMs on this host would use if there were no CPU contention or CPU limit
CPU Capacity Entitlement	MHz	absolute	average	CPU resources devoted by the ESX scheduler to virtual machines and resource pools
CPU Capacity Provisioned	MHz	absolute	average	Capacity in MHz of the physical CPU cores
CPU Capacity Usage	MHz	rate	average	CPU usage in MHz during the interval
CPU Core Count Contention	Percent	rate	average	Time the VM is ready to run, but is unable to run due to co-scheduling constraints
CPU Core Count Provisioned	Number	absolute	average	The number of physical cores provisioned to the entity
CPU Core Count Usage	Number	absolute	average	The number of virtual processors running on the host
Co-stop	Millisecond	delta	summation	Time the VM is ready to run, but is unable to due to co-scheduling constraints
Core Utilization	Percent	rate	none	CPU utilization of the corresponding core (if hyper-threading is enabled) as a percentage during the interval (A core is utilized if either or both of its logical CPUs are utilized)
Core Utilization	Percent	rate	average	CPU utilization of the corresponding core (if hyper-threading is enabled) as a percentage during the interval (A core is utilized if either or both of its logical CPUs are utilized)
Core Utilization	Percent	rate	maximum	CPU utilization of the corresponding core (if hyper-threading is enabled) as a percentage during the interval (A core is utilized if either or both of its logical CPUs are utilized)
Core Utilization	Percent	rate	minimum	CPU utilization of the corresponding core (if hyper-threading is enabled) as a percentage during the interval (A core is utilized if either or both of its logical CPUs are utilized)
Demand	MHz	absolute	average	The amount of CPU resources a VM would use if there were no CPU contention or CPU limit
Entitlement	MHz	absolute	latest	CPU resources devoted by the ESX scheduler
Idle	Millisecond	delta	summation	Total time that the CPU spent in an idle state
Latency	Percent	rate	average	Percent of time the VM is unable to run because it is contending for access to the physical CPU(s)
Max limited	Millisecond	delta	summation	Time the VM is ready to run, but is not run due to maxing out its CPU limit setting
Overlap	Millisecond	delta	summation	Time the VM was interrupted to perform system services on behalf of that VM or other VMs
Ready	Millisecond	delta	summation	Percentage of time that the virtual machine was ready, but could not get scheduled to run on the physical CPU
Reserved capacity	MHz	absolute	average	Total CPU capacity reserved by virtual machines
Run	Millisecond	delta	summation	Time the VM is scheduled to run
Swap wait	Millisecond	delta	summation	CPU time spent waiting for swap-in
System	Millisecond	delta	summation	Amount of time spent on system processes on each virtual CPU in the virtual machine
Total	MHz	rate	average	Total amount of CPU resources of all hosts in the cluster
Total capacity	MHz	absolute	average	Total CPU capacity reserved by and available for virtual machines
Usage	Percent	rate	none	CPU usage as a percentage during the interval
Usage	Percent	rate	average	CPU usage as a percentage during the interval
Usage	Percent	rate	minimum	CPU usage as a percentage during the interval
Usage	Percent	rate	maximum	CPU usage as a percentage during the interval
Usage in MHz	MHz	rate	none	CPU usage in megahertz during the interval
Usage in MHz	MHz	rate	average	CPU usage in megahertz during the interval
Usage in MHz	MHz	rate	minimum	CPU usage in megahertz during the interval
Usage in MHz	MHz	rate	maximum	CPU usage in megahertz during the interval
Used	Millisecond	delta	summation	Total CPU usage
Utilization	Percent	rate	none	CPU utilization as a percentage during the interval (CPU usage and CPU utilization may be different due to power management technologies or hyper-threading)
Utilization	Percent	rate	average	CPU utilization as a percentage during the interval (CPU usage and CPU utilization may be different due to power management technologies or hyper-threading)
Utilization	Percent	rate	maximum	CPU utilization as a percentage during the interval (CPU usage and CPU utilization may be different due to power management technologies or hyper-threading)
Utilization	Percent	rate	minimum	CPU utilization as a percentage during the interval (CPU usage and CPU utilization may be different due to power management technologies or hyper-threading)
Wait	Millisecond	delta	summation	Total CPU time spent in wait state
Worst case allocation	MHz	absolute	latest	Amount of CPU resources allocated to the virtual machine or resource pool based on the total cluster capacity and the resource configuration of the resource hierarchy

Cluster services

Metric	Unit	Stats Type	Rollup	Description
CPU fairness	Number	absolute	latest	Fairness of distributed CPU resource allocation
Current failover level	Number	absolute	latest	vSphere HA number of failures that can be tolerated
Effective CPU resources	MHz	rate	average	Total available CPU resources of all hosts within a cluster
Effective memory resources	MB	absolute	average	Total amount of machine memory of all hosts in the cluster that is available for use for virtual machine memory and overhead memory
Memory fairness	Number	absolute	latest	Aggregate available memory resources of all the hosts within a cluster

Datastore

Metric	Unit	Stats Type	Rollup	Description
Average read requests per second	Number	rate	average	Average number of read commands issued per second to the datastore during the collection interval
Average write requests per second	Number	rate	average	Average number of write commands issued per second to the datastore during the collection interval
Highest latency	Millisecond	absolute	latest	Highest latency value across all datastores used by the host
Read latency	Millisecond	absolute	average	The average time a read from the datastore takes
Read rate	KBps	rate	average	Rate of reading data from the datastore
Storage DRS datastore bytes read	Number	absolute	latest	Storage DRS datastore bytes read
Storage DRS datastore bytes written	Number	absolute	latest	Storage DRS datastore bytes written
Storage DRS datastore normalized read latency	Number	absolute	latest	Storage DRS datastore normalized read latency
Storage DRS datastore normalized write latency	Number	absolute	latest	Storage DRS datastore normalized write latency
Storage DRS datastore outstanding read requests	Number	absolute	latest	Storage DRS datastore outstanding read requests
Storage DRS datastore outstanding write requests	Number	absolute	latest	Storage DRS datastore outstanding write requests
Storage DRS datastore read I/O rate	Number	absolute	latest	Storage DRS datastore read I/O rate
Storage DRS datastore read workload metric	Number	absolute	latest	Storage DRS datastore metric for read workload model
Storage DRS datastore write I/O rate	Number	absolute	latest	Storage DRS datastore write I/O rate
Storage DRS datastore write workload metric	Number	absolute	latest	Storage DRS datastore metric for write workload model
Storage I/O Control aggregated IOPS	Number	absolute	average	Storage I/O Control aggregated IOPS
Storage I/O Control datastore maximum queue depth	Number	absolute	latest	Storage I/O Control datastore maximum queue depth
Storage I/O Control normalized latency	Microsecond	absolute	average	Storage I/O Control size-normalized I/O latency
Write latency	Millisecond	absolute	average	The average time a write to the datastore takes
Write rate	KBps	rate	average	Rate of writing data to the datastore
busResets	Number	delta	summation	busResets
commandsAborted	Number	delta	summation	commandsAborted
contention	Millisecond	absolute	average	contention
usage	KBps	absolute	average	usage

Disk

Metric	Unit	Stats Type	Rollup	Description
Average commands issued per second	Number	rate	average	Average number of SCSI commands issued per second during the collection interval
Average read requests per second	Number	rate	average	Average number of disk reads per second during the collection interval
Average write requests per second	Number	rate	average	Average number of disk writes per second during the collection interval
Bus resets	Number	delta	summation	Number of SCSI-bus reset commands issued during the collection interval
Capacity	KB	absolute	latest	Configured size of the datastore
Command latency	Millisecond	absolute	average	Average amount of time taken during the collection interval to process a SCSI command issued by the Guest OS to the virtual machine
Commands issued	Number	delta	summation	Number of SCSI commands issued during the collection interval
Commands terminated	Number	delta	summation	Number of SCSI commands terminated during the collection interval
Disk SCSI Reservation Conflicts	Number	delta	summation	Number of SCSI reservation conflicts for the LUN during the collection interval
Disk SCSI Reservation Conflicts %	Percent	absolute	average	Number of SCSI reservation conflicts for the LUN as a percent of total commands during the collection interval
Disk Throughput Contention	Millisecond	absolute	average	Average amount of time for an I/O operation to complete
Disk Throughput Usage	KBps	rate	average	Aggregated disk I/O rate
Highest latency	Millisecond	absolute	latest	Highest latency value across all disks used by the host
Kernel command latency	Millisecond	absolute	average	Average amount of time, in milliseconds, spent by VMkernel to process each SCSI command
Kernel read latency	Millisecond	absolute	average	Average amount of time, in milliseconds, spent by VMKernel to process each SCSI read command
Kernel write latency	Millisecond	absolute	average	Average amount of time, in milliseconds, spent by VMKernel to process each SCSI write command
Maximum queue depth	Number	absolute	average	Maximum queue depth
Overhead due to delta disk backings	KB	absolute	latest	Storage overhead of a virtual machine or a datastore due to delta disk backings
Physical device command latency	Millisecond	absolute	average	Average amount of time, in milliseconds, to complete a SCSI command from the physical device
Physical device read latency	Millisecond	absolute	average	Average amount of time, in milliseconds, to read from the physical device
Physical device write latency	Millisecond	absolute	average	Average amount of time, in milliseconds, to write to the physical device
Queue command latency	Millisecond	absolute	average	Average amount of time spent in the VMkernel queue, per SCSI command, during the collection interval
Queue read latency	Millisecond	absolute	average	Average amount of time spent in the VMkernel queue, per SCSI read command, during the collection interval
Queue write latency	Millisecond	absolute	average	Average amount time spent in the VMkernel queue, per SCSI write command, during the collection interval
Read latency	Millisecond	absolute	average	Average amount of time taken during the collection interval to process a SCSI read command issued from the Guest OS to the virtual machine
Read rate	KBps	rate	average	Average number of kilobytes read from the disk each second during the collection interval
Read requests	Number	delta	summation	Number of disk reads during the collection interval
Space actually used	KB	absolute	latest	Amount of space actually used by the virtual machine or the datastore
Space not shared	KB	absolute	latest	Amount of space associated exclusively with a virtual machine
Space potentially used	KB	absolute	latest	Amount of storage set aside for use by a datastore or a virtual machine
Usage	KBps	rate	none	Aggregated disk I/O rate. For hosts, this metric includes the rates for all virtual machines running on the host during the collection interval.
Usage	KBps	rate	average	Aggregated disk I/O rate. For hosts, this metric includes the rates for all virtual machines running on the host during the collection interval.
Usage	KBps	rate	minimum	Aggregated disk I/O rate. For hosts, this metric includes the rates for all virtual machines running on the host during the collection interval.
Usage	KBps	rate	maximum	Aggregated disk I/O rate. For hosts, this metric includes the rates for all virtual machines running on the host during the collection interval.
Write latency	Millisecond	absolute	average	Average amount of time taken during the collection interval to process a SCSI write command issued by the Guest OS to the virtual machine
Write rate	KBps	rate	average	Average number of kilobytes written to disk each second during the collection interval
Write requests	Number	delta	summation	Number of disk writes during the collection interval
contention	Percent	absolute	average	contention
provisioned	KB	absolute	average	provisioned
usage	KB	absolute	average	usage

Management agent

Metric	Unit	Stats Type	Rollup	Description
CPU usage	MHz	rate	average	Amount of Service Console CPU usage
Memory swap in	KBps	rate	average	Amount of memory that is swapped in for the Service Console
Memory swap out	KBps	rate	average	Amount of memory that is swapped out for the Service Console
Memory swap used	KB	absolute	average	Sum of the memory swapped by all powered-on virtual machines on the host
Memory used	KB	absolute	average	Amount of total configured memory that is available for use

Memory

Metric	Unit	Stats Type	Rollup	Description
Active	KB	absolute	none	Amount of memory that is actively used, as estimated by VMkernel based on recently touched memory pages
Active	KB	absolute	average	Amount of memory that is actively used, as estimated by VMkernel based on recently touched memory pages
Active	KB	absolute	minimum	Amount of memory that is actively used, as estimated by VMkernel based on recently touched memory pages
Active	KB	absolute	maximum	Amount of memory that is actively used, as estimated by VMkernel based on recently touched memory pages
Active write	KB	absolute	average	Amount of memory that is actively being written to by the VM
Balloon	KB	absolute	none	Amount of memory allocated by the virtual machine memory control driver (vmmemctl), which is installed with VMware Tools
Balloon	KB	absolute	average	Amount of memory allocated by the virtual machine memory control driver (vmmemctl), which is installed with VMware Tools
Balloon	KB	absolute	minimum	Amount of memory allocated by the virtual machine memory control driver (vmmemctl), which is installed with VMware Tools
Balloon	KB	absolute	maximum	Amount of memory allocated by the virtual machine memory control driver (vmmemctl), which is installed with VMware Tools
Balloon target	KB	absolute	none	Target value set by VMkernal for the virtual machine's memory balloon size
Balloon target	KB	absolute	average	Target value set by VMkernal for the virtual machine's memory balloon size
Balloon target	KB	absolute	minimum	Target value set by VMkernal for the virtual machine's memory balloon size
Balloon target	KB	absolute	maximum	Target value set by VMkernal for the virtual machine's memory balloon size
Compressed	KB	absolute	average	Amount of memory compressed by ESX
Compression rate	KBps	rate	average	Rate of memory compression for the VM
Consumed	KB	absolute	none	Amount of memory consumed by a virtual machine, host, or cluster
Consumed	KB	absolute	average	Amount of memory consumed by a virtual machine, host, or cluster
Consumed	KB	absolute	minimum	Amount of memory consumed by a virtual machine, host, or cluster
Consumed	KB	absolute	maximum	Amount of memory consumed by a virtual machine, host, or cluster
Decompression rate	KBps	rate	average	Rate of memory decompression for the VM
Entitlement	KB	absolute	average	Amount of host physical memory the VM is entitled to, as determined by the ESX scheduler
Granted	KB	absolute	none	Amount of machine memory or physical memory that is mapped for a virtual machine or a host
Granted	KB	absolute	average	Amount of machine memory or physical memory that is mapped for a virtual machine or a host
Granted	KB	absolute	minimum	Amount of machine memory or physical memory that is mapped for a virtual machine or a host
Granted	KB	absolute	maximum	Amount of machine memory or physical memory that is mapped for a virtual machine or a host
Heap	KB	absolute	none	VMkernel virtual address space dedicated to VMkernel main heap and related data
Heap	KB	absolute	average	VMkernel virtual address space dedicated to VMkernel main heap and related data
Heap	KB	absolute	minimum	VMkernel virtual address space dedicated to VMkernel main heap and related data
Heap	KB	absolute	maximum	VMkernel virtual address space dedicated to VMkernel main heap and related data
Heap free	KB	absolute	none	Free address space in the VMkernel's main heap
Heap free	KB	absolute	average	Free address space in the VMkernel's main heap
Heap free	KB	absolute	minimum	Free address space in the VMkernel's main heap
Heap free	KB	absolute	maximum	Free address space in the VMkernel's main heap
Host cache used for swapping	KB	absolute	none	Space used for caching swapped pages in the host cache
Host cache used for swapping	KB	absolute	average	Space used for caching swapped pages in the host cache
Host cache used for swapping	KB	absolute	maximum	Space used for caching swapped pages in the host cache
Host cache used for swapping	KB	absolute	minimum	Space used for caching swapped pages in the host cache
Latency	Percent	rate	average	Percentage of time the VM is waiting to access swapped or compressed memory
Low free threshold	KB	absolute	average	Threshold of free host physical memory below which ESX will begin reclaiming memory from VMs through ballooning and swapping
Memory Capacity Contention	Percent	rate	average	Percentage of time the VM is waiting to access swapped, compressed, or ballooned memory
Memory Capacity Entitlement	KB	absolute	average	Amount of host physical memory the VM is entitled to, as determined by the ESX scheduler
Memory Capacity Provisioned	KB	absolute	average	Total amount of memory configured for the VM
Memory Capacity Usable	KB	absolute	average	Amount of physical memory available for use by virtual machines on this host
Memory Capacity Usage	KB	absolute	average	Amount of physical memory actively used
Memory Consumed by VMs	KB	absolute	average	Amount of physical memory consumed by VMs on this host
Memory Consumed by userworlds	KB	absolute	average	Amount of physical memory consumed by userworlds on this host
Memory Reserved Capacity %	Percent	absolute	average	Percent of memory that has been reserved either through VMkernel use, by userworlds, or due to VM memory reservations
Memory saved by zipping	KB	absolute	latest	Memory (KB) saved due to memory zipping
Overhead	KB	absolute	none	Memory (KB) consumed by the virtualization infrastructure for running the VM
Overhead	KB	absolute	average	Memory (KB) consumed by the virtualization infrastructure for running the VM
Overhead	KB	absolute	minimum	Memory (KB) consumed by the virtualization infrastructure for running the VM
Overhead	KB	absolute	maximum	Memory (KB) consumed by the virtualization infrastructure for running the VM
Overhead touched	KB	absolute	average	Actively touched overhead memory (KB) reserved for use as the virtualization overhead for the VM
Reserved capacity	MB	absolute	average	Total amount of memory reservation used by powered-on virtual machines and vSphere services on the host
Reserved overhead	KB	absolute	average	Memory (KB) reserved for use as the virtualization overhead for the VM
Shared	KB	absolute	none	Amount of guest memory that is shared with other virtual machines, relative to a single virtual machine or to all powered-on virtual machines on a host
Shared	KB	absolute	average	Amount of guest memory that is shared with other virtual machines, relative to a single virtual machine or to all powered-on virtual machines on a host
Shared	KB	absolute	minimum	Amount of guest memory that is shared with other virtual machines, relative to a single virtual machine or to all powered-on virtual machines on a host
Shared	KB	absolute	maximum	Amount of guest memory that is shared with other virtual machines, relative to a single virtual machine or to all powered-on virtual machines on a host
Shared common	KB	absolute	none	Amount of machine memory that is shared by all powered-on virtual machines and vSphere services on the host
Shared common	KB	absolute	average	Amount of machine memory that is shared by all powered-on virtual machines and vSphere services on the host
Shared common	KB	absolute	minimum	Amount of machine memory that is shared by all powered-on virtual machines and vSphere services on the host
Shared common	KB	absolute	maximum	Amount of machine memory that is shared by all powered-on virtual machines and vSphere services on the host
State	Number	absolute	latest	One of four threshold levels representing the percentage of free memory on the host. The counter value determines swapping and ballooning behavior for memory reclamation.
Swap in	KB	absolute	none	Amount swapped-in to memory from disk
Swap in	KB	absolute	average	Amount swapped-in to memory from disk
Swap in	KB	absolute	maximum	Amount swapped-in to memory from disk
Swap in	KB	absolute	minimum	Amount swapped-in to memory from disk
Swap in from host cache	KB	absolute	none	Amount of memory swapped-in from host cache
Swap in from host cache	KB	absolute	average	Amount of memory swapped-in from host cache
Swap in from host cache	KB	absolute	maximum	Amount of memory swapped-in from host cache
Swap in from host cache	KB	absolute	minimum	Amount of memory swapped-in from host cache
Swap in rate	KBps	rate	average	Rate at which memory is swapped from disk into active memory during the interval
Swap in rate from host cache	KBps	rate	average	Rate at which memory is being swapped from host cache into active memory
Swap out	KB	absolute	none	Amount of memory swapped-out to disk
Swap out	KB	absolute	average	Amount of memory swapped-out to disk
Swap out	KB	absolute	maximum	Amount of memory swapped-out to disk
Swap out	KB	absolute	minimum	Amount of memory swapped-out to disk
Swap out rate	KBps	rate	average	Rate at which memory is being swapped from active memory to disk during the current interval
Swap out rate to host cache	KBps	rate	average	Rate at which memory is being swapped from active memory to host cache
Swap out to host cache	KB	absolute	none	Amount of memory swapped-out to host cache
Swap out to host cache	KB	absolute	average	Amount of memory swapped-out to host cache
Swap out to host cache	KB	absolute	maximum	Amount of memory swapped-out to host cache
Swap out to host cache	KB	absolute	minimum	Amount of memory swapped-out to host cache
Swap target	KB	absolute	none	Target size for the virtual machine swap file
Swap target	KB	absolute	average	Target size for the virtual machine swap file
Swap target	KB	absolute	minimum	Target size for the virtual machine swap file
Swap target	KB	absolute	maximum	Target size for the virtual machine swap file
Swap unreserved	KB	absolute	none	Amount of memory that is unreserved by swap
Swap unreserved	KB	absolute	average	Amount of memory that is unreserved by swap
Swap unreserved	KB	absolute	minimum	Amount of memory that is unreserved by swap
Swap unreserved	KB	absolute	maximum	Amount of memory that is unreserved by swap
Swap used	KB	absolute	none	Amount of memory that is used by swap
Swap used	KB	absolute	average	Amount of memory that is used by swap
Swap used	KB	absolute	minimum	Amount of memory that is used by swap
Swap used	KB	absolute	maximum	Amount of memory that is used by swap
Swapped	KB	absolute	none	Current amount of guest physical memory swapped out to the virtual machine's swap file by the VMkernel
Swapped	KB	absolute	average	Current amount of guest physical memory swapped out to the virtual machine's swap file by the VMkernel
Swapped	KB	absolute	minimum	Current amount of guest physical memory swapped out to the virtual machine's swap file by the VMkernel
Swapped	KB	absolute	maximum	Current amount of guest physical memory swapped out to the virtual machine's swap file by the VMkernel
Total	MB	absolute	average	Total amount of machine memory of all hosts in the cluster that is available for virtual machine memory (physical memory for use by the Guest OS) and virtual machine overhead memory
Total capacity	MB	absolute	average	Total amount of memory reservation used by and available for powered-on virtual machines and vSphere services on the host
Unreserved	KB	absolute	none	Amount of memory that is unreserved
Unreserved	KB	absolute	average	Amount of memory that is unreserved
Unreserved	KB	absolute	minimum	Amount of memory that is unreserved
Unreserved	KB	absolute	maximum	Amount of memory that is unreserved
Usage	Percent	absolute	none	Memory usage as percentage of total configured or available memory
Usage	Percent	absolute	average	Memory usage as percentage of total configured or available memory
Usage	Percent	absolute	minimum	Memory usage as percentage of total configured or available memory
Usage	Percent	absolute	maximum	Memory usage as percentage of total configured or available memory
Used by VMkernel	KB	absolute	none	Amount of machine memory used by VMkernel for core functionality, such as device drivers and other internal uses
Used by VMkernel	KB	absolute	average	Amount of machine memory used by VMkernel for core functionality, such as device drivers and other internal uses
Used by VMkernel	KB	absolute	maximum	Amount of machine memory used by VMkernel for core functionality, such as device drivers and other internal uses
Used by VMkernel	KB	absolute	minimum	Amount of machine memory used by VMkernel for core functionality, such as device drivers and other internal uses
Worst case allocation	MB	absolute	latest	Memory allocation as calculated by the VMkernel scheduler based on current estimated demand and reservation, limit, and shares policies set for all virtual machines and resource pools in the host or cluster
Zero	KB	absolute	none	Memory that contains 0s only
Zero	KB	absolute	average	Memory that contains 0s only
Zero	KB	absolute	minimum	Memory that contains 0s only
Zero	KB	absolute	maximum	Memory that contains 0s only
Zipped memory	KB	absolute	latest	Memory (KB) zipped
swapIn	KB	absolute	none	swapIn
swapIn	KB	absolute	average	swapIn
swapIn	KB	absolute	minimum	swapIn
swapIn	KB	absolute	maximum	swapIn
swapOut	KB	absolute	none	swapOut
swapOut	KB	absolute	average	swapOut
swapOut	KB	absolute	minimum	swapOut
swapOut	KB	absolute	maximum	swapOut
userworld	KB	absolute	average	userworld
userworld	KB	absolute	average	userworld
vm	KB	absolute	average	vm
vm	KB	absolute	average	vm
vmOvhd	KB	absolute	average	vmOvhd
vmOvrhd	KB	absolute	average	vmOvrhd
vmkOvrhd	KB	absolute	average	vmkOvrhd
vmkOvrhd	KB	absolute	average	vmkOvrhd

Network

Metric	Unit	Stats Type	Rollup	Description
Broadcast receives	Number	delta	summation	Number of broadcast packets received during the sampling interval
Broadcast transmits	Number	delta	summation	Number of broadcast packets transmitted during the sampling interval
Data receive rate	KBps	rate	average	Average rate at which data was received during the interval
Data receive rate	KBps	rate	average	Average amount of data received per second
Data transmit rate	KBps	rate	average	Average rate at which data was transmitted during the interval
Data transmit rate	KBps	rate	average	Average amount of data transmitted per second
Multicast receives	Number	delta	summation	Number of multicast packets received during the sampling interval
Multicast transmits	Number	delta	summation	Number of multicast packets transmitted during the sampling interval
Packet receive errors	Number	delta	summation	Number of packets with errors received during the sampling interval
Packet transmit errors	Number	delta	summation	Number of packets with errors transmitted during the sampling interval
Packets received	Number	delta	summation	Number of packets received during the interval
Packets transmitted	Number	delta	summation	Number of packets transmitted during the interval
Receive packets dropped	Number	delta	summation	Number of receives dropped
Transmit packets dropped	Number	delta	summation	Number of transmits dropped
Unknown protocol frames	Number	delta	summation	Number of frames with unknown protocol received during the sampling interval
Usage	KBps	rate	none	Network utilization (combined transmit-rates and receive-rates) during the interval
Usage	KBps	rate	average	Network utilization (combined transmit-rates and receive-rates) during the interval
Usage	KBps	rate	minimum	Network utilization (combined transmit-rates and receive-rates) during the interval
Usage	KBps	rate	maximum	Network utilization (combined transmit-rates and receive-rates) during the interval
pNic Packets Received and Transmitted per Second	Number	rate	average	Average rate of packets received and transmitted per second
pNic Throughput Provisioned	KBps	absolute	average	Provisioned pNic I/O Throughput
pNic Throughput Usable	KBps	absolute	average	Usable pNic I/O Throughput
pNic Throughput Usage for FT	KBps	rate	average	Average pNic I/O rate for FT
pNic Throughput Usage for NFS	KBps	rate	average	Average pNic I/O rate for NFS
pNic Throughput Usage for VMs	KBps	rate	average	Average pNic I/O rate for VMs
pNic Throughput Usage for VR	KBps	rate	average	Average pNic I/O rate for VR
pNic Throughput Usage for iSCSI	KBps	rate	average	Average pNic I/O rate for iSCSI
pNic Throughput Usage for vMotion	KBps	rate	average	Average pNic I/O rate for vMotion
vNic Throughput Contention	Number	delta	summation	Count of vNic packet drops
vNic Throughput Usage	KBps	rate	average	Average vNic I/O rate

Power

Metric	Unit	Stats Type	Rollup	Description
Cap	Watt	absolute	average	Maximum allowed power usage
Energy usage	Joule	delta	summation	Total energy used since last stats reset
Host Power Capacity Provisioned	Percent	absolute	average	Current power usage as a percentage of maximum allowed power
Host Power Capacity Usable	Watt	absolute	average	Current maximum allowed power usage
Power Capacity Usage	Watt	absolute	average	Current power usage
Usage	Watt	rate	average	Current power usage

Resource group CPU

Metric	Unit	Stats Type	Rollup	Description
Active (1 min. average)	Percent	absolute	latest	CPU active average over 1 minute
Active (1 min. peak)	Percent	absolute	latest	CPU active peak over 1 minute
Active (15 min. average)	Percent	absolute	latest	CPU active average over 15 minutes
Active (15 min. peak)	Percent	absolute	latest	CPU active peak over 15 minutes
Active (5 min. average)	Percent	absolute	latest	CPU active average over 5 minutes
Active (5 min. peak)	Percent	absolute	latest	CPU active peak over 5 minutes
Group CPU sample count	Number	absolute	latest	Group CPU sample count
Group CPU sample period	Millisecond	absolute	latest	Group CPU sample period
Running (1 min. average)	Percent	absolute	latest	CPU running average over 1 minute
Running (1 min. peak)	Percent	absolute	latest	CPU running peak over 1 minute
Running (15 min. average)	Percent	absolute	latest	CPU running average over 15 minutes
Running (15 min. peak)	Percent	absolute	latest	CPU running peak over 15 minutes
Running (5 min. average)	Percent	absolute	latest	CPU running average over 5 minutes
Running (5 min. peak)	Percent	absolute	latest	CPU running peak over 5 minutes
Throttled (1 min. average)	Percent	absolute	latest	Amount of CPU resources over the limit that were refused, average over 1 minute
Throttled (15 min. average)	Percent	absolute	latest	Amount of CPU resources over the limit that were refused, average over 15 minutes
Throttled (5 min. average)	Percent	absolute	latest	Amount of CPU resources over the limit that were refused, average over 5 minutes

Storage adapter

Metric	Unit	Stats Type	Rollup	Description
Average commands issued per second	Number	rate	average	Average number of commands issued per second by the storage adapter during the collection interval
Average read requests per second	Number	rate	average	Average number of read commands issued per second by the storage adapter during the collection interval
Average write requests per second	Number	rate	average	Average number of write commands issued per second by the storage adapter during the collection interval
Highest latency	Millisecond	absolute	latest	Highest latency value across all storage adapters used by the host
Read latency	Millisecond	absolute	average	The average time a read by the storage adapter takes
Read rate	KBps	rate	average	Rate of reading data by the storage adapter
Storage Adapter Number Queued	Number	absolute	average	The current number of I/Os that are waiting to be issued
Storage Adapter Outstanding I/Os	Percent	absolute	average	The percent of I/Os that have been issued but have not yet completed
Storage Adapter Outstanding I/Os	Number	absolute	average	The number of I/Os that have been issued but have not yet completed
Storage Adapter Queue Command Latency	Millisecond	absolute	average	Average amount of time spent in the VMkernel queue, per SCSI command, during the collection interval
Storage Adapter Queue Depth	Number	absolute	average	The maximum number of I/Os that can be outstanding at a given time
Storage Adapter Throughput Contention	Millisecond	absolute	average	Average amount of time for an I/O operation to complete
Storage Adapter Throughput Usage	KBps	rate	average	The storage adapter's I/O rate
Write latency	Millisecond	absolute	average	The average time a write by the storage adapter takes
Write rate	KBps	rate	average	Rate of writing data by the storage adapter

Storage path

Metric	Unit	Stats Type	Rollup	Description
Average commands issued per second	Number	rate	average	Average number of commands issued per second on the storage path during the collection interval
Average read requests per second	Number	rate	average	Average number of read commands issued per second on the storage path during the collection interval
Average write requests per second	Number	rate	average	Average number of write commands issued per second on the storage path during the collection interval
Highest latency	Millisecond	absolute	latest	Highest latency value across all storage paths used by the host
Read latency	Millisecond	absolute	average	The average time a read issued on the storage path takes
Read rate	KBps	rate	average	Rate of reading data on the storage path
Storage Path Bus Resets	Number	delta	summation	Number of SCSI-bus reset commands issued during the collection interval
Storage Path Command Aborts	Number	delta	summation	Number of SCSI commands aborted during the collection interval
Storage Path Throughput Contention	Millisecond	absolute	average	Average amount of time for an I/O operation to complete
Storage Path Throughput Usage	KBps	rate	average	Storage path I/O rate
Write latency	Millisecond	absolute	average	The average time a write issued on the storage path takes
Write rate	KBps	rate	average	Rate of writing data on the storage path

System

Metric	Unit	Stats Type	Rollup	Description
Disk usage	Percent	absolute	latest	Amount of disk space usage for each mount point
Heartbeat	Number	delta	summation	Number of heartbeats issued per virtual machine during the interval
Heartbeat	Number	absolute	latest	Number of heartbeats issued per virtual machine during the interval
OS Uptime	Second	absolute	latest	Total time elapsed, in seconds, since last operating system boot-up
Resource CPU active (1 min. average)	Percent	absolute	latest	CPU active average over 1 minute of the system resource group
Resource CPU active (5 min. average)	Percent	absolute	latest	CPU active average over 5 minutes of the system resource group
Resource CPU allocation maximum, in MHz	MHz	absolute	latest	CPU allocation limit, in MHz, of the system resource group
Resource CPU allocation minimum, in MHz	MHz	absolute	latest	CPU allocation reservation, in MHz, of the system resource group
Resource CPU allocation shares	Number	absolute	latest	CPU allocation shares of the system resource group
Resource CPU maximum limited (1 min.)	Percent	absolute	latest	CPU maximum limited over 1 minute of the system resource group
Resource CPU maximum limited (5 min.)	Percent	absolute	latest	CPU maximum limited over 5 minutes of the system resource group
Resource CPU running (1 min. average)	Percent	absolute	latest	CPU running average over 1 minute of the system resource group
Resource CPU running (5 min. average)	Percent	absolute	latest	CPU running average over 5 minutes of the system resource group
Resource CPU usage (Average)	MHz	rate	average	Amount of CPU used by the Service Console and other applications during the interval
Resource CPU usage (Maximum)	MHz	rate	maximum	Amount of CPU used by the Service Console and other applications during the interval
Resource CPU usage (Minimum)	MHz	rate	minimum	Amount of CPU used by the Service Console and other applications during the interval
Resource CPU usage (None)	MHz	rate	none	Amount of CPU used by the Service Console and other applications during the interval
Resource memory allocation maximum, in KB	KB	absolute	latest	Memory allocation limit, in KB, of the system resource group
Resource memory allocation minimum, in KB	KB	absolute	latest	Memory allocation reservation, in KB, of the system resource group
Resource memory allocation shares	Number	absolute	latest	Memory allocation shares of the system resource group
Resource memory mapped	KB	absolute	latest	Memory mapped by the system resource group
Resource memory overhead	KB	absolute	latest	Overhead memory consumed by the system resource group
Resource memory share saved	KB	absolute	latest	Memory saved due to sharing by the system resource group
Resource memory shared	KB	absolute	latest	Memory shared by the system resource group
Resource memory swapped	KB	absolute	latest	Memory swapped out by the system resource group
Resource memory touched	KB	absolute	latest	Memory touched by the system resource group
Resource memory zero	KB	absolute	latest	Zero filled memory used by the system resource group
Uptime	Second	absolute	latest	Total time elapsed, in seconds, since last system startup

Virtual disk

Metric	Unit	Stats Type	Rollup	Description
Average number of outstanding read requests	Number	absolute	latest	Average number of outstanding read requests to the virtual disk during the collection interval
Average number of outstanding write requests	Number	absolute	latest	Average number of outstanding write requests to the virtual disk during the collection interval
Average read requests per second	Number	rate	average	Average number of read commands issued per second to the virtual disk during the collection interval
Average write requests per second	Number	rate	average	Average number of write commands issued per second to the virtual disk during the collection interval
Read latency	Millisecond	absolute	average	The average time a read from the virtual disk takes
Read rate	KBps	rate	average	Rate of reading data from the virtual disk
Read workload metric	Number	absolute	latest	Storage DRS virtual disk metric for the read workload model
Virtual Disk Throughput Contention	Millisecond	absolute	average	Average amount of time for an I/O operation to complete
Virtual Disk Throughput Usage	KBps	rate	average	Virtual disk I/O rate
Write latency	Millisecond	absolute	average	The average time a write to the virtual disk takes
Write rate	KBps	rate	average	Rate of writing data to the virtual disk
Write workload metric	Number	absolute	latest	Storage DRS virtual disk metric for the write workload model
busResets	Number	delta	summation	busResets
commandsAborted	Number	delta	summation	commandsAborted

Virtual machine operations

Metric	Unit	Stats Type	Rollup	Description
Storage vMotion count	Number	absolute	latest	Number of migrations with Storage vMotion (datastore change operations for powered-on VMs)
VM clone count	Number	absolute	latest	Number of virtual machine clone operations
VM create count	Number	absolute	latest	Number of virtual machine create operations
VM datastore change count (non-powered-on VMs)	Number	absolute	latest	Number of datastore change operations for powered-off and suspended virtual machines
VM delete count	Number	absolute	latest	Number of virtual machine delete operations
VM guest reboot count	Number	absolute	latest	Number of virtual machine guest reboot operations
VM guest shutdown count	Number	absolute	latest	Number of virtual machine guest shutdown operations
VM host and datastore change count (non-powered-on VMs)	Number	absolute	latest	Number of host and datastore change operations for powered-off and suspended virtual machines
VM host change count (non-powered-on VMs)	Number	absolute	latest	Number of host change operations for powered-off and suspended VMs
VM power off count	Number	absolute	latest	Number of virtual machine power off operations
VM power on count	Number	absolute	latest	Number of virtual machine power on operations
VM reconfigure count	Number	absolute	latest	Number of virtual machine reconfigure operations
VM register count	Number	absolute	latest	Number of virtual machine register operations
VM reset count	Number	absolute	latest	Number of virtual machine reset operations
VM standby guest count	Number	absolute	latest	Number of virtual machine standby guest operations
VM suspend count	Number	absolute	latest	Number of virtual machine suspend operations
VM template deploy count	Number	absolute	latest	Number of virtual machine template deploy operations
VM unregister count	Number	absolute	latest	Number of virtual machine unregister operations
vMotion count	Number	absolute	latest	Number of migrations with vMotion (host change operations for powered-on VMs)

vCenter debugging information

Metric	Unit	Stats Type	Rollup	Description
Activation count	Number	absolute	maximum	Activation operations in vCenter
Activation count	Number	absolute	minimum	Activation operations in vCenter
Activation count	Number	absolute	summation	Activation operations in vCenter
Activation latency	Millisecond	absolute	maximum	The latency of an activation operation in vCenter
Activation latency	Millisecond	absolute	minimum	The latency of an activation operation in vCenter
Activation latency	Millisecond	absolute	summation	The latency of an activation operation in vCenter
Host sync count	Number	absolute	maximum	The number of host sync operations in vCenter
Host sync count	Number	absolute	minimum	The number of host sync operations in vCenter
Host sync count	Number	absolute	summation	The number of host sync operations in vCenter
Host sync latency	Millisecond	absolute	maximum	The latency of a host sync operation in vCenter
Host sync latency	Millisecond	absolute	minimum	The latency of a host sync operation in vCenter
Host sync latency	Millisecond	absolute	summation	The latency of a host sync operation in vCenter
Inventory statistics	Number	absolute	maximum	vCenter inventory statistics
Inventory statistics	Number	absolute	minimum	vCenter inventory statistics
Inventory statistics	Number	absolute	summation	vCenter inventory statistics
Locking statistics	Number	absolute	maximum	vCenter locking statistics
Locking statistics	Number	absolute	minimum	vCenter locking statistics
Locking statistics	Number	absolute	summation	vCenter locking statistics
Managed object reference statistics	Number	absolute	maximum	Managed object reference counts in vCenter
Managed object reference statistics	Number	absolute	minimum	Managed object reference counts in vCenter
Managed object reference statistics	Number	absolute	summation	Managed object reference counts in vCenter
Miscellaneous	Number	absolute	maximum	Miscellaneous statistics
Miscellaneous	Number	absolute	minimum	Miscellaneous statistics
Miscellaneous	Number	absolute	summation	Miscellaneous statistics
Scoreboard statistics	Number	absolute	maximum	Object counts in vCenter
Scoreboard statistics	Number	absolute	minimum	Object counts in vCenter
Scoreboard statistics	Number	absolute	summation	Object counts in vCenter
Session statistics	Number	absolute	maximum	The statistics of client sessions connected to vCenter
Session statistics	Number	absolute	minimum	The statistics of client sessions connected to vCenter
Session statistics	Number	absolute	summation	The statistics of client sessions connected to vCenter
System statistics	Number	absolute	maximum	The statistics of vCenter as a running system such as thread statistics and heap statistics
System statistics	Number	absolute	minimum	The statistics of vCenter as a running system such as thread statistics and heap statistics
System statistics	Number	absolute	summation	The statistics of vCenter as a running system such as thread statistics and heap statistics
vCenter LRO statistics	Number	absolute	maximum	vCenter LRO statistics
vCenter LRO statistics	Number	absolute	minimum	vCenter LRO statistics
vCenter LRO statistics	Number	absolute	summation	vCenter LRO statistics
vCenter service statistics	Number	absolute	maximum	vCenter service statistics such as events, alarms, and tasks
vCenter service statistics	Number	absolute	minimum	vCenter service statistics such as events, alarms, and tasks
vCenter service statistics	Number	absolute	summation	vCenter service statistics such as events, alarms, and tasks

vCenter resource usage information

Metric	Unit	Stats Type	Rollup	Description
CPU privileged	Percent	rate	average	CPU used by vCenter in privileged mode
CPU process	Percent	rate	average	Total CPU used by vCenter
CPU queue length	Number	absolute	average	Processor queue length on the system where vCenter is running
CPU system	Percent	rate	average	Total system CPU used on the system where vCenter in running
CPU user	Percent	rate	average	CPU used by vCenter in user mode
Context switch rate	Number	rate	average	Number of context switches per second on the system where vCenter is running
Disk bytes read rate	Number	rate	average	Number of bytes read from the disk per second on the system where vCenter is running
Disk bytes written rate	Number	rate	average	Number of bytes written to the disk per second on the system where vCenter is running
Disk queue length	Number	absolute	average	Disk queue length on the system where vCenter is running
Disk read rate	Number	rate	average	Number of disk reads per second on the system where vCenter is running
Disk write rate	Number	rate	average	Number of disk writes per second on the system where vCenter is running
Network queue length	Number	absolute	average	Network queue length on the system where vCenter is running
Network usage	Percent	rate	average	Total network bytes received and sent per second on the system where vCenter is running
Page fault rate	Number	rate	average	Number of page faults per second on the system where vCenter is running
Physical memory	KB	absolute	average	Physical memory used by vCenter
Pool non-paged bytes	KB	absolute	average	Memory pooled for non-paged bytes on the system where vCenter is running
Pool paged bytes	KB	absolute	average	Memory pooled for paged bytes on the system where vCenter is running
Process handles	Number	absolute	average	Handles used by vCenter
Process threads	Number	absolute	average	Number of threads used by vCenter
Received packet rate	Number	rate	average	Rate of the number of total packets received per second on the system where vCenter is running
Sent packet rate	Number	rate	average	Number of total packets sent per second on the system where vCenter is running
System call rate	Number	rate	average	Number of systems calls made per second on the system where vCenter is running
System threads	Number	absolute	average	Number of threads on the system where vCenter is running
Total packet rate	Number	rate	average	Number of total packets sent and received per second on the system where vCenter is running
Virtual memory	KB	absolute	average	Virtual memory used by vCenter

vSphere Replication

Metric	Unit	Stats Type	Rollup	Description
Replication Data Receive Rate	KBps	rate	average	Average amount of data received per second
Replication Data Transmit Rate	KBps	rate	average	Average amount of data transmitted per second
vSphere Replication VM Count	Number	absolute	average	Current Number of Replicated VMs

Exploring VMware vSphere PowerCLI with F#

2013-02-19T06:00:00.000-08:00

I was recently asked to vet some VMware cluster capacity numbers. It seemed like a task that might be repeated in the future and I really hate to manually transcribe the data and compare them. So that means I need to write a script to automate it. Fortunately, VMware has vSphere PowerCLI for this job. Unfortunately, the documentation for it was rather sparse. I also looked at the book VMware vSphere PowerCLI Reference: Automating vSphere Administration, but really did not want to script in PowerShell simply because I'm not that familiar with PowerShell. However, PowerCLI is accessible from .NET also, so I can write my script in F#.

The following lines of script simply drills down from Datacenter object to the VirtualMachine object. Once you grab all the objects, you can explore the properties of each object.

#r @"C:\pkg\VMware\Infrastructure\vSphere PowerCLI\VMware.Vim.dll"

open System
open VMware.Vim
open System.Collections.Specialized

let serviceUrl = "https://myVSphereHost/sdk"
let userId = "someUserId"
let password="somePassword"

let client = new VimClient()
let service = client.Connect(serviceUrl)

// Must login to do anything - if you are getting null values, it means the session automatically timed out
client.Login(userId,password)

// Let us get all the datacenters
let dataCenters = client.FindEntityViews(typeof<Datacenter>,null,null,null)

// Drill down into the first datacenter
let dc = dataCenters |> Seq.cast<Datacenter> |> Seq.head

// Get a cluster
let cluster = client.GetView(dc.Parent,null) :?> ClusterComputeResource

// Get the first host in the cluster
let host = client.GetView(cluster.Host |> Seq.head,null) :?> HostSystem

// Get the first VM on the physical host
let vm = client.GetView(host.Vm |> Seq.head,null) :?> VirtualMachine

Let's do something interesting with PowerCLI. In the following scripts, I wanted to grab the capacity information at the VMware cluster level and the combine allocation/utilization info of all the virtual machines hosted on the VMware cluster.


// Utility function to help use get vSphere Entities
let getEntityViews viewType searchParams =
    match searchParams with
    | Some(searchParams0) ->
        let filters = new NameValueCollection()
        searchParams0 |> Seq.iter (fun (k,v) -> filters.Add(k,v))
        client.FindEntityViews(viewType,null,filters,null) 

    | None ->
        client.FindEntityViews(viewType,null,null,null) 

// Get vSphere Entities with specific properties (reduces returned data)
// Don't know how to define a function with multiple arities in F# - clumsy workaround
let getEntityViews2 viewType searchParams props =
    match searchParams with
    | Some(searchParams0) ->
        let filters = new NameValueCollection()
        searchParams0 |> Seq.iter (fun (k,v) -> filters.Add(k,v))
        client.FindEntityViews(viewType,null,filters,props) 
    | None ->
        client.FindEntityViews(viewType,null,null,props) 


// Get Cluster usage summary!
let getUsageSummary clusterName =

    let clusterProps = [|"Summary"; "Host"|]
    let hostProps = [|"Vm";"Name";"Hardware";"Runtime"|]
    let vmProps = [|"Name";"Config";"Runtime";"Summary";"ResourceConfig"|]

    let toMB memoryInBytes = memoryInBytes / (1024L*1024L)

    // Get cluster - Expect only one result
    printfn "Getting cluster data..."
    let cluster = 
        let filters = Some([("name",clusterName)])
        getEntityViews2 typeof<ClusterComputeResource> filters clusterProps
            |> Seq.cast<ClusterComputeResource>
            |> Seq.head


    // Get all Hosts for this cluster
    printfn "Getting host list..."
    let hostList =
        cluster.Host
        |> Seq.map(fun moRef -> client.GetView(moRef,hostProps))
        |> Seq.cast<HostSystem>
        |> Seq.cache


    printfn "Getting VM list..."
    let vmList =
        hostList
        |> Seq.map (fun host -> host.Vm)
        |> Seq.concat
        |> Seq.map (fun moRef -> client.GetView(moRef,vmProps))
        |> Seq.cast<VirtualMachine>
        |> Seq.cache

    let clusterCores = cluster.Summary.NumCpuCores
    let clusterCPU = cluster.Summary.TotalCpu
    let clusterMemory = cluster.Summary.TotalMemory


    // Utility function to get summation results from selected fields
    let inline total (extractor:VirtualMachine -> Nullable<'b>) =
        vmList
        |> Seq.map extractor
        |> Seq.map (fun x -> x.GetValueOrDefault())
        |> Seq.sum


    printfn "Getting Cluster Summary Info..."
    let cpuUsed        = total (fun vm -> vm.Summary.Config.NumCpu)
    let memoryUsed     = total (fun vm -> vm.Summary.Config.MemorySizeMB)
    let cpuReserved    = total (fun vm -> vm.Summary.Config.CpuReservation)
    let memoryReserved = total (fun vm -> vm.Summary.Config.MemoryReservation)
    let maxMemoryUsage = total (fun vm -> vm.Runtime.MaxMemoryUsage)
    let maxCpuUsage    = total (fun vm -> vm.Runtime.MaxCpuUsage)

    let cpuReservation    = total (fun vm -> vm.ResourceConfig.CpuAllocation.Reservation) 
    let cpuLimit          = total (fun vm -> vm.ResourceConfig.CpuAllocation.Limit) 
    let memoryReservation = total (fun vm -> vm.ResourceConfig.MemoryAllocation.Reservation)
    let memoryLimit       = total (fun vm -> vm.ResourceConfig.MemoryAllocation.Limit) 

    
    printfn "Cluster Name               : %s" clusterName
    printfn "Number of Hosts in Cluster : %i" (Seq.length hostList)
    printfn "Number of VMs in Cluster   : %i" (Seq.length vmList)
    printfn "Cluster Total Cores        : %i" clusterCores
    printfn "Cluster Total CPU (MHz)    : %i" clusterCPU
    printfn "Cluster Total Memory       : %i" clusterMemory
    printfn "Cluster Total Memory (MB)  : %i" (toMB clusterMemory)
    printfn "CPU Used by VMs            : %i" cpuUsed
    printfn "Memory Used by VMs         : %i" memoryUsed
    printfn "CPU Reserved by VMs        : %i" cpuReserved
    printfn "Memory Reserved by VMs     : %i" memoryReserved
    printfn "Max Memory Usage by VMs    : %i" maxMemoryUsage
    printfn "Max CPU Usage by VMs (MHz) : %i" maxCpuUsage
    printfn "Total Allocated CPU Reservations    : %i" cpuReservation
    printfn "Total Allocated CPU Limits          : %i" cpuLimit
    printfn "Total Allocated Memory Reservations : %i" memoryReservation
    printfn "Total Allocated Memory Limits       : %i" memoryLimit
    printfn "Memory Used / Cluster Total Memory  : %f" (double(memoryUsed) / double(toMB clusterMemory))
    printfn "vCPU Allocated / Cluster Total Cores : %f" (float(cpuUsed) / float(clusterCores))    
    printfn "Done!"

// Invoke getUsageSummary to get the summary info on my cluster
getUsageSummary "MyTestCluster"

Sample results:

Number of Hosts in Cluster : 5
Number of VMs in Cluster   : 10
Cluster Total Cores        : 120
Cluster Total CPU (MHz)    : 276000
Cluster Total Memory       : 1374369136640
Cluster Total Memory (MB)  : 1310700
CPU Used by VMs            : 38
Memory Used by VMs         : 253952
CPU Reserved by VMs        : 0
Memory Reserved by VMs     : 0
Max Memory Usage by VMs    : 253952
Max CPU Usage by VMs (MHz) : 87400
Total Allocated CPU Reservations    : 0
Total Allocated CPU Limits          : -10
Total Allocated Memory Reservations : 0
Total Allocated Memory Limits       : -10
Memory Used / Cluster Total Memory  : 0.193753
vCPU Allocated / Cluster Total Cores : 0.316667

I wish VMware had PowerCLI class documentation similar to those for the .NET library hosted on MSDN. If VMware does have those documentation, I can't seem to find them. Lack of documentation has forced me to interactively explore the PowerCLI with F#. Thankfully, I can do this in F# and shudder at the thought of exploring PowerCLI in C#.

Clojure Script to Convert Java GC log to CSV Format

2013-02-08T06:00:00.000-08:00

My previous blog post showed how to use R to perform Java GC log analysis. Ajit Joglekar asked that I post the Clojure script. Here's the Clojure script that I used to convert GC log to CSV format. I have not tested this with a wide variety of GC logs but I do know that this script will not work for CMS or G1GC gc logs. Modify minor-gc-pattern and full-gc-pattern to match the gc log output as needed.


(use '[clojure.string :only (join)])  

; Match for pause time "0.1566980 secs]"
(def ^:constant pause-time "([\\d\\.]+) secs\\]")

; Match for Java heap space stat "524288K->32124K(2009792K)"
(def ^:constant space "(\\d+)K->(\\d+)K\\((\\d+)K\\)")

; Match for Execution stat "[Times: user=0.24 sys=0.06, real=0.16 secs]"
(def ^:constant exec-stat " \\[Times: user=([\\d\\.]+) sys=([\\d\\.]+), real=([\\d\\.]+) secs\\]")

; Example Minor GC entry   
; 212.785: [GC [PSYoungGen: 524288K->32124K(611648K)] 524288K->32124K(2009792K), 0.1566980 secs] [Times: user=0.24 sys=0.06, real=0.16 secs] 
; Define regex pattern to parse young gen GC event
(defn minor-gc-pattern []
  (let [timestamp  "([\\d\\.]+): \\[GC .*\\[PS.*: "
        young-gen   (str space "] ")
        heap        (str space ", ")]
    (re-pattern (str timestamp young-gen  heap pause-time exec-stat))))                

; Example Full GC entry   
;"43587.513: [Full GC (System) [PSYoungGen: 964K->0K(598912K)] [PSOldGen: 142673K->120674K(1398144K)] 143637K->120674K(1997056K) [PSPermGen: 82179K->82179K(147520K)], 0.7556570 secs] [Times: user=0.76 sys=0.00, real=0.77 secs]"
; Define the regex pattern to parse each line in gc log
(defn full-gc-pattern []
    (let [timestamp   "([\\d\\.]+): \\[Full.*"
          young-gen  (str ": " space "]")
          old-gen    (str " \\[\\w+: " space "\\] ")
          perm-gen   (str " \\[\\w+: "space "\\], ")
          heap       space]
         (re-pattern (str timestamp 
                          young-gen 
                          old-gen 
                          heap 
                          perm-gen 
                          pause-time 
                          exec-stat))))                
 
; Variable definitions (for both process-full-gc & process-minor-gc
;     ts - timestamp (in seconds)
;     ys - YoungGen space starting heap size (in KB)
;     ye - YoungGen space ending heap size (in KB)
;     ym - YoungGen space max heap size (in KB)
;     os - OldGen space starting heap size (in KB)
;     oe - OldGen space ending heap size (in KB)
;     om - OldGen space max heap size (in KB)
;     hs - Total heap space starting heap size (in KB)
;     he - Total heap space ending heap size (in KB)
;     hm - Total heap space max heap size (in KB)
;     pt - GC Pause Time (in seconds)
;     ps - PermGen space starting heap size (in KB)
;     pe - PermGen space ending heap size (in KB)
;     pm - PermGen space max heap size (in KB)
;     ut - User Time (in seconds)
;     kt - Kernel Time (in seconds)
;     rt - Real Time (in seconds)
(defn process-full-gc [entry]
  (let [[a ts ys ye ym os oe om hs he hm ps pe pm pt ut kt rt & e] entry]
    (join \, [ts "full" pt 
              ys ye ym 
              hs he hm 
              ut kt rt 
              os oe om 
              ps pe pm])))
   
(defn process-minor-gc [entry]
  (let [[a ts ys ye ym hs he hm pt ut kt rt & e] entry]
    (join \, [ts "minor" pt 
              ys ye ym 
              hs he hm 
              ut kt rt])))

(def headers (join \,  ["timestamp" "gc.type" "pause.time"
                        "young.start" "young.end" "young.max"
                        "heap.start" "heap.end" "heap.max"
                        "time.user" "time.sys" "time.real"
                        "old.start" "old.end" "old.max"
                        "perm.start" "perm.end" "perm.max"]))    
       

(defn process-gc-file [infile outfile]
  (let [gcdata (line-seq (clojure.java.io/reader (clojure.java.io/file infile)))]
    (with-open [w (clojure.java.io/writer outfile)]
      (let [writeln (fn [x] (.write w (str x "\n")))]
        (writeln headers)
        (doseq [line gcdata]
          (let [minor-gc (re-seq (minor-gc-pattern) line)
                full-gc  (re-seq (full-gc-pattern) line)]
            (when-not (nil? full-gc) 
              (writeln (process-full-gc (first full-gc))))
            (when-not (nil? minor-gc) 
              (writeln (process-minor-gc (first minor-gc))))))))))

;-----------------------------------------------------------------------   
; Convert Java GC log csv format
;-----------------------------------------------------------------------    
(process-gc-file "gc.log" "data.csv")

You can download the script directly here : gcanalysis.clj.

Java Garbage Collection Log Analysis with R

2013-02-04T05:00:00.000-08:00

In the past, I generally perform Java GC log analysis using a combination of scripts such as the one I have written in my previous blog post for parsing the gc log and transforming it into csv format, GCHisto tool as mentioned in the book Java Performance, and Microsoft Excel for mostly plotting purposes. However, as I find myself doing more and more GC analysis for troubleshooting performance issues, I'm finding that these tools are not flexible enough for my purposes.

I heard about The R Project from one my coworker. In addition, I see R gaining traction in the industry; I decided to try using R for GC analysis. This blog post is to document some of my experiences with using R for GC analysis.

There are some posts in the blogosphere that seem to suggest that it is difficult to learn R. I did not find that to be the case. Whether that's due to knowing functional programming or past experience with Matlab, I don't know. I did purchase a bunch of books on R to learn it including R in a Nutshell, R by Example, R in Action, Introductory Statistics with R. I got the most mileage with the last book.

In working with R, I still would use a script to parse out raw GC log and converted into csv file. So far, most of my GC analysis are on non-CMS GC logs so this blog post is restricted to discussing analysis of mostly parallel GC logs. In order to be able to better read my R scripts, I have changed my script to output the csv file with the following headers:

timestamp,gc.type,young.start,young.end,young.max,heap.start,heap.end,heap.max,pause.time,old.start,old.end,old.max,perm.start,perm.end,perm.max,time.user,time.sys,time.real

I can load the csv file in R as follows:

data.file <- "gc.log.csv"
data.gc <- read.csv(data.file,sep=',')

I created the following utility functions because I find that I often want to work in MB or GB for heap space rather than in KB. Also, I want to work with timestamp in hours rather than seconds.

sec.to.hours  <- function(x) x/(60.0*60.0)
kb.to.mb <- function(x) x/1024.0
kb.to.gb <- function(x) x/1024.0^2

Printing GC Pause Time Statistics

With this, I can dump the GC pause time statistical information as follows:

# Dump mean, standard deviation, GC counts for GC pause time
attach(data.gc)
 by(pause.time,gc.type,mean)
 by(pause.time,gc.type,sd)
 by(pause.time,gc.type,length)

 # GC Frequency statistics
 summary(diff(timestamp))

 # Time spent in GC
 sum(pause.time)

 # Percentage of time spent in GC
 sum(pause.time)*100.0/max(timestamp)
 
detach(data.gc)

For the above code, a sample output would be:

# by(pause.time,gc.type,mean) output 
# GC Average Pause Time
gc.type: full
[1] 2.125116
-------------------------------------------------------------- 
gc.type: minor
[1] 0.06472383

# by(pause.time,gc.type,sd) output 
# GC pause time distribution
gc.type: full
[1] 1.01972
-------------------------------------------------------------- 
gc.type: minor
[1] 0.02684703

# by(pause.time,gc.type,length) output 
# GC count by type
gc.type: full
[1] 7
-------------------------------------------------------------- 
gc.type: minor
[1] 2172

# summary(diff(timestamp)) output 
# GC Frequency statistics - (e.g. GC occurs on average every 84 seconds)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   0.08   22.44   71.34   84.83  143.10  252.90 
   
# sum(pause.time) output
# Total time spent in GC   
[1] 155.456

# sum(pause.time)*100.0/max(timestamp) output 
# Percentage of time spent in GC
[1] 0.08412511

Plotting GC Timeline

Here's how to plot GC Timeline:

with(data.gc, {
 plot.title <-"GC Timeline"
 gc.minor <- pause.time[gc.type == "minor"]
 gc.full <- pause.time[gc.type == "full"]

 # Plot bars for full GC pause time 
 plot(sec.to.hours(timestamp[gc.type == "full"]),
   gc.full*1000.0,
   xlab="Elapsed Time (hrs)",
   ylab="GC Pause Time Time(ms)",type='h',
   main=plot.title,col="red",ylim=c(0,1000),xlim=c(0,18))
   
 # Plot bars for minor GC pause time
 lines(sec.to.hours(timestamp[gc.type == "minor"]),
       gc.minor*1000.0,col="dark green",type="h")

 # Set tick marks for every hour
 axis(1,seq(0,18,1))
 grid()
})

A sample output of the above code would be:

Plotting GC Pause Time Distribution

To plot GC pause time distribution:

# Plot GC Pause Time Distribution
plot.title <- "GC Pause Time Distribution"
with(data.gc, {
 hist(pause.time,breaks=100, 
   xlab="GC Pause Time(sec)",
   xlim=c(0,1.0),
   main=plot.title)
})

Plotting GC Pause Time Distribution as a Boxplot

Alternatively, you can view this as a boxplot:

plot.title <- "GC Pause Time Distribution as Boxplot"
with(data.gc, {
 boxplot(pause.time ~ gc.type,main=plot.title)
})

Plotting Promoted Heap Memory

One particular case, I was analyzing a situation where minor GC pause time was unusually large. My original hypothesis is that a lot of memory was being promoted into old generation space. Therefore, I wanted a plot of heap memory promoted into old gen space:

plot.title <- "Heap Promoted into Old Gen Space"
with(data.gc, {
 old.end <- kb.to.mb(heap.end - young.end)
 x <- sec.to.hours(timestamp)
 # If there is a full GC, promoted heap size would go negative
 y <- diff(old.end)
 plot(x[-1],y,
   xlab="Elapsed Time (hrs)",
   ylab="Promoted Heap (MB)",type='h',
   main=plot.title)
 grid()
})

In this particular case, the promoted heap was not all that large as shown in the diagram below:

Plotting Young Gen Survived Heap

I then investigated survivor heap spaces. Here's the script that plots the survivor heap space along with GC pause time on the same chart:

# Plot survived Heap
xtitle <- 'Elapsed Time (hrs)'
ytitle <-'Survivor Heap (MB)'
plot.title <- "Survived Heap after Minor GC"

with(data.gc, {
   
 x <- sec.to.hours(timestamp)
 y <- kb.to.mb(young.end)
 plot(x,y,
  xlab=xtitle,ylab=ytitle,
  main=plot.title,type='o',col='blue')
  
 # Plot on secondary axis
 par(new=TRUE)
 plot(x,pause.time,
   axes=FALSE,type='o',
   col="green",pch=23,xlab="",ylab="")
 axis(4)
 mtext("GC Pause Time (sec)",4)
})

In addition, I can also check the correlation coefficient via :

> with(data.gc,cor(young.end[gc.type == 'minor'],pause.time[gc.type == 'minor']))
[1] 0.9659936

As we can see from the result, the correlation was pretty high that size of the survivor heap and its effect on GC pause time.

Checking for Memory Leaks

Another plot that I often have to generate is used as a quick check for memory leaks. Here's the R script that plots old gen heap space usage after every full GC:

# Check for memory leak
xtitle <- 'Elapsed Time (hrs)'
ytitle <-'Old Gen Heap (MB)'


with(data.gc, {
 plot.title <- "Old Gen Space after Full GC"
 x <- sec.to.hours(timestamp[gc.type=="full"])
 y <- kb.to.mb(old.end[gc.type=="full"])
 plot(x,y,
   xlab=xtitle,ylab=ytitle,main=plot.title,type='o',col='blue')

 # Add fitted line
 abline(lm(y ~ x),col="red")
 retval <<- lm(y ~ x)
})
retval

With the trend analysis line, we can potentially project when the system will run out of memory. For the following plot, checking the linear model fit along with eyeballing the chart suggests that we don't really have a memory leak issue:

Call:
lm(formula = y ~ x)

Coefficients:
(Intercept)            x  
   1285.978        0.298

I am delighted by the power afforded by R to perform GC analysis. It has now become my preferred tool to perform GC analysis. In addition, because I learned how to use R, I'm beginning to use R in other capacity, including inventory , trending, and capacity planning analysis.

Checking Tibco EMS Queue Connected Users with Clojure

2013-01-28T05:00:00.000-08:00

During our Tibco EMS infrastructure upgrade project, we had some rogue connections that just refused to disconnect from the old infrastructure and switch to the new Tibco EMS infrastructure. In order to track down these connections and evict those connections, I had to generate a report which would tell me which user, from which host is connected to particular queues in Tibco EMS. Here's the script that I wrote in Clojure that generates that report.

; Get destination users (and which host the user is connecting from) along with which queue/topic it is connected to.
(defn get-destination-users [server-url username password]
  (with-open [admin (TibjmsAdmin. server-url username password)]
    (let [connections  (.getConnections admin) 
          consumers    (.getConsumers admin)
          dest-userids (->> consumers
                         (map (fn [c] {:userid (.getConnectionID c) 
                                       :dest (.getDestinationName c)})))
          get-user (fn [dest-name id]
                     (let [conn (first (filter #(= id (.getID %)) connections))]
                       (if (nil? conn)
                         {:dest dest-name :user (str "Unknown user : " id) :host "unknown"}
                         {:dest dest-name :user (.getUserName conn) :host (.getHost conn)})))]
      (map (fn [x] (get-user (:dest x) (:userid x))) dest-userids))))

; Dump results in CSV format
(def results (get-destination-users server-url username password))   
(doseq [r (sort-by :user (set results))]
  (println  (str (:user r) "," (:host r) "," (:dest r))))

Who's Connecting to My Servers?

2013-01-14T04:00:00.000-08:00

I have been reading through the book Clojure Programming. In the course of reading this book, I've looked for all sorts of opportunities to try applying Clojure at work. Most of the time, I've used Clojure to implement convenience scripts to help with my day job. I would typically use Clojure whenever I have to work with Java based platforms and F# for .NET based platforms. Occasionally, I would develop scripts that does not have any dependency and I could choose any language to implement. What would typically happen is that I would choose the programming language that I used last. This strategy, unfortunately, would typically end up biasing me toward one programming language and lately, it has been biasing me toward Clojure. After noticing this trend, I have decided to deliberately and consciously choose to implement in the less frequently used language so I don't become completely rusty in the other programming languages.

Recently, I had to opportunity to write a small script. I was managing an infrastructure upgrade and needed to know the downstream impact. It was an infrastructure component that that a lot of persistent inbound connections, but unfortunately, the inbound connections were neither monitored nor documented. One way to check the connections is ask the the network engineers to setup monitoring on the servers and collect the information on the incoming connections. Our network engineers are generally pretty busy and we hate to add to their existing workloads. However, we can effectively do the same thing by running netstat -an on each of the target servers and taking that output dump and parse that for incoming connections. We would do this over a period of time to try to capture most of the client connections.

The following Clojure script loads all the netstat dump output files and generate a list of all the hosts that are connected to the target servers:

(import '(java.net InetAddress))
(use '[clojure.string :only (join)])
(use '[clojure.java.io :as io])

; Load all the data from all *.data files in c:\work\servers folder
(def data (->> "c:\\work\\servers"
   (io/file)
   (file-seq)
   (map #(.getAbsolutePath %))
   (filter #(re-matches #".*\.data$" %))
   (map #(slurp %))
   (join " ")))

; Find all ip addresses in the netstat dump
; Perform hostname lookup, discard duplicates, sort the hostnames   
(def hosts (->> data
   (re-seq #"(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\.\d+")
   (map #(second %))
   (set)
   (map #(.getCanonicalHostName (InetAddress/getByName %)))
   (sort)
   (join "\n")))

; Dump output to clients.out file
(spit "c:\\work\\servers\\results.out" hosts)

The above script runs with the assumption that all data fits into memory. However, if that becomes a problem, it is fairly trivial to sequentially read and process netstat dump one file at a time and combine the results to write to the output.

The F# version is similar to Clojure version. Grabbing the files from the folder is easier but the need to explicitly handle exceptions adds back the additional lines of code to be about on par with code verbosity of the Clojure version.

open System.IO
open System.Net
open System.Text.RegularExpressions

// Load all the data from all *.data files in c:\work\servers folder
let data = Directory.GetFiles(@"c:\work\servers","*.data")
           |> Seq.map File.ReadAllText
           |> String.concat " "

// Return hostname if it can be resolved
// otherwise return the ip address
let getHostEntry (ipaddress:string) =
    try
        Dns.GetHostEntry(ipaddress).HostName
    with
      | err -> ipaddress

// Find all ip addresses in the netstat dump
// Perform hostname lookup, discard duplicates, sort the hostnames
let hosts = Regex.Matches(data,@"(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\.\d+")
            |> Seq.cast<Match>
            |> Seq.map (fun m -> m.Groups.[1].Value)
            |> Set.ofSeq
            |> Seq.map getHostEntry
            |> Seq.sort
            |> String.concat "\n"

File.WriteAllText(@"c:\work\servers\results.out",hosts)

Quick and easy way to monitor for memory leaks in Java application servers

2012-06-11T06:00:00.000-07:00

I was part of a postmortem analysis team that performed a root cause analysis on why a particular node in a HA paired system failed and needed to provide recommendations to prevent future occurrences. Initial examination of the log showed that the node that failed had a permgen out of memory problem. Management team also wanted to know if there were memory leaks with the application.

Unfortunately, the Java application server was not configured with GC logging nor configured to perform a heap dump on out of memory error condition. Without the gc log data or the heap dump, it becomes harder to be able to explain why did the application server run out of permgen space at the time it failed. In the interest of collecting data and to prevent future node failures, we recommended that permgen space be increased and add the following startup parameters to the Java application server

    -XX:+PrintGCTimeStamps
    -XX:+PrintGCDetails
    -XX:+HeapDumpOnOutOfMemoryError
    -XX:HeapDumpPath=/myapp/heapdumps
    -Xloggc:/myapp/logs/gc.log

One problem that arose during the writeup of the analysis was how to answer the question whether the application had any memory leak issue. The application team could not reproduce the problem in non-production environment but wanted some assurance that the application did not have a memory leak problem with PermGen space or anywhere else in the heap. I proposed that we track heap usage after every Full GC over a period of time and see the heap usage trends over time. The heap usage trends should provide a good indication if the application has a problematic memory leak issue or not.

In order to do so, I wrote a small script to parse out only the heap and permgen space info after a full GC from the gc log. I wrote this script in Clojure as follows:

; Define the regex pattern to parse each line in gc log
(defn full-gc-pattern []
    (let [timestamp  "([\\d\\.]+): .* " 
          space      "(\\d+)K->(\\d+)K\\((\\d+)K.+ "
          new-gen    space
          old-gen    space
          perm-gen   space
          heap       "(\\d+)K->(\\d+)K\\((\\d+)K\\)\\], "
          exec-stat  "(\\d+\\.\\d+).*" ]
         (re-pattern (str timestamp new-gen old-gen perm-gen heap exec-stat))))                 

; We only want to dump timestamp, total heap and perm gen space
; after each full GC
; Variable definitions:
;     ts - timestamp (in seconds)
;     ys - YoungGen space starting heap size (in KB)
;     ye - YoungGen space ending heap size (in KB)
;     ym - YoungGen space max heap size (in KB)
;     os - OldGen space starting heap size (in KB)
;     oe - OldGen space ending heap size (in KB)
;     om - OldGen space max heap size (in KB)
;     hs - Total heap space starting heap size (in KB)
;     he - Total heap space ending heap size (in KB)
;     hm - Total heap space max heap size (in KB)
;     ps - PermGen space starting heap size (in KB)
;     pe - PermGen space ending heap size (in KB)
;     pm - PermGen space max heap size (in KB)
(defn process-full-gc [entry]
    (let [parsed (first (re-seq (full-gc-pattern) entry))
         [a ts ys ye ym os oe om hs he hm ps pe pm & e] parsed]
        (println (format "%s,%s,%s" ts he pe))))

; Load the gc log data
(def gcdata (line-seq (clojure.java.io/reader (clojure.java.io/file "gc.log"))))
        
; Process each full GC entry after filtering out minor GC entries          
(doseq [line (keep #(re-find #".*Full GC.*" %) gcdata)]
    (process-full-gc line))

Using this script against the gc log file, and taking that output and loading into a spreadsheet, I can plot and generate a trendline to see if memory is growing or not. Here's the heap usage plot over approximately a month.

As seen on the chart, the slope of the trendlines are small and negative, which is a good indicator that we do not have a memory leak problem with this application.

Character Encoding Troubles

2012-05-28T06:00:00.000-07:00

In the past, I encountered an issue with a legacy application that migrated from WebLogic Server running on a old Windows server to a newer Tomcat application server running on Red Hat Linux environment. This application has been running fine without issues for weeks until the developer suddenly started to complain that this application is not saving the registered ® and copyright © trademark symbols to the database. The developer also said that when he runs the application from his Windows laptop, he's able to save these two symbols into the database.

Earlier in my career, I was developer for a software company that built multilingual software specializing in Asian languages, I recognize this as a character encoding issue. So I asked the developer to send me the source code related to this issue and here's the relevant part of the code:

    // Sending updates to the database
    update.setValue("data",encodeString(text));
    // Inserting the data to the database
    insert.setValue("data",encodeString(text));

That seemed odd, what does the method encodeString do? Here's the implementation:

public String encodeString(String value) throws java.io.UnsupportedEncodingException
{
  log("encodeString", "Begin");
  if (value == null)
  {
    log("encodeString", "value == null");
    return value;
  }
  
  byte [] btValue = value.getBytes();
  String encodedValue = new String(btValue, _ISO88591);
  
  /*
  Charset utf8charset = Charset.forName("UTF-8");
  Charset iso88591charset = Charset.forName("ISO-8859-1");

  ByteBuffer inputBuffer = ByteBuffer.wrap(btValue);

  // decode UTF-8
  CharBuffer data = utf8charset.decode(inputBuffer);

  // encode ISO-8559-1
  ByteBuffer outputBuffer = iso88591charset.encode(data);
  byte[] outputData = outputBuffer.array();
  byte[] inputData = inputBuffer.array();
 
  log("ISO-8859-1: ", new String(outputData));
  log("UTF-8: ", new String(inputData));
  
  //String encodedValue = new String(btValue, _ISO88591);
  String encodedValue = new String(inputData);
  String encodedValue_ISO88591 = new String(inputData, _ISO88591);
  //encodedValue.getBytes("UTF-8");
  log("Encoded UTF: ", encodedValue);
  log("Encoded ISO88591: ", encodedValue_ISO88591);
  */
  
  return encodedValue;
}

Wow! I can see that the developer is trying to get a handle on this encoding business and hence all the commented out R&D code, but clearly this developer is not familiar with character set encoding issues.

The main problem is with the following line of code:

  byte [] btValue = value.getBytes();

From Java API manual, the getBytes() method encodes the string into a sequence of bytes using the platform's default charset. On the old legacy Windows Server that this application was originally running on, it was probably using Windows code page 1252, which is basically ISO-8859-1 and hence the register and copyright symbols were correctly encoded. However, on the Red Hat Linux operating system, the default encoding was ascii and therefore the register/copyright symbol got converted into question marks.

Java strings are internally unicode (UTF-16) and typically the JDBC drivers will provide the appropriate conversions to and from the database. Therefore, fix to this application is simple, all one have to do is merely change the following two lines of code and get rid of the entire encodeString() method:

    // Changed from : update.setValue("data",encodeString(text));
    update.setValue("data",text);
    // Changed from : insert.setValue("data",encodeString(text));
    insert.setValue("data",text);

Working with Tibco EMS Message Topics using F# and Clojure

2012-05-14T06:00:00.000-07:00

In my previous blog post, I showed how to connect to a Tibco EMS Queue with F# and Clojure to represent integration interoperability from .NET and Java platforms. Message queues are a way to implement the Request-Reply pattern, which is one of the many enterprise integration patterns described in the book Enterprise Integration Patterns. Another basic enterprise integration pattern is the Publish-Subscribe pattern, which can be implement via JMS topics. This blog post shows how to connect to JMS topic from .NET and Java using Tibco EMS as the JMS provider.

Here is the F# version:

#r @"C:\tibco\ems\6.3\bin\TIBCO.EMS.dll"

open System
open TIBCO.EMS

let serverUrl = "tcp://localhost:7222"
let producer = "producer"
let consumer = "consumer"
let password = "testpwd"
let topicName = "testTopic"


let subscribeToTopic serverUrl userid password topicName messageProcessor =
    async {
        let connection = (userid,password)
                         |> (new TopicConnectionFactory(serverUrl)).CreateTopicConnection
        let session = connection.CreateTopicSession(false,Session.AUTO_ACKNOWLEDGE)
        let topic = session.CreateTopic(topicName)
        let subscriber =  session.CreateSubscriber(topic)
        connection.Start()
        printf "Subscriber connected!\n"
        while true do
            try
                subscriber.Receive() |> messageProcessor
            with _ ->  ()
        connection.Close()
    }

let publishTopicMessages serverUrl  userid password topicName messages =
    let connection = (userid,password)
                     |> (new TopicConnectionFactory(serverUrl)).CreateTopicConnection
    let session = connection.CreateTopicSession(false,Session.AUTO_ACKNOWLEDGE)
    let topic = session.CreateTopic(topicName)
    let publisher = session.CreatePublisher(topic)
    connection.Start()

    messages
    |> Seq.iter (fun item -> session.CreateTextMessage(Text=item)
                             |> publisher.Send)
                             
    connection.Close()

// Just dump message to console for now
let myMessageProcessor (msg:Message) =
    msg.ToString() |> printf "%s\n"

let consumeMessageAsync = subscribeToTopic "tcp://localhost:7222" "consumer" "testpwd"


let produceMessages topicName messages = publishTopicMessages "tcp://localhost:7222" "producer" "testpwd" topicName messages 


// Asynchronously start the topic subscriber
Async.Start(consumeMessageAsync "testTopic" myMessageProcessor)


// Publish messages to the Tibco EMS topic
[ "Aslund"; "Barrayar"; "Beta Colony"; "Cetaganda"; "Escobar"; "Komarr"; "Marilac"; "Pol"; "Sergyar"; "Vervain"]
|> produceMessages "testTopic"


printf "Done!"

One thing to point out is that Tibco, unfortunately, did not implement IDisposable for it's Connection objects; perhaps in it's bid to stay faithful to the Java API. That design choice seems unfortunate to me in the sense that I no longer can leverage C#'s using keyword or F#'s use keyword to automatically close connection. I suppose it is fairly trivial to subclass the QueueConnection and TopicConnection class and add the IDisposable interface, but I feel that Tibco should have done this and developed the Tibco .NET API using idioms that are .NET specific.

Putting my rants aside, here is the equivalent Clojure code to connect to Tibco Topics:

(import '(java.util Enumeration)
        '(com.tibco.tibjms TibjmsTopicConnectionFactory)
        '(javax.jms Message JMSException  Session
                    Topic TopicConnectionFactory
                    TopicConnection TopicSession
                    TopicSubscriber))
                  
(def serverUrl "tcp://localhost:7222")
(def producer "producer")
(def consumer "consumer")
(def password "testpwd")
(def topicName "testTopic")

;------------------------------------------------------------------------------
; Subscribe to Topic asynchronously
;------------------------------------------------------------------------------
(defn subscribe-topic [server-url user password topic-name process-message]
    (future
        (with-open [connection (-> (TibjmsTopicConnectionFactory. server-url)
                                   (.createTopicConnection user password))]
            (let [session (.createTopicSession connection false Session/AUTO_ACKNOWLEDGE)
                  topic (.createTopic session  topic-name)]
                (with-open [subscriber (.createSubscriber session topic)]
                    (.start connection)
                    (loop []                       
                        (process-message (.receive subscriber))
                        (recur)))))))

;------------------------------------------------------------------------------
; Publishing to a Topic
;------------------------------------------------------------------------------
(defn publish-to-topic [server-url user password topic-name messages]
    (with-open [connection (-> (TibjmsTopicConnectionFactory. server-url)
                               (.createTopicConnection user password))]
        (let [session (.createTopicSession connection false Session/AUTO_ACKNOWLEDGE)
              topic (.createTopic session  topic-name)
              publisher (.createPublisher session topic)]
            (.start connection)
            (doseq [item messages]
                (let [message (.createTextMessage session)]
                    (.setText message item)
                    (.publish publisher message))))))
                    
                      
; Create function aliases with connection information embedded                    
(defn produce-messages [topic-name messages]
    (publish-to-topic "tcp://localhost:7222" "producer" "testpwd" topic-name messages))

(defn consume-messages [topic-name message-processor]
    (subscribe-topic "tcp://localhost:7222" "consumer" "testpwd" topic-name message-processor))

; Just dump messages to console for now
(defn my-message-processor [message]
    (println (.toString message)))
    
; Start subscribing messages asynchronously
(consume-messages "testTopic" my-message-processor)                            
    
; Publish to topic
(def my-messages '("alpha" "beta" "gamma" "delta"
                   "epsilon" "zeta" "eta" "theta"
                   "iota" "kappa" "lambda" "mu" "nu"
                   "xi", "omicron" "pi" "rho"
                   "signma" "tau" "upsilon" "phi",
                   "chi" "psi" "omega"))                    

(produce-messages  "testTopic"  my-messages)

When I fire up both scripts, the messages published to the topic would be received by both the .NET and Java clients. With these scripts, I can easily swap out the message generators or message processors as needed for any future testing scenarios.

F#, Clojure and Message Queues on Tibco EMS

2012-04-30T06:00:00.000-07:00

It looks like I will be getting much more hands on with Tibco EMS. Since the Tibco EMS system in use will have connections from both .NET platforms and Java platforms, I wanted to write some scripts to run some engineering tests on Tibco EMS. I decided to simulate .NET side connections with F# and Java side connections with Clojure. Taking the sample code from Tibco installation, I created the following F# script that sends messages to a queue from the sample C# code:

#r @"C:\tibco\ems\6.3\bin\TIBCO.EMS.dll"

open System
open TIBCO.EMS

let serverUrl = "tcp://localhost:7222"
let producer = "producer"
let consumer = "consumer"
let password = "testpwd"
let queueName = "testQueue"


let getQueueTextMessages serverUrl  userid password queueName messageProcessor =
    async {
        let connection = (userid,password)
                         |> (new QueueConnectionFactory(serverUrl)).CreateQueueConnection
        let session = connection.CreateQueueSession(false,Session.AUTO_ACKNOWLEDGE)
        let queue = session.CreateQueue(queueName)
        let receiver =  session.CreateReceiver(queue)
        connection.Start()
        printf "Queue connection established!"
        while true do
            try
                receiver.Receive() |> messageProcessor
            with _ ->  ()
    }


let sendQueueTextMessages serverUrl  userid password queueName messages =
    let connection = (userid,password)
                     |> (new QueueConnectionFactory(serverUrl)).CreateQueueConnection
    let session = connection.CreateQueueSession(false,Session.AUTO_ACKNOWLEDGE)
    let queue = session.CreateQueue(queueName)
    let sender = session.CreateSender(queue)
    connection.Start()

    messages
    |> Seq.iter (fun item -> session.CreateTextMessage(Text=item)
                             |> sender.Send)
                             
    connection.Close()



// Just dump message to console for now
let myMessageProcessor (msg:Message) =
    msg.ToString() |> printf "%s\n"


let consumeMessageAsync = getQueueTextMessages "tcp://localhost:7222" "consumer" "testpwd"
let produceMessages queueName messages = sendQueueTextMessages "tcp://localhost:7222" "producer" "testpwd" queueName messages 

// Start message consumer asynchronously
Async.Start(consumeMessageAsync "testQueue" myMessageProcessor)


// Send messages to the Tibco EMS   
[ "Aslund"; "Barrayar"; "Beta Colony"; "Cetaganda"; "Escobar"; "Komarr"; "Marilac"; "Pol"; "Sergyar"; "Vervain"]
|> produceMessages "testQueue"

The queue consumer is implemented asynchronously so it won't block executing subsequent statements. To test Tibco JMS from Java, here is the equivalent Clojure code:

(import '(java.util Enumeration)
        '(com.tibco.tibjms TibjmsQueueConnectionFactory)
        '(javax.jms Message JMSException  Session
                    Queue QueueBrowser 
                    QueueConnection QueueReceiver 
                    QueueSession QueueSender))
                  
(def serverUrl "tcp://localhost:7222")
(def producer "producer")
(def consumer "consumer")
(def password "testpwd")
(def queueName "testQueue")

; Consume Queue Text messages asynchronously
(defn get-queue-text-messages [server-url user password queue-name process-message]
    (future
        (with-open [connection (-> (TibjmsQueueConnectionFactory. server-url)
                                   (.createQueueConnection user password))]
            (let [session (.createQueueSession connection false Session/AUTO_ACKNOWLEDGE)
                  queue (.createQueue session  queue-name)]
                (with-open [receiver (.createReceiver session queue)]              
                    (.start connection)
                    (loop []                       
                        (process-message (.receive receiver))
                        (recur)))))))
                   
; Send multiple Text messages
(defn send-queue-text-messages [server-url user password queue-name messages]
    (with-open [connection (-> (TibjmsQueueConnectionFactory. server-url)
                               (.createQueueConnection user password))]
        (let [session (.createQueueSession connection false Session/AUTO_ACKNOWLEDGE)
              queue (.createQueue session  queue-name)
              sender (.createSender session queue)]
            (.start connection)
            (doseq [item messages]
                (let [message (.createTextMessage session)]
                    (.setText message item)
                    (.send sender message))))))


; Create function aliases with connection information embedded                    
(defn consume-messages [queue-name message-processor]
    (get-queue-text-messages  serverUrl producer password queue-name message-processor))

(defn produce-messages [queue-name messages]
    (send-queue-text-messages  serverUrl producer password queue-name messages))

; Just dump messages to console for now
(defn my-message-processor [message]
    (println (.toString message)))

    
; Start consuming messages asynchronously
(consume-messages "testQueue" my-message-processor)                            

; Send messages to queue
(def my-messages '("alpha" "beta" "gamma" "delta"
                   "epsilon" "zeta" "eta" "theta"
                   "iota" "kappa" "lambda" "mu" "nu"
                   "xi", "omicron" "pi" "rho"
                   "signma" "tau" "upsilon" "phi",
                   "chi" "psi" "omega"))                    

(produce-messages  "testQueue"  my-messages)

With these scripts, I can easily swap in different message generators and message processors as needed for any testing purposes. When I fired up both these scripts up to the part where queue consumers are running in both F# and Clojure version and then send the messages, I could see that Tibco EMS send half the messages to my F# script and the other half to my Clojure script. Since both of these scripts run in REPL environment, I can easily adjust my level of testing as I get results.

F# and SharePoint 2010 Object Hierarchy/Properties

2012-04-16T06:00:00.000-07:00

I had the opportunity to take a SharePoint 2010 class recently. In the class, the labs were mostly a cut and paste affair due to time limitations. Those lab exercises only helps me to become familiar with working in Visual Studio and seeing some of the SharePoint API, but does not really help me engage actively in thinking about what I was actually doing. After the class, I decided to write a F# equivalent of the lab exercises to help me get a deeper understanding and to learn. The class did impress on me that the tooling for SharePoint 2010 development is so much more superior in C# such that I've decided to build only the logic code in F# and retain the C# SharePoint project. This blog post describes my effort in getting that first class exercise working with F# code.

I ended up creating an empty SharePoint 2010 C# project that references a F# library project. I added two application pages to the SharePoint 2010 project, the first being FarmHierarchy.aspx. In FarmHierarchy.aspx, I added the following markup between the opening and closing tags of the <asp:Content> element that has an ID of Main:

<h2>My Farm</h2>
<asp:TreeView ID="farmHierarchyViewer" runat="server"
    ShowLines="true" EnableViewState="true"></asp:TreeView>

The second application page I created was PropertyChanger.aspx. I added the following markup between the opening and closing tags of the <asp:Content> elsement that has an ID of Main:

    <h2>Properties:</h2>
    <asp:Label ID="objectName" runat="server" Text=""></asp:Label><br/><br/>
    <asp:Panel ID="webProperties" runat="server" Visible="false" BorderColor="Orange" BorderStyle="Dashed" BorderWidth="1">
        <asp:Label ID="WebLabel" runat="server" Text="Web Title"></asp:Label>
        <br/>
        <asp:TextBox ID="webTitle" runat="server" EnableViewState="true"></asp:TextBox>
        &nbsp;
        <asp:Button ID="webTitleUpdate" runat="server" Text="Update"/>
        &nbsp;
        <asp:Button ID="webCancel" runat="server" Text="Cancel" />
    </asp:Panel>
    <asp:Panel ID="listProperties" runat="server" Visible="false" BorderColor="Orange" BorderStyle="Dashed" BorderWidth="1">
        <asp:Label ID="ListLabel" runat="server" Text="List Properties"></asp:Label>
        <br/>
        <asp:CheckBox ID="listVersioning" runat="server" EnableViewState="true" Text="Enable Versioning" />
        <br/>
        <asp:CheckBox ID="listContentTypes" runat="server" EnableViewState="true" Text="Enable Content Types" />
        &nbsp;
        <asp:Button ID="listPropertiesUpdate" runat="server" Text="Update" />
        &nbsp;
        <asp:Button ID="listCancel" runat="server" Text="Cancel"/>
    </asp:Panel>

The F# code would iterate through the services, Web applications, site collections, and lists in the SharePoint farm, with the details added to nodes in the TreeView control:


module FsLab01

open System
open System.Web.UI.WebControls
open Microsoft.SharePoint
open Microsoft.SharePoint.Administration
open System.Web.UI

// Convenience methods
let nullfunc _ = ()

// Copied from Clojure
let cond clauses =
    let (_,func) = clauses |> Seq.find (fun (pred,func) -> pred)
    func()


let navListUrl url (id:Guid) =
    sprintf "%s/_layouts/lab01/PropertyChanger.aspx?type=list&objectID=%s" url <| id.ToString()


let navWebUrl url (id:Guid) =
    sprintf "%s/_layouts/lab01/PropertyChanger.aspx?type=web&objectID=%s" url <| id.ToString()

// Recursively add SPWeb & SPList objects
let rec addWeb (web:SPWeb) (parentNode:TreeNode) =

    let node = new TreeNode(web.Title,null,null,navWebUrl web.Url web.ID, "_self")
    node |> parentNode.ChildNodes.Add

    [0..(web.Lists.Count-1)]
    |> Seq.map (fun i -> web.Lists.[i])
    |> Seq.iter (fun item -> 
                      new TreeNode(item.Title,null,null,navListUrl web.Url item.ID,"_self")
                      |> node.ChildNodes.Add)
        
    [0..(web.Webs.Count-1)]
    |> Seq.map (fun i -> web.Webs.[i])
    |> Seq.iter (fun item -> try addWeb item node finally item.Dispose())


// Main function to be called by FarmHierarchy.aspx code to build TreeView control
let loadViewer (viewer:TreeView) (farm:SPFarm) =

    let processSite (webappnode:TreeNode) (site:SPSite) =
        site.CatchAccessDeniedException <- false
        try
            let node = new TreeNode(Text=site.Url)
            node |> webappnode.ChildNodes.Add

            addWeb site.RootWeb node

        finally
            site.CatchAccessDeniedException <- false
            

    let processWebApp (svcNode:TreeNode) (webapp:SPWebApplication) = 
        let node = new TreeNode(Text=webapp.DisplayName )
        node |> svcNode.ChildNodes.Add

        cond [(not webapp.IsAdministrationWebApplication,
               fun _ -> webapp.Sites |> Seq.iter (processSite node));
              (true,nullfunc)]
            
                

    let processService (svc:SPService) =
        let label = sprintf "FarmService (Type=%s; Status=%s)" (svc.TypeName) (svc.Status.ToString())
        let node = new TreeNode(Text=label)
        node |> viewer.Nodes.Add
        match svc with
        | :? SPWebService as websvc -> websvc.WebApplications |> Seq.iter (processWebApp node)
        | _ -> ()

    
    viewer.Nodes.Clear()
    farm.Services |> Seq.iter processService
    viewer.ExpandAll()

As I was creating loadViewer method, I was bothered by the if-else-then clauses in the code. I had a previous blog that talked about this issue when it struck me that what I really wanted was something similar to the cond macro in Clojure. Hence the convenience function called cond in the above F# code.

With the F# code written and packaged as a library, I can now use the above F# function in FarmHierarchy.aspx as follows:

        protected void Page_Load(object sender, EventArgs e)
        {
            SPFarm thisFarm = SPFarm.Local;
            FsLab01.loadViewer(farmHierarchyViewer, thisFarm);
        }

The second part of this lab was to create code to manipulate properties of SharePoint SPWeb or SPList objects. The F# code that manipulates the SharePoint object properties and the UI are as follow:



let changeProperty (page:Page) (objectName:Label) (webtitle:TextBox) 
                   (listpanel:Panel) (webpanel:Panel) 
                   (listVersioning:CheckBox) (listContentTypes:CheckBox) 
                   (webUpdateBtn:Button) (listUpdateBtn:Button)
                   (webCancelBtn:Button) (listCancelBtn:Button)   =    


    let homeurl  baseurl = sprintf "%s/_layouts/lab01/FarmHierarchy.aspx" baseurl

    let wrapupdates (web:SPWeb) action =
        web.AllowUnsafeUpdates <- true
        action()
        web.AllowUnsafeUpdates <- false
        

    let hidepanels () =
        listpanel.Visible <- false
        webpanel.Visible <- false

    let cancel (web:SPWeb) =
        page.Response.Redirect(homeurl web.Url)
    
        
    let checkNull item = 
        cond [(item=null,  fun _ -> objectName.Text <- "Malformed URL"
                                    hidepanels()
                                    "");
              (item<>null, fun _ -> item.ToString())]


    try
        let objectType = checkNull page.Request.["type"]
        let objectId = checkNull page.Request.["objectID"]

        match objectType with
        | "web" -> 
            listpanel.Visible <- false
            webpanel.Visible <- true
            use web = SPContext.Current.Site.OpenWeb(new Guid(objectId))

            // Hook up the events
            let myupdates _ =
                web.Title <- webtitle.Text
                web.Update()

            webUpdateBtn.Click.Add(
                fun _ ->
                    try 
                        wrapupdates web myupdates

                        page.Response.Redirect(homeurl web.Url)
                    with ex ->
                        objectName.Text <- ex.Message
                        hidepanels()
                )

            webCancelBtn.Click.Add(fun _ -> cancel web)

            objectName.Text <- sprintf "Web: %s" web.Title

            cond [(not page.IsPostBack,
                   fun () -> webtitle.Text <- web.Title);
                   (true,nullfunc)]

        | "list" -> 
            listpanel.Visible <- true
            webpanel.Visible <- false
            let web = SPContext.Current.Web
            let splist = web.Lists.[new Guid(objectId)]


            let myupdates _ =
                splist.EnableVersioning <- listVersioning.Checked
                splist.ContentTypesEnabled <- listContentTypes.Checked
                splist.Update()

            listUpdateBtn.Click.Add(
                fun _ ->
                    try 
                        wrapupdates web myupdates
                        page.Response.Redirect(homeurl web.Url)
                    with ex ->
                        objectName.Text <- ex.Message
                        hidepanels())

            listCancelBtn.Click.Add(fun _ -> cancel web)

            cond [(not page.IsPostBack,
                   fun _ -> listVersioning.Checked   <- splist.EnableVersioning
                            listContentTypes.Checked <- splist.ContentTypesEnabled)]
        | _ -> ()


    with  ex-> objectName.Text <- ex.Message

I needed to sign my F# code in order it to work in SharePoint. Since the C# SharePoint 2010 already created the key, all I had to do is to add --keyfile:<Location to keyfile>\key.snk to the F# build setting's "Other Flags" properties. In addition, before I can build and deploy, I needed to add the F# library dll to the C# SharePoint project's package. I can do so by clicking on Package in the SharePoint project and then click on the Advanced button.

I can now call this F# function from PropertyChanger.aspx code as follows:

            FsLab01.changeProperty(this.Page, objectName, webTitle,
                                   listProperties, webProperties, 
                                   listVersioning, listContentTypes,
                                   webTitleUpdate, listPropertiesUpdate,
                                   webCancel,listCancel);

Here's how what FarmHierarchy.aspx would look like :

Here's how what PropertyChanger.aspx would look like if I wanted to modify a SPWeb object:

Here's how what PropertyChanger.aspx would look like if I wanted to modify a SPList object:

F# and Windows Server AppFabric Cache

2012-03-26T06:00:00.000-07:00

I recently started investigating Windows Server AppFabric, which is different from Windows Azure AppFabric, and trying understand how to use AppFabric, what are some of the operational support implications and functional capabilities and limitations. To help me get going, I have been reading the book Pro Windows Server AppFabric by Stephen Kaufman and going through the Windows Server AppFabric Training Kit. The free downloadable training kit was more valuable in helping me understand Windows Server AppFabric.

As always, the only way to truly learn new technology is to take it out for a spin. Here are some of the powershell scripts that I've used to create the cache and get it going:

# Import the necessary administration modules
Import-Module DistributedCacheAdministration
Import-Module DistributedCacheConfiguration

# List the available powershell commands
Get-Command -module DistributedCacheAdministration   
Get-Command -module DistributedCacheConfiguration


# These were run on the localhost of AppFabric
Use-CacheCluster

# Start the cache cluster
Start-CacheCluster

# Check to make sure it is up and running
Get-CacheHost

# Create a new cache 
New-Cache -CacheName MyTestCache -TimeToLive 60 -Expirable true

# Check cache is created
Get-Cache

# Grant local user access to cache
Grant-CacheAllowedClientAccount MyUserId

# Check security settings
Get-CacheAllowedClientAccounts

# After a few runs, check Cache statistics
Get-CacheStatistics -CacheName MyTestCache 

# Get Cache configuration information
Get-CacheConfig MyTestCache 


# Creating a Cache with HA - all server needs to be on 
# Windows Server Enterprise Edition
# Any host not on Enterprise Edition will not start cache cluster
New-Cache MyNewHACache -Secondaries 1

# Modifying the cache to enable callbacks
Set-CacheConfig -CacheName MyTestCache -NotificationsEnabled true -TimeToLive 180

For the client, I wrote some F# scripts to experiment with Windows Server AppFabric. I ran these on the local server and the loopback adapter enabled:

// I copied the libraries from c:\windows\system32\appfabric
#r @"C:\lib\Microsoft.ApplicationServer.Caching.Client.dll"
#r @"C:\lib\Microsoft.ApplicationServer.Caching.Core.dll"

open System
open System.Collections.Generic
open Microsoft.ApplicationServer.Caching

//  Make sure you grant
// grant-cacheallowedclientaccount my-user-id

// Expensive operation, do this once on startup
let dcf = new DataCacheFactory(
            let servers = new List<DataCacheServerEndpoint>()
            new DataCacheServerEndpoint("localhost",22233) |> servers.Add
            new DataCacheFactoryConfiguration(Servers=servers))
printfn "Data Cache Factory Created!"

// Testing add/get in default region
let cache = dcf.GetCache("MyTestCache")
let retval = cache.Add("mykey","hello app fabric!") 
cache.Get("mykey")

// Create a new region in the cache, this will pin to a single cache node
cache.CreateRegion("stocks")

// Helper function to create DataCacheTags
let createTags tags = Seq.map (fun tag -> new DataCacheTag(tag)) tags

   
type Company =
  { Symbol : string; Name: string; Address : string;  Phone : string; Tags :seq<string> }

// Create a bunch of company info and put it in the "stocks" region
[{Symbol="AAPL"; Name="Appl Inc."; Phone="408-996-1010"; 
  Address="1 Infinite Loop, Cupertino, CA 95014";
  Tags = ["Technology";"Nasdaq";"Personal Computers"]};
 {Symbol="CAT"; Name="Caterpillar, Inc."; Phone="309-675-1000"; 
  Address="100 North East Adams Street, Peoria, IL 61629";
  Tags = ["Dow";"Industrial Goods";"Farm & Construction Machinery"]};
 {Symbol="ACI"; Name="Arch Coal, Inc."; Phone="314-994-2700"; 
  Address="Once City Place Drive, Suite 300, St. Louis, MO 63141";
  Tags = ["Basic Materials";"Industrial Metals & Materials"]};
 {Symbol="HP"; Name="Hewlett-Packard Company"; Phone="650-857-1501"; 
  Address="3000 Hanover Street, Palo Alto, Ca 94304";
  Tags = ["Dow";"Technology"; "Diversified Computer Systems"]};
 {Symbol="JPM"; Name="JP Morgan Chase & Co."; Phone="212-270-6000"; 
  Address="270 Park Avenue, New York, NY 10017";
  Tags = ["Dow";"Financial"; "Money Center Banks"]};
 {Symbol="XOM"; Name="Exxon Mobile Corporation"; Phone="972-444-1000"; 
  Address="5959 Las Colinas Boulevard, Irving, TX 75039-2298";
  Tags = ["Dow";"Basic Materials"; "Major Integrated Oil & Gas"]};]
|> Seq.iter (fun item -> cache.Put(item.Symbol, item, (createTags item.Tags),"stocks") |> ignore)


// Get all stocks in "stocks" region as a HashMap 
let stocks = cache.GetObjectsInRegion("stocks")


// Bulk get, only available with defined regions not default region
let myportfolio = ["AAPL";"CAT";"HP";"JPM"]
let mystocks = cache.BulkGet(myportfolio,"stocks")

// Getting objects from cache by tags (again, only available in defined region)
cache.GetObjectsByTag(new DataCacheTag("Dow Jones"),"stocks")


// Getting stuff by all tags (AND filter)
let candidateTags = ["Technology";  "Personal Computers"] |> createTags
cache.GetObjectsByAllTags(candidateTags,"stocks") 

// Getting stuff by any tags (OR filter)
let interestTags = ["Technology";  "Financial"; "Basic Materials"] |> createTags
cache.GetObjectsByAnyTag(interestTags,"stocks")   

// Concurrency policy is strictly done by the client
// No explicit cache server control on concurrency

// Optimistic Currency example
cache.Remove("mykey")
let version = cache.Add("mykey","mystuff")

// Pretend another thread snuck in and modified this cache item.
cache.Put("mykey","changed value")

// This will check that somebody else modified this and throw an exception
cache.Put("mykey","another value",version)

// Pessimistic locking - locks across all cache nodes
let mutable lockHandle:DataCacheLockHandle = null
let item = cache.GetAndLock("mykey",TimeSpan.FromMinutes(60.0), &lockHandle,true)   


// This will throw error due to the first lockHandle
let mutable lockHandle2:DataCacheLockHandle = null

// This will throw an exception as it was already previously locked
let item2 = cache.GetAndLock("mykey",TimeSpan.FromMinutes(60.0),&lockHandle2, true)   

// This will totally ignore locking policies and overwrite things
// Locking is controlled by diligence on the client side.
cache.Put("mykey","no enforced concurrency on the server")


// Finally, this will update the data and unlock this item in the cache
// Need to redo the GetAndLock for this to work.  Previous statement would wipe out the lock
cache.PutAndUnlock("mykey","new stuff",lockHandle)


// Working with Callbacks
(DataCacheOperations.AddItem,
 fun cacheName regionName key version cacheOperation nd -> 
   printfn "Item added to cache : %s\n" key)
|> cache.AddCacheLevelCallback
|> ignore


// When this is call, it could be seconds or minutes before anything 
// is printed to the console
cache.Add("newkey","new stuff")

F# and SharePoint 2010

2012-03-12T06:00:00.000-07:00

I bought a bunch of SharePoint 2010 books and haven't had the time to go through those books. Recently, I finally had some time to take look at those books and try it out in F#. Trying out examples with SharePoint 2007 was fairly painless as I get to do everything in a 32-bit environment. Trying to get Visual Studio 2010 F# to work with SharePoint 2010 was more challenging. While I was able to get the compiled F# code to work, I couldn't overcome the hurdles to get SharePoint 2010 to work with F# Interactive, which is really my prefer way to investigate SharePoint server object models. I can tell from this Stack Overflow entry that I was not alone in having trouble to get F# interactive to work with SharePoint API. The combination of .NET 3.5 framework and 64-bit defeated my effort at getting it to work. I have tried Igor Dvorkin's steps to run F# interactive in 64-bit and then taking the command line arguments for the F# compiler and apply to F# interactive where applicable. So I would start up my F# interactive with the following command line parameters:

"C:\Program Files (x86)\Microsoft F#\v4.0\fsi.exe" ^
  --debug:full ^
  --noframework ^
  --define:DEBUG ^
  --define:TRACE ^
  --optimize- ^
  --tailcalls- ^
  -r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\FSharp\2.0\Runtime\v2.0\FSharp.Core.dll" ^
  -r:"C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\ISAPI\Microsoft.SharePoint.dll" ^
  -r:C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscorlib.dll ^
  -r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\v3.5\System.Core.dll" ^
  -r:C:\Windows\Microsoft.NET\Framework\v2.0.50727\System.dll ^
  -r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\v3.0\System.ServiceModel.dll"

When I try to run with the above arguments, I get the following error:

error FS0084: Assembly reference 
'C:\Program Files (x86)\Reference Assemblies\Microsoft\FSharp\2.0\Runtime\v2.0\FSharp.Compiler.Interactive.Settings.dll' 
was not found or is invalid

So something in FSharp.Compiler.Interactive.Settings.dll is not working with .NET Framework 3.5. I wish there was an option in F# interactive that allows me to target the runtime .NET framework to use (e.g. --target:3.5)

The good news is that the F# compiler works fine as long as you make sure that you are targeting .NET 3.5 runtime and x64 CPU. Here's the F# version of a sample code that I pulled from the book Inside Microsoft SharePoint 2010 by Ted Pattison, Andrew Connell, Scott Hillier and David Mann

open System
open Microsoft.SharePoint

let siteUrl = "http://localhost/"
let sites = new SPSite(siteUrl)

let site = sites.RootWeb
http://www.blogger.com/blogger.g?blogID=18281936#editor/target=post;postID=3313071912798963533
seq {for i in 0..(site.Lists.Count-1) do
       if site.Lists.[i].Hidden <> true then
         yield (site.Lists.[i]) }
|> Seq.iter (fun item -> printf "%s\n" item.Title)

For those who have read my past posts on F# and SharePoint and my past blog post on Revisiting the SharePoint collection adapter for F#, I am sorry to say that I have been led astray by C# example codes and my brain temporary malfunctioned as it produced the SharePoint adapter nonsense in that previous blog post. There was no need to write any SharePoint utility library in C# to get F# to work with SPListCollection as sequences. In the above example, I used sequence expressions to generate SPList sequences from SPListCollection. Another way is to simply create a sequence by use map function as shown in the following example:

// Create a sequence by mapping over all items in the collection
let spcollection = site.Lists
[0..spcollection.Count-1] 
|> Seq.map (fun i -> spcollection.[i])
|> Seq.iter (fun item -> printf "%s\n" item.Title)

There was no need to leave the confines of F# development and it is also type safe compared to my previous clumsy attempt at converting SPListCollection into a sequence. If you really do want to use an adapter, it could also be implemented in F# as follows:

// SPCollectionAdapter written in F#
let fromSPCollection<'a,'b when 'b :> SPBaseCollection> (collection:'b) =
    let enumerator = collection.GetEnumerator()
    let rec enumerate (e:Collections.IEnumerator) acc =
        let flag = e.MoveNext()
        if flag = false then
            acc
        else
            enumerate e (e.Current :?> 'a :: acc)
    enumerate enumerator [] 

// Example usages
fromSPCollection<SPList,SPListCollection> site.Lists
|> Seq.iter (fun item  -> printf "%s\n" item.Title)
    

// webapp is an instance of SPWebApplication object
fromSPCollection<SPContentDatabase,SPContentDatabaseCollection>  webapp.ContentDatabases
|> Seq.iter (fun db -> 
              printf "Content Database : %s\n" db.Name
              printf "Connection String : %s\n" db.DatabaseConnectionString)

Adventures in troubleshooting out of memory errors with Coherence cluster.

2012-03-06T06:00:00.000-08:00

One day, an application team manager called me and said that their application caused an out of memory error condition in their Oracle Coherence cluster. This same code base ran in the old Coherence 3.1 environment for months without running into out of memory conditions and now is failing in the new Coherence 3.6 environment in matter of a few weeks on a regular basis. He said that he had heap dumps and logs and asked whether I could take a look at it and troubleshoot it.

Initially, I was skeptical about being able to help this team manager out. After all, I know almost nothing about their application code and in all practical terms, I had no previous development experience with Coherence with the exception that I read the book Oracle Coherence 3.5 by Aleksandar Seovic in the past. My previously participated in testing Coherence performance on VMware and that really did not require me to delve into the Coherence API at all.

Despite these misgivings, I decided to provide my support and told the application team manager that I'll try my best.

The system with problems was a multi-node Coherence cluster. When I took a look at the logs, all of them had these similar verbose GC output:

[GC [1 CMS-initial-mark: 1966076K(1966080K)] 2083794K(2084096K), 0.1923110 secs] [Times: user=0.18 sys=0.00, real=0.19 secs] 
[Full GC [CMS[CMS-concurrent-mark: 1.624/1.626 secs] [Times: user=3.22 sys=0.00, real=1.62 secs] 
 (concurrent mode failure): 1966079K->1966078K(1966080K), 6.6177340 secs] 2084093K->2084082K(2084096K), [CMS Perm : 13617K->13617K(23612K)], 6.6177900 secs] [Times: user=8.21 sys=0.00, real=6.62 secs] 
[Full GC [CMS: 1966078K->1966078K(1966080K), 4.1110330 secs] 2084093K->2084089K(2084096K), [CMS Perm : 13617K->13615K(23612K)], 4.1111070 secs] [Times: user=4.11 sys=0.00, real=4.11 secs] 
[Full GC [CMS: 1966078K->1966078K(1966080K), 4.2973090 secs] 2084092K->2084087K(2084096K), [CMS Perm : 13615K->13615K(23612K)], 4.2973630 secs] [Times: user=4.28 sys=0.00, real=4.30 secs] 
[Full GC [CMS: 1966078K->1966078K(1966080K), 4.1831450 secs] 2084093K->2084093K(2084096K), [CMS Perm : 13615K->13615K(23612K)], 4.1831970 secs] [Times: user=4.18 sys=0.00, real=4.18 secs] 
[Full GC [CMS: 1966078K->1966078K(1966080K), 4.2524850 secs] 2084093K->2084093K(2084096K), [CMS Perm : 13615K->13615K(23612K)], 4.2525380 secs] [Times: user=4.24 sys=0.00, real=4.25 secs] 
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid23607.hprof ...
Heap dump file created [2274434953 bytes in 28.968 secs]

This garbage collection log output tells me that they are using CMS for the JVM GC. The concurrent mode failure entries certainly grabbed my attention. Normally one would fix concurrent mode failures by tuning the CMS initiation occupancy fraction via -XX:CMSInitiatingOccupancyFraction flag, but in this case, looking that the heap numbers in the lines labeled "Full GC" showed that GC could not clean up any memory at all. So this problem could not be solved by GC tuning. By the way, for a great book on tuning garbage collection, I would recommend Charlie Hunt's book Java Performance.

My next step was to take a look a the heap. The heap was slightly over 2 GB, which was expected since Coherence cluster node was each configured with a 2GB heap. Well, that presented a problem for me because I'm still mainly working on a 32-bit Windows laptop. I needed to find a 64-bit system with preferably 4 GB of ram or more to look at this. Once I was able to get such a machine and fired up Eclipse Memory Analyzer Tool (MAT). Once I looked at the heap, it was pretty obvious what was the biggest memory offender. The biggest memory offender was a top level hashmap chewing up 1.6 GB of memory. Delving further into that hash map structure, it reveals that Coherence caching structure is a hash of hashes. Looking at the hashes, I notice that there were over 2000+ items in the top level hash. That would imply that there were over 2000+ caches in the Coherence cluster. Studying each individual cache, I would notice cache names like

alpha-FEB-21
alpha-FEB-22
alpha-FEB-23
alpha-FEB-24
beta-FEB-21
beta-FEB-22
beta-FEB-23
beta-FEB-24

and so forth. I ask the application team manager whether he expected to have this many caches in the cluster. The application team manager said no; he expected a much smaller set of caches. The application normally destroy caches older than 2 days. The developers provided me their code related to the creation and destruction of caches and I saw the following lines of code and it seems pretty innocuous:

    public static void destroyCache(String name) {
        Collection listOfCacheNames = getListOfCacheNames(name, false);
        Iterator iterator = listOfCacheNames.iterator();
        while (iterator.hasNext()) {
            String name = (String) iterator.next();
            NamedCache namedCache = CacheFactory.getCache(name);
            namedCache.destroy();
        }
    }

I went back to the memory analyzer tool and performed a GC to root analysis and saw the top level object that's holding onto this heap as:

com.tangosol.coherence.component.net.Cluster$IpMonitor @ 0x77ff4b18

with the label "Busy Monitor" next to it. This line item seems to suggest that there's a monitor lock on this cache. Looking at the Coherence API documentation, I see the following entry:

destroy

void destroy()

Release and destroy this instance of NamedCache.

Warning: This method is used to completely destroy the specified cache across the cluster. All references in the entire cluster to this cache will be invalidated, the cached data will be cleared, and all internal resources will be released.

Caches should be destroyed by the same mechansim in which they were obtained. For example:

new Cache() - cache.destroy()
CacheFactory.getCache() - CacheFactory.destroyCache()
ConfigurableCacheFactory.ensureCache() - ConfigurableCacheFactory.destroyCache()

Except for the case where the application code expicitly allocated the cache, this method should not be called by application code.

Looking at this documentation, we initially thought that since the cache was obtained via CacheFactory and therefore should be destroyed via CacheFactory ergo CacheFactory had a monitor lock on the underlying collections. The code provided by the developers used one mechanism to create the cache and another mechanism to destroy the cache so we presume that was the problem. So I implemented a test script to test out that theory and surprisingly, even destroying via CacheFactory, I still encounter out of memory issues. Only by clearing the cache before destroying the cache was I able to avoid out of memory errors. Here's the script that I developed in Clojure to test my theories:

(import '(org.apache.commons.lang3 RandomStringUtils) 
        '(java.math BigInteger)
        '(java.util Random Date HashMap)
        '(com.tangosol.net NamedCache CacheFactory CacheService Cluster))

(defn random-text [] (RandomStringUtils/randomAlphanumeric 1048576))
(defn random-key [] (RandomStringUtils/randomAlphanumeric 12))
        
(CacheFactory/ensureCluster)
(def buffer (new HashMap))

(defn print-horizontal-line [c] (println (apply str (repeat 80 c))))

(def caches '("alpha" "beta" "gamma" "delta" 
              "epsilon" "zeta" "eta" "theta" 
              "iota" "kappa" "lambda" "mu" "nu"
              "xi", "omicron" "pi" "rho"
              "signma" "tau" "upsilon" "phi", 
              "chi" "psi" "omega"))

(defn load-cache [cache-name n]
    (let [cache (CacheFactory/getCache cache-name)]
         (print-horizontal-line  "=")
         (println "Creating cache : " cache-name)
         (print-horizontal-line  "=")         
         (.clear buffer)
         (dotimes [_ n] (.put buffer (random-key) (random-text)))
         (.putAll cache buffer)))


(defn recreate-oom-problem [cache-name]
    (let [cache (CacheFactory/getCache cache-name)]
         (load-cache cache-name 200)
         (.destroy cache)))
         
(defn try-fix-oom-1 [cache-name]
    (let [cache (CacheFactory/getCache cache-name)]
         (load-cache cache-name 200)
         (CacheFactory/destroyCache cache)))

(defn try-fix-oom-2 [cache-name]
    (let [cache (CacheFactory/getCache cache-name)]
         (load-cache cache-name 200)
         (.clear cache)
         (CacheFactory/destroyCache cache)))
         
; Test run recreation of original problem.  Was able to reproduce OOM issues         ; 
(doseq [cache caches] (recreate-oom-problem cache))

; Surprise! Still have OOM issues
(doseq [cache caches] (try-fix-oom-1 cache))

; No longer have OOM issues, but memory is still leaking (slowly)
(doseq [cache caches] (try-fix-oom-2 cache))

However, I still suspect memory leaks, it's just that my memory leak is a lot smaller now. To verify that I had a memory leak, I would run my Clojure test script and the deliberately create and fill a cache without clearing it. I then forced a full garbage collection followed by a heap dump. In memory analyzer tool, I would look up the cache that I did not clear, and list all the incoming references. Then I would look for a HashMap in the incoming references and select one of those and check for outgoing references. And in that outgoing references, I could see that the key contains the name of a cache that I had called CacheFactory.destroyCache() on and the retained heap sizes range anywhere from 24 to 160 with the sizes that seems proportional to the size of the cache name.

In conclusion, it would seem Oracle Coherence does have a memory leak issues with the cache creation and destruction process. If we clear the cache before destroying the cache, I suspect it would be a long time before the memory leak is even noticeable by this particular application.

To verify that this leak did not exist in the older 3.1 version, we ran this test code on and and was unable to reproduce the out of memory errors. We also tested this against Oracle Coherence 3.7.1 and was unable to reproduce the out of memory error. So, it looks like that this memory error is specific to Oracle Coherence 3.6 only.

Throughout this entire process, I thought that the secret sauce that enabled me to quickly learn Coherence, reproduce and troubleshoot the Coherence out of memory problem was Clojure. Clojure allowed me to interactively manipulate Coherence clusters and explore the API, which would have been a lot slower if I had to go through the normal edit-compile-run cycle with plain old Java.

Coding for reuse versus coding for specific use

2012-02-21T06:00:00.000-08:00

One day, a C# developer came to me asked me to troubleshoot their application, which involves some data transformation code. In the course of troubleshooting this particular application's runtime issue, I saw this particular code implementation:

private string[] SplitTextwithSpecialCharacter(string lineFromGUI, int splitLength) 
{

    List<string> lines = new List<string>();
    int start = 0;


    while (start < lineFromGUI.Length)
    {
        string temp = string.Empty;
      
        //if length is longer it will throw exception so size the last segment appropriately
        string line = lineFromGUI.Substring(start, Math.Min(splitLength, lineFromGUI.Length - start));
        
        //temp = line.Replace(SpecialCharacter, string.Empty);

        temp = line.Replace("^", ""); //New 6/30
        if (temp.Length > 0) 
        {
            //lines.Add(line.Replace(SpecialCharacter, string.Empty));
            lines.Add(temp);
        }
        //lines.Add(line);
        start += splitLength;
    }
    return lines.ToArray();
}

When I saw this code, I inwardly cringed due to the very narrow scope in the code implementation to solve a seemingly generic problem. This code screams for refactoring to me. From the commented out section of the code, I could see that one of the developers tried to implement some reuse before reverting back to the specific use.

I see a couple of issues with this particular implementation. The first issue being that this method is trying to do two very different things; one is trying to split the text into lines and the second is removing all the caret characters from the text. In my opinion, these 2 different operations should be separated out into 2 different methods. The second issue is that the caret character removal function should be generalized such that you should be able to remove any character or perhaps even a set of characters beyond just the caret symbol. In the specific implementation, what would happen if they needed code to remove an asterisk instead of caret? What would happen if they need to remove more than one special character? Do they add additional methods? Do they modify the existing method? If they modify the existing method to take parameters, then they would have to change all the existing client code that calls this method. So it would seem to be easier for future maintenance to go ahead and make these 2 functions reusable.

Here's how I would refactor this particular method into some utilities namespace and use extension methods to add capability to the string object. Here's how that code would look like:

public static class Extensions
{
    public static string FilterOut(this string line, ISet<char> filter)
    {

        if (line == null ||  filter == null || filter.Count == 0) return line;


        return new String(line.Where(c => !filter.Contains(c)).ToArray());
    }


    public static IList<string> SplitIntoLines(this string data, int splitLength) 
    {

        // Check preconditions
        if (data == null) throw new NullReferenceException();

        if (splitLength <= 0) throw new IndexOutOfRangeException("Must be greater than 0!");

        IList<string> lines = new List<string>();

        while (data.Length > splitLength)
        {
            string line = data.Substring(0,splitLength);
            data = data.Substring(splitLength, data.Length-splitLength);
            lines.Add(line);
        }
        lines.Add(data);
        return lines;
    }
}

With this extension method, you can now use this utility method as follows:

int linesize = 30;
ISet<char> filter = new HashSet<char> { '^', '*', '&', '$' };
string[] results = data.SplitIntoLines(linesize)
                       .Select(line => line.FilterOut(filter))
                       .ToArray();

Detect multiple occurrences of Java classes in jar files

2012-02-04T07:43:00.000-08:00

A developer came to me for help and said that a recent upgrade to a newer Java library broke the code. The developer was getting was a class not found exception. The developer said that the old jar file was replaced with a newer jar file. I know this same library upgrade in other applications did not have any problems, so this sounded like an issue with duplicate occurrence of the same class in the classpath. Since my Ruby Programming Language book was in handy access when this developer came to me, I ended up writing a small ruby script that went through all the jar files packed in the web archive and dumped an output of the fully qualified class along with the jar file that it can be found in and a count of the number of occurrences. Here's the ruby script running on a Windows platform:

java_home="C:\\sdk\\jdk1.6.0_24\\bin"

classmaps = Hash.new(0)

Dir["*.jar"].each do |file|
  cmd = sprintf("%s\\jar  -tf %s",java_home,file)
  lines = `#{cmd}`
  lines.each do |line|
    if line =~ /.class/
      key = line.chomp
      if classmaps.key?(key) then
        old = classmaps[key]
        data = sprintf("%s,%s",old,file)
        classmaps[key] = data 
      else
        classmaps[key] = file
      end
    end
  end
end


classmaps.each do |k,v|
    tokens = v.split(",")
    printf("%s,%i,%s\n",k,tokens.size,v)
end

I then loaded the output file in csv format into Excel and sorted the occurrences in descending order and was flabbergasted to find numerous entries that look something like the following entry:

Class Name	Occurrence	Found in Jar files:
kodo/jdo/KodoExtent.class	6	mylib.jar	mylib.jar	mylib.jar	kodo-api.jar	kodo-runtime.jar	kodo.jar

Somehow, for this application team, they were able to add the same class to the same jar file multiple times. I never thought that it was possible to add the same class to the same jar file multiple times nor would I ever want to do that. When I finally confronted the application team about this, they recognize that their build process is broken and need to fix their build process. But as a quick test, the developer removed some of the duplicate classes related to the library upgrade and the problems went away. Until Java 8 is introduced with Modularity capabilities, this tool has will be a handy way for me to check duplicate classes given a list of jar files.

Translating If-Then-Else Control Flow Idiom to F#

2012-01-20T17:52:00.000-08:00

I was reading through Juval Löwy's Programming WCF Services book and wondering if I should do a series of WCF blog posts in F# based on Löwy's book when I ran into a common construct found in C# programs. That construct looks something like the following C# code:

    public static void MyMethod(String oldstuff, String newstuff, bool flag)
    {
        if (oldstuff == null)
         throw new Exception("oldstuff is null!");

        if (newstuff == null)  {
            DoSomething("Default");
            return;
        }
        if (flag == false)  {
            DoSomething(oldstuff);
            return;
        }
        DoSomething(newstuff);
    }

This is a construct that I oftened have used in the past and have never thought about it much. But when you translate the above code directly into F#, it becomes a lot more verbose because F# requires you to implement the then clause. A direct translation to F# as follows:

let mymethod oldstuff newstuff flag =
    if oldstuff = null then 
        raise (new Exception("oldstuff is null!"))
    else
        if newstuff = null then
            DoSomething("Default")
        else
            if flag = false then
                DoSomething(oldstuff)
            else
                DoSomething(newstuff)

If I had a lot of these if-then-else statements in my C# method, then my F# version would disappear off the screen to the right if I tried to implement it by direct translation. I thought about how I could implement this in F# and came up with this following possibility:

let revised_mymethod oldstuff newstuff flag =
    let (_,action) =
        [(oldstuff=null,  lazy (raise (new Exception("oldstuff is null!"))));
         (newstuff=null,  lazy (DoSomething("Default")));
         (flag=false,     lazy (DoSomething(oldstuff)));
         (flag=true,      lazy (DoSomething(newstuff)))]
        |> List.filter fst
        |> List.head
    action.Force()

Rewriting the C# code in this fashion makes me think of rules engines and after refactoring out some common code, I could rewrite the above F# code as follows:

let followrules (xs:(bool*Lazy<unit>) list) =
    (xs |> List.filter fst |> List.head |> snd).Force()

let revised_mymethod2 oldstuff newstuff flag =
    [(oldstuff=null , lazy (raise (new Exception("oldstuff is null!"))));
     (newstuff=null,  lazy (DoSomething("Default")));
     (flag=false,     lazy (DoSomething(oldstuff)));
     (flag=true,      lazy (DoSomething(newstuff)))]
    |> followrules

With this new construct, I can easily re-arrange the order of evaluation, add or remove new conditions. This new construct just seems to have more advantages than the old if-then-else construct in F#.

Testing Coherence with Clojure

2012-01-11T10:32:00.000-08:00

A developer came to me the other day asking for help in diagnosing some issues with their application and the interaction with Oracle's Coherence product. I wanted to write some testing harness to quickly test some Coherence configuration and gave some thought about how I would go and try to replicate the issues that the application had. I wanted a REPL environment so that I can interactive manipulate the Coherence API and dump outputs on demand. I decided to use Clojure to experiment with Coherence, although I could have used JRuby, Jython, Groovy or Scala. From purely a familiarity perspective, I would rank my usage of these listed languages in the order of Ruby first, Python second, Groovy third, Clojure fourth and Scala last. But for some unknown, deep-seated and probably emotional reasons, I like Clojure more and relish the opportunity to use it. One of the first thing I tried with Clojure and Coherence is to perform a timing test on adding data to a 2 node distributed cache in serial vs concurrent mode. Here's the example Clojure script:

(import '(org.apache.commons.lang3 RandomStringUtils) 
        '(java.math BigInteger)
        '(java.util Random Date HashMap)
        '(com.tangosol.net NamedCache CacheFactory CacheService Cluster))

(CacheFactory/ensureCluster)
(def cache (CacheFactory/getCache "sandbox"))

(defn random-text [] (RandomStringUtils/randomAlphanumeric 1048576))
(defn random-key [] (RandomStringUtils/randomAlphanumeric 12))


; Testing serial puts
(new Date)
(dotimes [_ 200] (.put cache (random-key) (random-text)))
(new Date)

; Testing concurrent puts
(def buffer (new HashMap))
(new Date)
(dotimes [_ 200] (.put buffer (random-key) (random-text)))
(.putAll cache buffer)
(new Date)

Running this code showed a 2x gain in speed of data load. On one of the Coherence nodes, I had JVisualVM connected to it and watched the realtime GC behaviors with VisualGC. It has been fascinating to watch the behaviorial differences between serial vs parallel data load and the memory activities of the Coherence node when my Clojure script was idle. I hope to conduct more tests in the future looking at GC behaviors and leverage my Clojure script as load driver in my testing efforts and assist me in GC tuning of Coherence instances.

Revisiting the SharePoint collection adapter for F#

2011-11-11T16:41:00.000-08:00

Recently, I got to work with SharePoint 2007 again and I revisited a previously written blog Exploring SharePoint 2007 Object Model with F#. In that blog, I complained about the fact that SharePoint collections did not implement IEnumerable, causing me to implement a type specific collection adapter as pilfered from Asfar Sadewa. However, as I started to work with more different SharePoint collections, I realized that I need a generic SharePoint collection adapter. But once again, I was foiled by SharePoint's library designers. I wanted to ensure some level of type safety for my generic collection adapter and wanted to bound the collection item generic type variable so the client code is restricted to types such as SPWeb, SPList, etc. Looking up the hierarchy, I see the type SPSecurableObject. Unfortunately, SharePoint library designers did not make SPSecurableObject visible to others and the next level up that hierarchy is the Object type. So I'm stuck with an unbounded generic type for the collection item in my generic version SharePoint collection adapter. Actually, what I really want is to ensure that the collection item type is consistent with the collection type. I haven't quite figure out if that's even possible to apply that kind of type parameter constraints in C#. Here is generic version of the SharePoint collection adapter (with the runtime type error leakages) implementation:

using System;
using System.Collections.Generic;
using Microsoft.SharePoint;

namespace SharePoint.Utility
{
    // I really wanted to apply constraint to TPart to SPSecurableObject
    public class SPCollectionAdapter<TPart, TCollection> : List<TPart> where TCollection : SPBaseCollection
    {
        private TCollection _listCol;

        public SPCollectionAdapter(TCollection listCol)
        {
            _listCol = listCol;
            Refresh();
        }

        private void Refresh()
        {
            this.Clear();
            this.Capacity = _listCol.Count;

            foreach (TPart item in _listCol)
            {
                this.Add(item);
            }
        }

    }
}

This allows me to write my F# code as follows:

open Microsoft.SharePoint
open SharePoint.Utility

let path="http://localhost/"
let collection = new SPSite(path)
let site = collection.RootWeb
let lists = SPCollectionAdapter<SPList,SPListCollection>(site.Lists)
Seq.iter (fun (x:SPList) -> printf "%s\n" x.Title) lists

Porting the Log Analysis Code to Haskell

2011-05-09T17:24:00.000-07:00

My coworker approached me the other day and ask what open source log analysis tools would I recommend. I personally do not have much experience with a general purpose open source log analysis tools so I would have probably recommended him to take a look at Splunk. Since I've recently written a customized log analysis software, I became curious and asked him what he intend to do with the log analysis software.

My coworker said that he needed to analyze Tibco EMS logs. Tibco EMS logs incoming messages in the order it receives. My coworker is interested in a set of related messages that is identified by the message ID tag. His particular issue is that the logs entries that he's interested in are interspersed with other log entries that he's not interested. He wanted a log file where the log entries are grouped by message id in historical order.

Once I understood his needs, I realize that he did not need the Splunk and that I could quickly adapt my F# log analysis software written in the previous blog post for his need. When I gave him the modified F# code, he asked me if I could port it to Linux. That threw me for a loop. I briefly entertained the idea of building a Mono system and compile F# on Mono but decided against it for now. I thought it would be easier to just port it to Haskell, which I already have on Linux.

Here's the ported Haskell log analysis software with modifications to work with Tibco log entries.


import Data.Time.Calendar
import Data.Time.LocalTime
import Data.Time.Parse
import List
import System.Environment

type Category = String
type Entry = [String]
type TimeStamp = (LocalTime,String)
type LogHeader = (TimeStamp, Category)

alphaTime = LocalTime (fromGregorian 2000 1 1) midnight 

data LogEntry = LogEntry (TimeStamp, String) [String]
                deriving (Show)

{- Grab label -}
categorize (_ : _ : label : _) = label
categorize words = ""

{- Grab timestamp -}
timestamp (date : time : _ )  = strptime "%Y-%m-%d %H:%M:%S" (date ++ " " ++ time)
timestamp words  = Nothing
           
{- header :: String -> (String, Maybe (LocalTime, String)) -}           
header line = (timestamp tokens, categorize tokens)    
    where tokens = words line

{- Concrete implementation of Tibco log parser -}    
logparser :: [String] -> LogHeader -> [String] -> [LogEntry] -> [LogEntry]
logparser (line : rest) xheader entry entries  = process (header line)    
    where process (Just (ts),label) = 
              logparser rest h [line] ((LogEntry xheader (reverse entry)):entries ) where h = (ts,label)
          process (Nothing,_) = logparser rest xheader (line : entry) entries

logparser [] xheader entry entries = reverse ((LogEntry xheader entry) : entries) 

{- Utility method  to pull items out of LogEntry -}
entry (LogEntry _ entries) = entries    
category (LogEntry (_,label) _) = label

{- comparator based on category -}
categorysort (LogEntry (_,a) _) (LogEntry (_,b) _) 
    | a > b = GT
    | a < b = LT
    | otherwise = EQ

parselog parser lines = parser lines ((alphaTime,".000"),"STARTFLAG") [] []
    
processlog = unlines
             . map (unlines . entry)
             . sortBy categorysort
             . (parselog logparser)
             . lines 

main = do (filename:_) <- getArgs
          contents <- readFile filename
          putStr (processlog contents)

Log Analysis with F#

2011-03-21T12:36:00.000-07:00

I am often pulled into investigative teams to resolve performance issues. In one particular instance, I was involved in troubleshooting a Java-based batch job that had experience performance issue after it migrated from an older database platform to a newer database platform. Typically in these performance related issues, I would analyze the logs to see if I can spot tell tale signs of performance issues. Some of these log files can range up to millions of lines of text. In this particular case, I received a log file with of almost 4 million lines of text. I do not want to wade through these log files by hand. In the past, I would have use a scripting language like perl or ruby to help parse these logs. But lately, I have leaned more towards F# to analyze log files. The following describes my experience with developing my log analysis software in F#.

The log files that I have received have sample entries with the following pattern:

02/04/11 21:15:55 xyz 123 CATEGORY_LABEL Lorem ipsum dolor sit amet 
    Lorem ipsum dolor sit amet, consectetur adipiscing elit. 
    Aliquam varius mollis purus, vel molestie lorem pellentesque sed.

where CATEGORY_LABEL categorizes the jobs performed and each log entry always starts with a timestamp. Typically, multiple jobs are running at the same time so the log entries have data interspersed. To start this effort, I want to capture the timestamp information, the category label and the entire log entry. To model this, I created the following type to capture a log entry:


type LogEntry =
    {
        Timestamp : DateTime;
        Category  : string;
        Entry     : string list;
    }

I want to be able to easily access Timestamp and Category label information while retaining the entire log entry for that timestamp/category. I decided to model the log entry as a list of strings. I had hope to leverage list operations later on to perform filtering capability. One of the drawbacks to this model for log entry is if the log file had text that split across 2 lines.

The first problem is to extract the timestamp and the category information. To capture the timestamp information, I implemented the following functions:


let parsedate datestring = DateTime.TryParse(datestring)

let getTimestamp (words:string []) =
    if words.Length > 2 then
        let timestamp = sprintf "%s %s" words.[0] words.[1]
        match parsedate timestamp with
            | true, d -> Some(d)
            | false,_ -> None
    else None

I leveraged F#'s Option type to return either a valid timestamp or None if no timestamp are found. You can see later how I use this as a filter to determine if I'm at the start of a log entry or not. Here's the code to get the category label:

let getCategory (words:string []) =
    if words.Length > 4 then 
        let tokens = words.[4].Split[|':'|]
        tokens.[0]
    // Once in a while, there is log entry with no labels
    else ""

I know that the category label always is the 5th token, but once in a while the category label shows up in the log as CATEGORY_LABEL:Lorem, therefore I have to split the word with the token ":" to extract just the category label.

With these 2 functions, I can now build my getheader function which would extract the timestamp and category label:

    
// Get the timestamp and the job category     
let getHeader (line:string) =
    let words = line.Trim().Split[| ' ' |]
    let timestamp = getTimestamp words
    let category = getCategory words
    (timestamp,category)

I now have all the pieces to be able to build my log parser:


// Utility function to create log entry
let createLogEntry (timestamp,category)  entry =
    {Timestamp = timestamp;  Category=category;  Entry = List.rev entry}      
    
    
// Core parser engine to parse the log files
let rec logparser lines oldheader entry (acc:LogEntry list) =
    match lines with
    | line :: rest -> 
        // Check with we got new timestamp with this line
        let header = getHeader line
        match header with
        | (Some(ts),category) -> 
            // Found new timestamp,create log entry from the collected lines
            let logentry = createLogEntry oldheader (line::entry)
            //  Begin new log entry collections
            logparser rest (ts,category) [] (logentry::acc)
        | (None,_) -> 
            // Add new entry to line
            logparser rest oldheader (line::entry) acc
    // We're done with all line processing!
    | [] -> 
        let logentry = createLogEntry oldheader entry
        logentry :: acc |> List.rev

This log parser would recursively processed each line in the log file and build a list of LogEntry to return. When I find a header, I would begin a new LogEntry. Because I built the list using the :: (cons) operator, I need to reverse the list when I finish building the list. I would use the above parser engine as shown in the following code, which would return to me list of all log entries.

let parselog file parser =
    let lines = File.ReadAllLines(file)
    let past = DateTime.Now
    parser (Array.toList lines) (past,"") [] []

Now I have all the pieces to begin performing log analysis. I typically begin by chopping the log into smaller pieces. In the following example, I'm looking for a specific category on specific dates, one before the database migration and one after the database migration.

let (_,premigration) = DateTime.TryParse("03/11/2011")
let (_,postmigration) = DateTime.TryParse("03/14/2011")

// Extraction engine to pull logs related to specific job @ specific date
let extract targetdate label file processor =
    parselog file logparser
    |> List.filter (fun x -> x.Timestamp.Date.Equals(targetdate))
    |> List.filter (fun x -> x.Category.Trim().Equals(label))
    |> processor

// End log entry text to file
let send2file file entries=
    entries 
    |> List.map (fun x -> x.Entry)
    |> List.concat
    |> List.toArray
    |> writefile file

// smaller working file - premigration for JOB A
(send2file @"JOBA.premigration.log") |>
extract premigration.Date "JOBA" @"jobs.log" 

// smaller working file - postmigration for JOB A
(send2file @"JOBA.postmigration.log") |>
extract postmigration.Date "JOBA" @"jobs.log"

Here's another example usage where I wanted to grab clusters of log entries along the time dimension (e.g. unique sets of Timestamp,Category pairs).

let getEntryClusters targetdate infile outfile  =
    parselog infile logparser
    |> List.filter (fun x -> x.Timestamp.Date.Equals(targetdate))
    |> List.map (fun entry -> (entry.Timestamp,entry.Category))
    |> Set.ofList |> Set.toList
    |> List.map (fun (ts,category) -> sprintf "%s,%s" (ts.ToString()) category)
    |> List.toArray 
    |> writefile outfile

getEntryClusters postmigration.Date @"jobs.log" @"clusters.log"

The interactive capability of F# in Visual Studio made building this log analysis tool a lot easier than I expected. While I can do similar interactive work with Ruby's irb, or Python's interactive mode, Visual Studio and F# just makes it feel much more productive than either of Ruby or Python's interactive environment. If I had to analyze another log file with a different format, I can certainly reuse my code and maybe just changed the logparser implementation. Indeed, just a few days after later, I was able to apply this code to look into some Tibco logs with minimal amount of changes.