Once upon a time...

Ping

2014-03-08T10:01:00.001-05:00

Been a long time since I posted anything over here. No better day than today to write an update. I am not really sure why I don't feel like writing these days (or reading for that matter) - I guess am just a bit dazed and don't even know what I want to write about. Feeling tired and run down right now, don't know if it cause I've been sick or it's just a state of a the mind!

Lone traveller

2013-03-08T11:49:00.000-05:00

The calmness at the railway station was disturbed by the loud siren of the train approaching. All of a sudden, everything sprung to life as if they have found their purpose; as if that’s the moment they all had been waiting for. I looked at the clock hanging at the platform, it was close to 2 AM, train was just late by about 4 hours, nothing special if you are used to how things operate in India, being late and laid-back is the way of life here. Everyone expects it, in fact, I guess quite a few people in India will miss their trains (or bus or whatever) if those things started being on time. The train made a stop and there was a mad rush everywhere, people who had reached their destination were in a hurry to get down but they were being pushed back to the compartment by the crowd which was in a rush to get inside! I don’t know why everyone over here is in such a rush, as-if you get a prize for being the first to climb up or get down, maybe you feel a sense of pride if you are the first. I didn’t have to put in any effort in boarding the train, I was pushed into the compartment by the crowd behind me trying to get in.

I settled myself besides the window and started looking at the lights from lamp posts outside. As the train slowly pulled away from the station, the lights went dimmer and dimmer and by the time the train had picked up speed, I was staring outside at the dark emptiness. I closed my eyes and the same set of questions cropped up again in my mind – Where am I headed? Why did I even board this train? Almost everyone who cared about me had called me out of my mind when I had said that I want to leave everything behind and start afresh, and for that I needed a little time off to do some soul searching, to find out what my higher purpose in this life was – the mountains seemed to be the best place for that. There is some sense of calm about the mountains, the sound of the river streams as they come down the mountains, that you just want to lose yourself in them. By the way, I didn’t understand when people said why I want to leave everything behind, heck I had nothing unless living an aimless life and going through the motions day-in day-out just to keep you afloat is considered everything. Yes, I had a well paying job; a plush house loaded with amenities but is that all that you need? Isn’t there more to life than things that money can buy?

It wasn’t an easy decision to just let go of a life that you are used to even if it is not worthwhile, we humans have the tendency to not come out of our comfort zone, we just hate changes even if they it is good for us. I took a long hard look at my life as it had panned out over last 5-8 years and didn’t find even a single moment that I cherished, a moment which made me smile – yes, there were good times and there were bad times and really really bad times but there was not even a single moment which brought a sense of satisfaction or inner peace. I think I had become like someone who is terminally ill, you don’t have any dreams, anything to look forward to, and you just live everyday as it comes, just trying to survive one more day. I knew I had to break out of that life as I don’t think I could have sustained that lifestyle any longer. And now that I have left my old life behind, I can only hope that the soul searching would lead me somewhere. Perhaps, I will love the mountains & the new life so much that I wouldn’t even want to return back (anyway you need to have something to return back to). Sometimes I do wonder though, if this soul searching is even needed? I already know what I need, it’s just that I am not lucky enough to have what I need. There is a saying – “you don’t always get what you want”, I guess in my case “I don’t ever get what I really want” would be fitting. Perhaps, I am indeed terminally ill.

MongoDB Tidbits - PHP, Mongo Collections and ODM

2012-05-22T04:17:00.000-04:00

So I jumped onto the NoSql (Not-Only Sql) bandwagon and off-late have been dabbling with MongoDB and PHP. Coming from a RDBMS background, NoSql is for sure a different paradigm and requires thinking differently. These series of posts are on my experience with MongoDB, the problems that I’ve faced and the solutions/workarounds that I’ve found so far. All the code samples in these posts are on 64-bit Linux version of MongoDB 2.0.3 and PECL Mongo driver 1.2.9. These posts assume basic familiarity with MongoDB.

Background

The reason why I started even looking outside of RDBMS was due to loosely defined schema that I had to support. Basically, on our website a user can submit activities, now these activities can be on different types and the activity attributes change based on the activity type. We started off with MySql DB and soon started realizing that supporting different schemas in a RDBMS was a bit painful. The most popular approach for modeling such kind of schema seems to be EAV (as used by Magento) but I just found it to be a bit too complex. We actually started serializing our entities into xml and dumping them into a My Sql column but I realized that by doing so, I am not getting any benefits of using a RDBMS (no referential integrity etc).

Why Mongo?
Once I decided that RDBMS was a no-go, I started looking out for alternatives in the NoSql world. I had tried playing with Cassandra and Hadoop/HDFS earlier but felt they had a steep learning curve. Also, I feel Cassandra and Hadoop/HDFS are more suitable for applications dealing with huge amount of data, given their distributed nature and complex processing (great support for Map-Reduce). I finally evaluated MongoDB, CouchDB and Redis – found Redis to be a glorified Key-Value store and MongoDB to be closest to a RDBMS (you can have indexes, dbrefs) without being a RDBMS, making it a little easier for people with RDBMS background to learn it.

Mongo Schema Design

After picking on Mongo and installing it, the first choice that you have to make is on the schema design and how to define relationshipts between entities . There are two ways in which you can define relationships –

Embedding: A document becomes a subdocument of another document. You can embed as many levels deep as you wish. Warning – Embedding generally works great if you are embedding only up to one level as Mongo currently does not have good support for querying/updating attributes which are nested multiple levels deep – more on it in a subsequent post where-in I ran into the issue with $ operator.

2. Linking: Documents are different entities (part of different collections) and are linked by their MongoIds (_Id). Enforcing the relational integrity is primarily the responsibility of the client application.

Based on the docs, I also followed the same principle: if a relationship between two entities is “Composition” , I embed sub- entitiy subdocument else I add it to a different collection and link them using MongoId. Below is one example of composition and linking –

> db.activities.find({}).pretty();

activityTitle:”This is example”,

tags:[“tag1”,”tag2”,”tag2”],

submittedBy:23

….

Here tag is a subdocument (1:n relationship), whereas submittedBy has the reference to the Id of the user stored in db.users. Tag by itself can be a complex object (i.e. it can have its own attributes like tag:[{tagId:1,name:”tag1”,submittedBy:23},…].

That’s pretty much on the schema design; currently our schema design is pretty straightforward with collections for “top-level” entities like activities, users, keywords (more on this later) and subdocuments for sub-entities like tags.

One caveat: Mongo column names are case sensitive, which is different from MySql. Quite a few times in the beginning, I have had the col name in wrong case and wasted bunch of time trying to figure out what’s wrong with my query.

ODM (Object Document Mapper) Strategy

Once the Mongo collections are finalized and the PHP DTO/Models (Data Transfer Object) classes defined, the next step is to figure out how are we going to map our models to Mongo collections and retrieve/persist the same? The PECL driver is pretty flexible but only support retrieving/persist PHP associative arrays, so there has to be an adapter in between which converts our PHP class into an associative array. There are few mappers available already like Doctrine and Php-ODM but somehow I wasn’t very comfortable with them – doctrine: seemed to be a bit too heavy, whereas Php-ODM relies on storing property in an internal array; what we wanted was a way to store protected variables and have getters and setters so that we can validate the values and typecast them. Also, we had the need for being able to return only a subset of properties (in case of partial update). Our implementation is pretty simple: every model inherits from BaseModel which has a method toArray(). The toArray() just calls get_object_vars() and takes two optional arrays as parameter: includeItems and excludeItems. Below is the implementation of our toArray() method –

Our Models have protected variables for properties which need to be serialized and override the toArray() method if needed.

That's pretty much it for our ODM

Still searching...

2012-03-08T01:25:00.001-05:00

Another year, same day when I think about where I am, what am I doing and what do I want to achieve. Another year, same day and I still haven't found answers to any of these rhetorical questions. Sometimes, i do wonder if I'm being even reasonable but then "what's rhyme or reason to a fool or a dreamer?" Who am I being these years, a fool, a dreamer, maybe both? Anyway, reasoning is a relative term, I know, I could convince others based on my reasoning if they are at least willing to listen.
I'll keep waiting for answers...there's nothing else for me to do, anyway.

DEL-ORD

2011-12-05T19:06:00.001-05:00

Seven years went under the bridge like time standing still...

Funny how, at times, life moves backwards and takes you to places you've been trying to run away from all this while. Perhaps, it's just life's way of reminding you of things that you've been missing and yearning for deep inside.

life

2011-11-01T04:27:00.001-04:00

Guess, woke up on the wrong side of the bed today - feeling little down and out. Don't know why but I have these phases where I just feel run down - times when I look back at my life and don't feel really great about where I'm headed. I know that it's just a passing phase and soon I'll be back to normal - living life day by day. The thing about living your life in a way which doesn't excite you or keep you involved everyday is that it's just not sustainable, one day, you'll break out of it for sure. There's no point in continuing with something which has long been broken long ago. The issue is with finding out the thing that moves you, that makes you feel good but as I've always quoted - "I can't tell you what I want from life but I do know for sure what I don't want!". Maybe, I do know what I want but you can't always have what you want, can you?

Windows Phone 7–A Review

2011-09-19T12:56:00.001-04:00

It’s been more than 3 months now that I’ve been using a WP7 and I guess now I have enough idea about what I like about WP7 and what I don’t. This post is a mini-review on the review and I would not touch upon the hardware much. Also, I haven’t upgraded to Mango (WP 7.5) so I won’t talk about Mango much, though most of the stuff that I’ll be writing about would be applicable even for Mango.

Preamble – I’ve been a Windows Mobile user since quite a while and have had Windows Mobile powered devices since Windows Mobile 2003 SE. I still own quite a few Windows Mobile powered devices from Windows Mobile 5 to Windows Mobile 6.5. I have also played with iOS 4 and Android (2.3) powered devices.

The Likes -

Firstly the good things about WP7 – the OS is actually quite snappy and live tiles is a pretty cool and refreshingly different idea than almost all the other mobile devices out in the market. The lock screen on WP7 is the best with all the important stuff right there on the lock screen itself including missed calls, email/text notifications and calendar stuff. Native integration with the cloud and social networks (facebook, twitter, linkedIn) is pretty slick too.

The Dislikes or things that need to be better -

This is going be a much bigger list than the “likes” list – not because there are more things to dislike about this OS but because some of the things are simply annoying and make you feel like pulling out your hair

No Outlook Backup - I have been used to syncing my older WM phones with Outlook – everything on my phone was backed up including Contacts, Tasks/Calendar using Outlook. I did try backing up on the cloud using Live Mesh but somehow was more comfortable with backing up to a local computer which I use daily. The first thing that I did when I got my new WP7 phone was to connect it to the computer and heck, figured out there is no Outlook sync – so there was no way for me to get my contacts back on the phone. Quite stupid given that Outlook is still the best personal manager (email, tasks, contacts and calendar) available and I use it almost daily. I don’t think it can get any more stupid that getting your contacts synced from Facebook/Google/Hotmail takes a matter of minutes but syncing with your own personal computer is impossible. So, I had to figure out a way to sync my Outlook contacts/calendar with the cloud – in the end I had to use GO Contact Sync to sync my contacts with my Google a/c so that I can get them on my phone.
No Text Message Backup - The text messages on my WM 6.5 were backed up using MS MyPhone. In WP7 there is no way to backup text messages so one hard-reset and all your messages are lost forever!
Look ma no paste – I know there is copy-paste support since NoDo but what I am talking about is the paste support in the dialer app. I don’t know what the devs who disabled this support were smoking but there have been numerous instances where the phone number on a website is not recognized by the OS and the only way to currently dial such number is to write it down somewhere (or use speech) and then dial it manually. Yes, there are 3rd party apps which let you paste a number, but seriously, why can’t the native dialer app have this?
No Tasks Support – Guess fixed in Mango but really silly that there is no way to sync your tasks and there is no native task app (with reminders) in WP 7.0.
Text Contact Details – If you want to share a contact’s details with somebody else on WP7 then you are quite out of luck – guess sharing contacts via text messages is not considered very social any longer – you should just share them on the cloud somewhere!
International Assist – Perhaps I should have RTFMed before making a phone call but this feature gave me headaches when I tried reaching a local 1-800 number and the dialler kept on adding a + before the 1 (i.e. making it an international call). A frantic Google search made me realize that it’s a feature! I hate it when some apps try to outsmart the user. Ideally, the International Assist should be turned off by default and not turned on.
Mail an attachment – The only attachment that you can send from the Pocket Outlook (or whatever it’s called) is of type pictures! I know I can send Office documents using the Office hub but seriously why I can only attach pictures from the mail client is beyond me.
Half Baked Call History – The call history on WP7 is like one endless list of calls without any support for fitlers like Incoming, Outgoing and Missed. I can perhaps live without the filters but there is no way in the Call history to tap on a number and view the call history log just for that number. The call history by number is important when you do get a missed call from an unknown number and you want to check if you have received any other calls from that number or not.
Lack of Customization/ General UI observation – Let’s face it, users love customizing their phones – customization is what makes a phone truly yours. Unfortunately, the only changes to the appearance that you can make is altering the accent color (from a list of predefined ones) and choosing a light v/s dark theme.

The live tiles home screen wastes a bit of real estate by showing the right arrow always. I have never clicked on the arrow ever and have always swiped to get to the app list – so not sure if that arrow is needed. Perhaps, the navigation arrow can be removed all together giving more horizontal space to the tiles. Maybe it’s just me but I somehow preferred the honeycomb UI of the 6.5 for the app list instead of an endless list (with jumplist in Mango) of WP7.
Capacitive Hardware Buttons – Maybe I am nit-picking but there should be some way of at least disabling these buttons while you are playing a game like Fruit Ninja – so many times I’ve had the Bing Search open because I moved my finger accidently on the Search Icon while playing.
Zune Now Playing Artist Info – The Zune Now playing integration thingie (where it pulls up artist information from the marketplace) has never worked for me even though I have an US Zune a/c with US mailing & billing address. Funnily, Zune has no issues in pulling artist information from marketplace on my computer.

That’s pretty much it – looking at the “dislikes list” it might look like that I have more to bitch about WP7 than what I like but the fact is I actually find WP7 to be quite slick and if the above inconveniences are fixed than it’s going to be quite a force to reckon with in the Mobile space.

* Image courtesy – Dell.

Another (Elegant) Pairing Function

2011-07-16T12:21:00.001-04:00

In my last post I talked about Cantor Pairing function for uniquely pairing two integers into a single number. Unfortunately, I ran into some overflow issues with that when generating pairs for very big integers like int.Max – luckily, I don’t have to deal with such large numbers in my application!

Anyway, in case you do have to deal with very large numbers and are looking at reducing them into a single number then there’s another pairing function called Elegant Pairing Function (warning pdf), which lets you reduce 2 non-negative integers to a single pair.

I did run into some issues with rounding off errors while generating the square root due to the precision of the double data type (Math.Sqrt takes a double) causing the Math.Ceil value to be one higher. Basically, even though the square-root of 1152921506754330623 (the paired value of int.Max & int.Max-1) is 1073741824.9999999990686774262519 (which after calling the Math.Floor would return back 1073741824), the Math.Sqrt returns it as 1073741825. Once I found out that the problem was with the Math.Sqrt, the fix was easy – given that flooring the Square-root of a number might result in rounding error by 1, if you just check for that condition where Square Of Floor Of Square Root Of Number > Number, and just decrement the Floor by 1, you can successfully calculate the elegant pair and reverse them for any non-negative integers up to Int.Max. Below is the C# code for the calculation of the Elegant Pair along with the reversal -

        static ulong ElegantPair(uint x, uint y)
        {
            if (x >= y)
            {
                return (ulong)x * x + x + y;
            }
            else
            {
                return (ulong)y * y + x;
            }
        }

        static uint[] ElegantReverse(ulong z)
        {
            uint[] pair = new uint[2];
            double preciseZ = Math.Sqrt(z);
            ulong floor = (ulong)Math.Floor(preciseZ);
            if (floor * floor > z)
            {
                floor--;
            }
            ulong t = z - (ulong)(floor*floor);
            if (t < floor)
            {
                pair[0] = (uint)t;
                pair[1] = (uint)floor;
            }
            else
            {
                pair[0] = (uint)floor;
                pair[1] = (uint)t - (uint)floor;
            }
            return pair;
        }

Cantor Pairing Function and Reversal

2011-06-09T11:32:00.002-04:00

Update - In case you have to pair very large non-negative integers, do read my post on Elegant Pairing Function.

In my last post on Dice Coefficients I talked about how a nested NxN loop for finding similarity can be changed to a reducing inner loop since Similarity Score between X,Y is reversible i.e. Sim{X,Y} = Sim{Y,X}. This means if you have 26 sets in the universe (from A-Z), you would hit the same pair twice while calculating the Dice Coefficient between them. If we can somehow, store the similarity already calculated in the first iteration between two sets (say A,B), then we don’t have to re-calculate the similarity between (B,A). Hence, our nested loop can be rewritten as-

            for (int i = 0; i < universe.length; i++)
{
for (int j = i + 1; j < universe.length; j++)
{
//calc similarity between set[i] & set[j]
                }
}

By the way, for a very large value of N (universe.length), the average complexity of the above algorithm is still ~ O(N^2) but at least we have reduced the number of iterations by nearly half.

Now that we have a similarity score between two sets, we would need a mechanism to store this score and somehow tag it with the Pair for which the similarity is calculated. Also, it would be nice if the scores can be stored in a hashtable (as retrieval is on average O(1)) and make our hashtable key uniquely identify the pair for which the score is calculated. In short, we need some way to uniquely encode two docIds into a single number – enter "Cantor Pairing Function”. Pairing functions is a reversible process to uniquely encode two natural numbers into a single number. Calculating the “Cantor Pair” is quite easy but the documentation on the reversible process is a little convoluted. Anyway, below is the C# code for generating the unique number and then reversing it to get back the original numbers (for x,y>0).

static int CantorPair(short x, short y)
{
return ((x + y) * (x + y + 1)) / 2 + y;
}

static short[] Reverse(int z)
{
short[] pair = new short[2];
int t = (int)Math.Floor((-1D + Math.Sqrt(1D + 8 * z))/2D);
int x = t * (t + 3) / 2 - z;
int y = z - t * (t + 1) / 2;
pair[0] = (short)x;
pair[1] = (short)y;
return pair;
}

As you can see the CantorPair() returns back an int for two shorts to avoid any number overflows, if you are trying to generate the pair number for two ints, you would need to use long.

Dice Coefficient–A naive Similarity engine using Set Theory

2011-06-04T12:16:00.001-04:00

One of the ways to keep a user engaged on your site is to build some sort of recommendation engine – wherein the application automatically recommends similar “content” based on the user browsing history. One classic example of this kind of recommendation engine would be the one at amazon, which shows you a list of recommendations based on your browsing history. Another approach of doing a recommendation engine would be to suggest the user a list of “similar” content based on the current item that he’s viewing, for e.g., if you are on a online music store, and currently viewing “Dark side of the moon” album by Pink Floyd, then perhaps the application can show you list of albums similar to this for e.g.”Wish You Were Here” by Pink Floyd & “Meddle” by Led Zeppelin etc. The similarity between two items in these cases is generally calculated based on some attributes which define the items. Obviously, similarity is a relative term – so to find which items are more similar to a given item, we need to score them – the one with the higher score is “more similar” than the one with the lower score for a given item. Hence, there is a need to calculate a similarity score between two items.

We had a similar problem to solve on our site - build a similarity engine based on the categories that the items are associated with. Categories are nothing more than “tags”; so to speak; in the current web 2.0 semantics. One very naive approach to calculate similarity score between two items X, Y; which have tags Tx & Ty respectively would be to find the number of tags which are common in both, so the similarity score between (X,Y) would be -
Sim(X,Y) = |{Tx} ∩ {Ty}|
For e.g. if
{Tx} = {“music”, “rock”, “pink floyd”, “cult”} for X=”Dark Side of the moon”, where Tx = list of tags for that album.
and
{Ty} = {"music","led zeppelin","cult","rock"} for Y="Meddle", where Ty = list of tags for Meddle.
Then Sim(X,Y) = 3.

We can similarly, calculate the similarity between all the items N in our collection (if you notice it is a O(N^2) operation- more on optimizing this later) and then easily find the Top-K most similar items for a given item as generally, you would only be showing top 10 or so similar items. Below is the sample C# code snippet -

        static int Similarity(IList<string> doc1Tags, IList<string> doc2Tags)
        {
            HashSet<string> tx = new HashSet<string>(doc1Tags);
            HashSet<string> ty = new HashSet<string>(doc2Tags);
            tx.IntersectWith(ty);
            return tx.Count;
        }

One big drawback with our above similarity engine is that it is heavily biased towards items with more number of tags – as items with more tags are more likely to have higher number of common elements. The other drawback is that there is no correlation between the similarity scores of two different items, it can be any integer, making it impossible to set any kind of threshold or find do cross-similarity analysis for e.g. you might want to limit the top-k similar documents to only documents which are really similar to reduce the noise documents or you might want to know if Doc A is more similar to Doc B than Doc C is to Doc D.

A better way of finding similarity would be to length normalize the similarity score, this way a. The score is not biased towards document with more tags & b. The similarity score is always between 0 & 1, so it’s easier to set the thresholds if required. So how do we length normalize, our scores? Below is one way of doing it, by dividing the intersection with |Tx| and |Ty|, where |Tx| = length of Tags for X. This gives us:

Sim(X,Y) = (2*|Tx ∩ Ty|)/(|Tx|+|Ty|)

This is what the Dice Coefficient is, a way of finding similarity measure between two sets. So lets change our earlier code to find Dice Coefficient:

        static float Similarity(IList<string> doc1Tags, IList<string> doc2Tags)
        {
            HashSet<string> tx = new HashSet<string>(doc1Tags);
            HashSet<string> ty = new HashSet<string>(doc2Tags);
            int lenTx = tx.Count;
            int lenTy = ty.Count;
            tx.IntersectWith(ty);
            float diceCoeff = (float)(2*tx.Count)/(float)(lenTx + lenTy);
            return diceCoeff;
        }

This is pretty much what we did on our project apart from few minor adjustments and optimizing the entire stuff so that we don’t iterate N^2 times (hint – Sim(X,Y) = Sim(Y,X). Also, I believe the F# sets are better equipped for this instead of the HashSet<T> class that is part of the BCL as F# sets are immutable i.e. when you do an intersection with another set, you get a completely new set instead of modifying the first set in-place – this is important when you are calculating similarity in a loop as you don’t want the original set to be touched. The other thing is to use a Heap structure for storing the Top-K documents for a given document instead of storing all the similar documents and then only picking the top K similar documents. Also, the loop for calculating similarity for a given document with other n documents; is a great candidate for parallelization using the TPL – pity we are still on 3.5! Have fun!

ASP.Net Cache and updating the value using the Indexer

2011-05-29T12:04:00.001-04:00

What happens if you update an object already in the cache using the indexer, something like -

HttpRuntime.Cache[“someKey”] = value; ?

A. The cached object is overwritten with the new value while retaining the other information like expiration (be it absolute or sliding) and cache dependencies.

B. The cache object is overwritten and the other cache information are reset to defaults.

The correct answer is B, calling the indexer on the cache object internally calls Cache.Insert(object) which passes default values for the CacheDependency, and uses default expiration policy (i.e. the object never expires unless the server is low on memory). So the next time, you find that your objects in cache are not honoring the cache TTLs or not getting evicted on dependency changes, make sure that you are not updating the object somewhere in the code using the indexer. I learnt it the hard-way while debugging the issue with a “forever-cached” object on our production site!

hearbeat...

2011-05-15T01:16:00.001-04:00

Wow, been a while since I last posted anything over here (not that I've been regular at posting somewhere else either). Guess, I've been a running a little out of steam lately when it comes to writing stuff. It's not like I don't have anything to share off-late, it's just that I don't have the energy to blog about it these days - & that pretty much sums up how the life is meandering along all this while. So this post is pretty much a beacon just to let people know that I am still alive!

Anyway, I've been working on couple of "different" stuff off-late. On our ASP.Net website we've be using the built in In-Proc Cache to cache our data-objects and when you have a pretty decent cluster size, you run into data-consistence issue apart from having to figure out how to purge caches which are not scattered all over your web-farm. I've been thinking of getting some centralized caching engine for a while and after giving MS Velocity (or whatever it's called now) & Memcached a trial, I've decided to go with memcached along with the enyim client library. We haven't rolled it out on production yet, but in our Dev environment things actually look quite good - no exceptions, good Cache-hit rates. I'll post the entire experience with memcached and integrating with asp.net in a separate blog.

The other thing that technically, I've been looking into is reverse-proxying our web servers. I compared both Squid & Varnish, and zeroed in on Varnish based on the online material available. We already have Varnish set-up before our solr-slaves (because it's quite easy to configure varnish to do that and you get about 40-50% reduction in response times) on production, and we haven't seen any issues with it. Our Varnish cache-hit rates have been actually quite low ~around 30%, which is something that we need to look at - cause to benefit from a reverse-proxy, you would want to have your cache-hits to be around 60-70% on an average. One drawback with Varnish is the documentation available on VCL (Varnish Configuration Language) but I guess if you are clear about your caching strategy, it's not that hard to write one.
Varnishing solr is actually a piece of cake (as you don't have to worry about anonymous v/s authenticated users, session cookies, persistent cookies etc) but when it comes to Varnishing your public web-servers, things can get quite tricky. I'll post in detail, the experiences & things that I learned while Varnishing our web front-ends in another post. By the way, we've been running Varnish before the IIS on our dev environment and things have been looking quite alright so far.
One last thing on the technical stuff - we've been monitoring the performances of our memcached & varnish using munin - more on this in some other post as usual. I know this is lot of Linux stuff for a .net shop but the fact is - I didn't find any good Windows alternative for the above stuff which was free and proved itself to be quite scalable.

Moving on - as some of you would know, photography has been a hobby & a stress-reliever for me. On the photography front - after quite a bit of contemplation I got myself the Cactus triggers so that I can move my Vivitar 285HV off-camera. Well, so far they have been working like a charm. The first shot that I tried; armed with my new off-camera flash capabilities was freezing the water splash - and after 54 shots in the darkness, I did get on keeper -

lemonade!

I'll post about the setup and how I went about this in another post. The other news on the photography front is that I did manage to sell on of my images on fotolia (no I am not a millionaire yet!).

Lastly, I have started playing the guitar again - it's been so many years since I last strummed so it's like I am learning anew!

Well, that pretty much sums up what I've been up to all this while, hopefully, I'll be a little bit more frequent with my posting habits.

View On Black flickr bookmarklet

2010-08-18T11:25:00.001-04:00

I’ll admit, I’m a fan of flickr and upload all of my pictures over there. One thing that I don’t like about flickr though is the default white background which unfortunately can make the photos look a little dull. So, generally I would include a link in the description to http://www.bighugelabs.com/onblack.php so that the viewer have an option to view the photos on black. Unfortunately, this meant that every time I upload a photo, I had to manually copy over the link to bighugelabs into the description. Since I am a sucker for redoing same stuff over and over again, I decided to automate the entire process and ended up writing a simple javascript bookmarklet. Now all I need to do is to navigate to my photo details page and just click on the bookmarklet and voila, the link to “onblack” version is automatically added in the description. Below is the bookmarklet just in case you are also a fan of “onblack”:

javascript:var%20evt%20=%20document.createEvent("MouseEvents");%20evt.initMouseEvent("click",%20true,%20true,%20window,%200,%200,%200,%200,%200,%20false,%20false,%20false,%20false,%200,%20null);%20var%20cb%20=%20document.getElementById("meta").getElementsByTagName("div")[0];cb.dispatchEvent(evt);var%20reg=/\/CHANGE-THIS\/(\d+)\//;var%20y=window.location.href.match(reg)[1];document.getElementById("meta").getElementsByTagName("div")[0].getElementsByTagName("textarea")[0].value='<a %20href="http://www.bighugelabs.com/onblack.php?size=large&id='%20+%20y%20+%20'">View%20On%20Black</a>';document.getElementById("meta").getElementsByTagName("button")[0].click();void(null);

The CHANGE-THIS should be changed to your flickr's NID or your flickr friendly Url i.e. whatever you see in the address bar of your flickr photostream for e.g. if your flickr photo url is http://www.flickr.com/photos/xyz you should replace CHANGE-THIS to xyz. Do keep in mind that this bookmarklet will replace whatever is there in the description field with just a link to “OnBlack” version with the text “VIew On Black”. Also, I have only tried this on Firefox and it works in almost all the scenarios that I tried.

Contextual spelling suggestions using solr SpellCheckComponent

2010-05-16T02:26:00.001-04:00

One of the problems that I faced while trying to implement the spelling suggestions (or "Did you mean") in solr was with phrase queries. If a user types in multiple terms and say one or more of the terms are mistyped, solr provides suggestions on the individual terms (using edit distances) in isolation; and then collates the results of the top most suggestions and returns back as spellcheck.collation. It's all dandy if the user just types in 1 search term or if your searches are OR searches but in case you do an AND search (or some other form of AND using the dismax), there might be instances when the spellchecker's collation actually returns 0 results; and thus resulting in a bad user experience. Let me try to explain this with an example:
Assume that the user types in "chicen fayita sadwich" and you have just two documents in your index:

Doc 1 ==> chicken fajita
Doc 2==> veg sandwich
Now, since solr/lucene treat the terms in isolation what you get in spellcheck.collation is:
"chicken fajita sandwich". Unfortunately, if you have a AND search and the user does click on this "Did you mean" link, it would result in a query (chicken AND fajita AND sandwich) resulting in zero results.
So, how do you solve this? One way to solve this is to use Shingles for creating your "spelling corpus", the only trouble with Shingles is that the number of suggestions generated is bound by the "MaxShinglesSize" and thus if you set your MaxShingleSize to say 4, you only get suggestions up to 4 terms.
Another cleaner albeit slightly slower approach is to extend the solr.SpellCheckComponent, and fire another solr query on the collation itself; if the suggestions' results are greater than 0 (or some other threshold), you return the collation back else try with the second set of suggestion (or just blank out the collation in case you don't want to fire multiple queries). This is what I did (though I am not 100% sure if this is the right way of firing solr queries): extend Solr.SpellCheckComponent, & hook onto the processRequest method to get the handle of ResponseBuilder object (so that you can then get SolrRequest and SolrSearcher objects from this). Override toNamedList method and in that get the collation string, fire another solr query using SolrSearcher and check the results' count; if the suggestion.Count > originalQuery.Count * THRESHOLD, let the collation be as is else blank it out. The guts of this is the overridden toNamedList method, which is below in case somebody is interested:

    protected NamedList toNamedList(SpellingResult spellingResult, String 
origQuery, 
            boolean extendedResults, boolean collate) 
    {
        NamedList result = super.toNamedList(spellingResult, origQuery, 
extendedResults, collate);
        if(collate){
            String collation = (String) result.get("collation");
            if(collation!=null && collation.length() > 0 && builder!=null){
                //fire a query and get the results
                try {
                    //only add spelling suggestion in case results are less than 
some threshold
                    int hits = builder.getResults().docList.matches();
                    if(hits>MIN_THRESHOLD){
                        result.remove("collation");
                        //result.add("collation", "");
                        return result;
                    }
                    SolrIndexSearcher searcher = builder.req.getSearcher();
                    QParser qp = QParser.getParser(collation, "dismax", 
builder.req);
                    NamedList params = new NamedList();
                    params.add("rows", 0);
                    params.add("omitHeader","true");
                    SolrParams localParams = SolrParams.toSolrParams(params);
                    qp.setLocalParams(localParams);
                    Query q = qp.getQuery();
                    TopDocs docs = searcher.search(q, 1);
                    int suggestionHits = docs.totalHits;
                    //try to get hits for this query
                    log.info("current hits:" + hits);
                    log.info("total number of hits:" + suggestionHits);
                    if(suggestionHits <= hits*MULTIPLIER){
                        //remove the collation
                        result.remove("collation");
                        //result.add("collation", "");
                    }
                } catch (IOException e) {
                    log.error(e.toString());
                }
                catch (ParseException e) {
                    log.error(e.toString());
                }                
            }
        }
        return result;
    }

Implementing Autosuggest in ASP.Net using WCF Rest service, JQuery and solr

2010-03-30T10:33:00.006-04:00

I finally had to do give in and implement autosuggest (aka autocomplete, aka predective text, aka look ahead) on our free text search box ala Google. The requirement was simple: implement autosuggest based on previous user typed in free text queries which return some results back (as it would not be a great UX to suggest a term which returns 0 results back). Implementing this required: a. persist user typed in queries somewhere b. write an autosuggest component based on a. and glue it to the UI to return suggestion list to the user.
Part I: Setting up solr index
Since, we already use solr 1.4 and I am not big fan of making database do anything more than just persist/retrieve data, I knew I would rather write my autosuggest component utilizing solr.
As far as I know, there are 3 ways (apart from wild card search) of achieving autosuggest using solr 1.4:
1. Use EdgeNGrams
2. Use shingles and prefix query.
3. Use the new Terms component (in solr 1.4, the terms component does not have the regex support, which means it can only do a "begins with" match).
After a bit of contemplation and research, I decided to go with using NGrams. Basically, the decision was made by the process of elimination:
a. shingles & prefix query: I know AOL does this way but unfortunately there's very little documentation online around shingles and how you go about using it; I figured; I would be groping in the dark if I went ahead with this.
b. Terms component: Even after applying the regex patch from JIRA, I felt there is lot more sanitizing of input that needs to be done (things like if the user types in multiple terms like: "joan of", I need to ensure that I replace the whitespace with \s etc). I somehow felt it's easy to break this in case a user types in slightly odd search terms.
Having decided on going the EdgeNGrams route, all I needed was two fields in schema file. Here's how the solr field definition for the "suggest" field looks like:

<fieldtype name="suggest" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.EdgeNGramFilterFactory"
minGramSize="2" maxGramSize="50" />
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>

The suggest field holds the NGrams, whereas the query field stores the query verbatim (apart from lowercasing it). This allows us to return the query back to the user in the suggestion list based on what he typed.
Part II: Building the index
Part I of the problem solved, now we have the solr index which would generate the NGrams based on the query that it needs to index and also store the query that would be returned as part of the suggestion list.
Now, we need to figure out how do we build this index i.e. how we get the user typed in queries into this index. Instead of indexing each and every query the moment the user types it in, I implemented a simple lazy writer (in fact two levels of lazy writer). Every time the user types in a query which return results, I store that in a cached dictionary. I hook on to CacheItemRemovedCallback and persist the aggregated queries into a database table, the db table also holds a DateModified timestamp column which is updated to getdate() for newly added items. We then have a offline task which runs on a nightly basis and indexes all the newest queries to the solr index. Part II solved, now we have the solr index built and ready to serve "suggestions".
Part III: gluing it with the presentation layer
There are two bits to the presentation layer:
a. Hooking the solr index to get back the suggestion list as JSON.
b. The presentation itself aka UX.

Since, we already use JQuery on our project I decided to go with Autocomplete plugin for UX. The plugin again lacks in-depth documentation but it wasn't very hard to get it working. Instead of calling the solr index directly from the autocomplete plugin (which would have the serious after-effect of exposing our solr endpoint to the public and also adding solr dependency to the UI layer), I decided to write a WCF REST shim around the solr index. Adding a shim/decorator on top of the solr also gave me a place to intercept the calls to the solr and sanitize the user input (basically escaping the Lucene special characters) and do some heavy duty caching. That's where I had a problem: WCF services in 3.5 don't have caching support built in, you need to install the REST Starter Kit to get the caching support but once I had it installed I was able to get caching working with my WCF REST service.

After thoughts
We haven't rolled out this implementation yet, so I don't know in terms of performance how it's gonna go but at least during my limited testing, I have been able to get quite fast response times. By the way, SOLR-1316 seems quite promising too and I've been watching that thread with quite some interest: it talks about implementing the terms component as a ternary search tree so that should be quite fast.

solr and indexing Chinese text

2010-03-23T10:40:00.001-04:00

So, I've been kinda bugged with trying to get our search functionality upto the mark for the Chinese (I also have the other JK in the CKJ to worry about, but more on that later) language. As some of you would know, the biggest issue with the CJK language is that there are no word boundaries unlike the latin languages. Due to the missing word boundaries, the issue is how do you tokenize the text? There are some brute force ways of tokenizing (tokenize on uni-gram) and some middle of the road approaches (tokenize on bi-gram, which is what the lucene's built in CJKAnalyzer does). We started with the CJKAnalyzer but unfortunately figured out that it just doesn't cut it esp. due to the fact that CJKAnalyzer does not let you search on single character tokens (as it only tokenizes on Bi-grams). So, I decided to bite the bullet and upgrade solr to 1.4 (I guess I should post my experiences on that too some time) which also means that now we have lucene 2.9 to play with. One of the contrib package with Lucene 2.9 is the smartCn analyzer (http://issues.apache.org/jira/browse/LUCENE-1882), and I decided to give it a shot. The smartCn uses a built in dictionary and Hidden Markov model (I have no clue what it means) to "smartly" tokenize the chinese string. SmartCn analyzer did solve the problem of single character searches and might get us through the next release but I still feel it still doesn't solve the problem to my comfort level (the multiple character searches return less result using smartCn than the CJKAnalyzer). I guess, one way to solve it is to seed the built in dictionary with chinese terms which are relevant to our site, unfortunately the built in dictionary seems to be serialized into some binary format from a java class which I am having some hard time to decipher. I need to dig a little more deeper into this and maybe shoot out an email to the contributor of this package to see if there is some easy way of seeding the built in dictionary.

oops, something when wrong...

2010-03-11T11:25:00.004-05:00

So, there's a lotsa buzz about Google Reader Play and I was tempted enough to try it. But when I when (sic) there, all I got was a "oops, something when wrong..retrying" message:

Well, maybe that's what beta means these days...

fly, fly away

2009-09-13T11:32:00.000-04:00

I still remember when I was a kid growing up in mid 80s, I would stand on my terrace and watch Woodpecker digging holes into the trunk, Owls sitting on the tree top. Watching hoards of vultures scavenge carcass on the road side was a sight everyone in those days would be quite familiar with. I would be in awe of all these wonderful creatures. Many years have passed since then and along with them most of the birds have disappeared or sighting them has become a rare phenomenon. Even, a "common" sparrow is not so common these days, all I get to see these days are crows, pigeons and kites, if you want to watch some exotic birds like the hill mynah, vultures, owls you need to go to places which are "reserved" for these creatures by us (read bird sanctuaries) or go to places where humans have not yet cut the trees and built concrete jungles. With the rate at which the birds are disappearing or are being confined to "reserved" places, I guess, the children of tomorrow would only have pictures of these birds without having a chance to see for themselves.

just my imagination

2009-08-19T13:00:00.003-04:00

It's been a while as usual since I posted last and I guess it has something to do with how I feel these days. I think writing posts require pretty decent amount of imagination and creativity (unless you want to write just for the sake of writing something), and off-late I don't seem to have either; it's not like that I don't have anything to write about; it's just that I have don't have enough creativity to elaborate it into a meaningful post. I've been thinking about writing on some of the technical stuff that I learned in the recent past (mostly the hard way), some SEO stuff, some photography tidbits, some gear lust but every time I feel like writing something; I just draw a blank. The other thing that I always wanted to try my hands on was short stories, I have a quite a few themes in my mind (some callous, some not so) but again just don't seem to have enough steam to get those into something concrete. I know I need to be a bit more frequent with my posts (just for my own sanity) but somehow I can't get my darn act together!

Photography tidbits

2009-07-11T12:15:00.002-04:00

One of the first thing that bites you if you're into photography is what subject to photograph? Do you just take pictures of the same ole' flowers, monuments or do you try to be different. The problem with trying to being different is finding out all those things which haven't been captured before. So, you hunt for those places/things (either mentally or physically) and then give up and in the end take the pictures of the same subjects which have been taken umpteen times before (butterfly on a flower, a flower petal, sunsets etc). You still believe that at least your photographs offer something different, maybe, a new perspective, a new *vision* (basically, all the things that you try to convince yourself with). Then, you get into some equipment acquisition syndrome and dream of owning the best in the class (read the most expensive); Zeiss & Leica lenses, Nikon D3X, Hasselblad: if something is so freaking expensive it gotta be good, right?

I am one of the 3 people in my country (the other two haven't been vociferous about it yet) who owns a 4/3rds system (a Olympus e-510), and honestly apart from a tiny-winy viewfinder; I haven't had too much to complain about the camera (I generally take all the blame for out of focus, horrifically shaky pics). Quite a few people have asked me why I invested into a 4/3rds system; well; just like my any other investment it wasn't something that I carefully thought about but was just an impulsive buy (I didn't even try it or any other brand before buying). It's a funny thing that I always believed I don't need a dSLR and was happy with my P&S (a Canon A95) till I decided to go for a trip to Corbett National Park where I had an opportunity to capture a tiger in the wild for the first time and boy, did I manage to capture it with a 3x optical zoom of Canon! I think sometimes you just need a better equipment to bail yourself out. Anyway, has the dSLR made me a better photographer than before? No, definitely not but it does allow me to experiment a bit more and push my own creative limits…it reminds me of a saying that I read long ago:

The difference between a picture and painting is that

a picture is a frozen captured moment of true life,

whereas a painting can be anything imagined in your mind.

..... someday, all these paintings will become pictures.

where art thou?

2009-05-27T09:51:00.002-04:00

Been a while since I wrote something, have been busy off-late with doing nothing. I guess, nothingness is a hard feeling/habit to get out of! Anyway, some time back I started playing with a cooked WM6.5 on my phone and I think though there still are miles to go but it's a step in the right direction by MS (finally a WM version which doesn't require you to pull out your stylus!). Anyway, one thing that I found missing in titanium was the task plugin so the developer in me hacked together something (more on how I did it later, given that it was my first attempt at building something for the mobile device) which did what I wanted and it now is available on xda-developers with a very original name of taskanium.

Apart from that I have revitalized my love for photography (albeit in a very small way) and have been bitten by LAS (Lens Acquisition Syndrome). I also think, I should start something like "How-I-Took-It" posts, where I can just dissect my photo and the technique that I used to capture the shot (Ok, I am no pro but I guess I have the basic concepts of photography clear).

solr and locale sensitive sort

2009-03-22T06:56:00.004-04:00

Another of those meaningless exercises:
============================
We've been using solr on our asp.net project for quite a bit now and have been quite pleased with it, everything went great till we had to support character based language (Chinese to be more specific). Based on my research, I decided to use CJKAnalyzer for analyzing and tokenizing and the results returned were generally acceptable, the only issue being the results were not sorted in any rather random order if I asked solr to sort results alphabetically. After couple of hours of hunting solr user groups and looking at the solr source, I realized that solr does'nt provide any support for locale sensitive sorting. Since finding out the problem is half the problem already solved, all I had to do was extend from solr's built-in StrField; add support for another attribute (locale) in schema.xml and pass that to new instance of SortField and voila, solr was happily returning the results sorted by pinyin. In case you are interested, below is the discussion around the problem and the JIRA issue for solr (which has the new locale sensitive custom java class):

http://www.nabble.com/CJKAnalyzer-and-Chinese-Text-sort-td22374195.html
https://issues.apache.org/jira/browse/SOLR-1073

do we need guitars?

2009-03-22T06:44:00.002-04:00

I was just reading the "starting over" post on Mars Need Guitars, a blog I used to follow earlier to check out music before it disappeared into oblivion. It's funny what depression can do to you, it can slowly eat away all that is inside of you and even before you know what hit you, you are already spiritually dead. A lot of things in that post reminded me of myself, someone who too is trying to get past every day; and (luckily) somehow succeeding at it. I just wish I could start afresh and do things that I want to do: write some short stories, write the book that always I wished to write, get away from this meaningless stuff and just break free. Maybe, one of these days, I will.

Questions

2009-03-08T01:32:00.003-05:00

A new year and same day to look back at life and think about where I am. Over the past few years a couple of questions have haunted me:

1) Where am I?
I don't think I know where I am in terms of my mental & emotional state; the scary thing is that I don't even care any longer.

2) Am I happy doing what I am doing?
No, I am not happy doing what I am doing either professionally or personally. Life has just been meandering along all these years. Earlier, I would get all worked up and try to change things that I wouldn't like but off-late it's more of sorta resigning to the facts...I don't even feel like doing anything to change my life and/or its course.

3) Do I know what I want?
Everyone will tell you that they are not getting what they want, the trouble is very few of us are even aware of what we really want. I have no guilt in admitting that I don't know what I want in my life. I can tell for sure what I DON'T want, but I don't know what will make me happy, any longer. It's just that sometimes, I feel like running away; don't know what I want to run away from, perhaps myself.

4) Have I changed over these years?
Yes, I have and quite a bit. I don't feel that strongly about anything. I used to fight for what I felt was right; I don't now. I have no problems in doing what others want me to do cause I don't feel strongly for or against anything. I believe feelings are strongly connected with your emotions and since I spent so many emotions over these years; it looks like my emotions have run out. I just go through the motions now; don't want to change anything or more rightly put; don't have any strength and motivation to change anything. Why shall I change something which doesn't matter to me, anyway?

5) Do I deserve more?
I don't know, anyway it doesn't matter what I think. My life has always been dictated by what others wanted in their life; perhaps I've let people come too close to me and yes, if I get a second chance I would still do the same thing over & over again without event a second thought.

bad programmer, no cookie!

2009-02-20T10:57:00.004-05:00

One of things that I'd been thinking of while trying to optimize the website we've been working on is cookies. One of the issue with a non-www site is that it generally is the root domain, so any cookie that is issued by the root domain will result in the browser sending that cookie with all the subsequent requests (even requests to a sub-domain and for static content). So, let's say you have a site hosted at http://example.com and all the static content is served through a sub-domain (say http://cdn.example.com) by a CDN, any cookies set by example.com have to be sent to cdn.example.com, even though the static content hosted at CDN doesn't need the cookies set by root domain. Most of the perf guides that I've read, generally recommend using a cookie-less domain , by hosting the main site on a www subdomain or by using a new domain altogether for serving the static content. Using a www or non-www version is sometimes a corporate choice (which might be influenced by the company's branding; as in our case), and using a different "cookie-less" domain might run you into XSS limitations of the browser. So, wouldn't it be nice if I could provide some kinda hint to the browser as to when not to send the cookies using some meta tag. For instance, if I want to exclude a sub-domain from receiving any parent domain's cookies, I could do:
<meta name="no-cookies" content="cdn.example.com,cdn1.example.com...">
And, in case I want to exclude certain file types on the same parent domain itself from receiving any cookies, I could do:
<meta name="no-cookies-for-files" content="jpg,js,css...">
The second meta tag will ensure that browser doesn't send back the cookies for any (static) content which doesn't need the cookies even though the content is served by the same domain. The no-cookies-for-files has to be file extension based instead of MIME based cause to figure out the MIME type, the browser has to make a request first. We can extend this idea even further; but I guess adding these two meta tag support (call it anything that you wish) would be great!
thoughts?