Shreeni's Tech Log

Too much security

2016-02-09T23:53:00.000-08:00

This happened today and this email was sent to Sharekhan. There are still engineers whose reaction to security threats is to make the product unusable.

Hi,

I keep receiving "Digital Transaction Cum DP Billing Statement" statements every now and then. These are in the form of PDF attachments with a password protection. I cannot Print them, I cannot Export them to a password less PDF. What is the point of this statement? How does any of your users use it?

I understand the need for having a read-password protection because email is unsafe. But why restrict the files from being printed AFTER they have been opened with a password? Or why not let me export them to non-password PDF AFTER I have applied the passwords?

I have not seen a more singularly useless statement in my life.

Please forward this to your tech team and tell them that their paranoia is leading to completely useless reports. They might as well not send these reports to anybody, because hey, that would be even more safe!!

Yours Sarcastically,
Shreeniwas

Code as a bottleneck

2016-02-03T06:02:00.002-08:00

When we talk about software performance, or performance bottlenecks, the general culprit is assumed to be big characters in the setup - the OS, the programming language, the storage (SQL/NoSQL) or then the web/app server.

When I used to build systems in Java, first and foremost comment I used to hear from non-Java developers was that Java is slow.

I am sure people of certain age (30 and above, I believe) were also caught up in arguments over Windows vs Unix vs Linux. Thankfully, much of that died off, what with Linux becoming the overwhelming OS favourite for servers.

Almost every programmer who hears about us (still) using SQL sneers at the fact that I am from the dinosaur era, heading for a well deserved extinction.

And yet, few people realise that the biggest culprit in most practical software setups is hardly either one of the above, it is generally the code (or the engineer) itself that can make or break performance.

Let me put that in perspective. If you are engineering a new feature for Amazon, then thinking about the choice of performant stack from day one makes absolute sense. Their engineers are so well versed with high performance patterns, tools and tricks that the choice of an OS or language or storage will make a real difference.

But if you are working anywhere else, most code will not hit the level of scale (read audience of 10s of millions operations per day). You probably work for a company that aspires to get there, but isn't yet there. Or you are happy working for a company that does work under that scale, simply because of the nature of audience or business or the software in question.

In such cases, performance issues still crop up, more often than you imagine, and that's all attributable to well meaning engineers, and well meaning code.

I will give you an example I say today at work.

We have a module that does something like this:

Scan an S3 folder for new files.
For each new file:

Read the contents
JSON-decode it
Read every record, and either write or update the data store (NoSQL)
Move the file after processing

Now, this script was processing about 20 files a minute, and due to unrelated issues, built up a backlog of 90,000 files. If nothing was done, this would have taken 3 days to get done, and that would have been unacceptable.

So, I sat down with the engineer and started poring over the code to see what we could. I spotted three issues and gave 3 recommendations:

In step 2, for each file, we were spawning a PHP process to process the file. From experience I know that spawning processes is very heavy, so I just brought the code that processed the file into that loop and did away with the spawning.
In step 2.1, we were moving the S3 file to a local temporary file and then opening to read them. I modified that code to read directly from S3.
In step 2.3, we were reading 10 records from the store and updating them in a batch of 10. I changed the batch size of 100.

These changes alone got us to processing about 2000 files a minute. 100x improvement for 3 relatively minor changes to a script.

No OS, language, store or server had to change.

Micro Releases

2016-02-02T06:06:00.000-08:00

Releases are painful. You are worried about everything you coded working out correctly. The whole idea of Sprints, and Continuous Integration, is to make sure that you release reasonably small updates, where the gap between design and accident is somewhat minimal. Obviously, the more you lump into a release, the more there are moving parts, the more you could get something wrong during integration or release.

At Vocanic, we were doing 2 week releases for our primary product VSocial. Which seemed to be going along just fine. I came back to writing code for that product a few weeks back, and I realised, to my horror, that even with a small, nimble team, end-of-sprint releases were still taking too much time and too many integration issues were sprouting up. In a typical 2 weeks cycle, you would end up touching a bit of frontend, a bit of backend, a bit of config and a bit of your storage structure, and all put together was ending up becoming too much.

So, we just decided to take continuous integration to the next level. We decided to release stuff as soon as a cogent, related task was ready to be released. We needed to do a few changes to our setup to enable us to do this. Most important, we needed a live beta environment, one where we could run the new version of our software on live data, on live set up, on live config, without affecting the stable version.

With a little bit of domain restructuring, we now have something similar to https://beta.facebook.com/ - a live version of Facebook's upcoming version, for our product.

And that was just enough. We are now able to release as soon as we want, sometimes as many as 4 dot-releases per day. On slow moving weeks, it could be one release in 2 days. Since we have started this, we have gone through 21 micro-releases**.

And it has been strangely liberating. Everyone knows that if they can code, test and make sure everything is working, they can get it out to live users before EOD. Partly it is hugely satisfying to complete the task the same day. And of course, needless to say, release early and release often.

[**I am led to believe that what we are doing is called 'Extreme Programming'. Since I have heard multiple interpretations of XP, if what we are doing fits your interpretation of XP, suit yourself. I will stick to calling them micro-releases.]

Redis as a Session Store

2014-04-14T19:58:00.004-07:00

(Cross posting from http://tech.vocanic.com/2014/04/switch-your-sessions-to-redis-without.html)

By default PHP's session management is to store them in temporary files on the file system. This doesn't scale up if you are developing a web application to run on multiple machines. You could set up your load balancer to have sticky sessions- where a session is associated to a machine - but the major drawback here is to have sessions become "bad" when that machine is out of rotation - primarily due to an error.

The typical solution, then, is to store the PHP sessions in Database, or some such store. We, at Vocanic, started off by storing the sessions in MySQL, but then as our traffic went up, scaling that instance of MySQL became a headache and so we replaced it with a MySQL+Memcache solution, which was good enough for quite a while.

Recently, we saw a bigger problem - MySQL tends to be blocking if you are doing edits, and if the MySQL instance is shared between Sessions and any other data, then any major edit operation on the other databases also slows down the Sessions code and eventually blocks all the users out - since Sessions is the first module that needs to be loaded up for all of this to work.

So, we wanted to a non-blocking, fast, scalable store for our session management. These days, I am personally going through a Redis-Admiration phase and hence I decided to migrate this code to use Redis.

(Src: http://rethinkdb.com/blog/thanks-redis/)

The pseudo code for this is something like this:

class RedisSessionHandler implements SessionHandlerInterface {
public function open($save_path, $name) {
// Set up your Redis Connection
}

public function close() {
/* */
return true;
}

public function read($id) {
// GET '$id' // unserialize and return
}

public function write($id, $data) {
$wrapper = new stdClass();
$wrapper->id = $id;
$wrapper->access = time();
$wrapper->data = $data;
// SET '$id' serialize($wrapper);
// ZADD 'ALL_SESSIONS' $wrapper->access, $id
}

public function destroy($id) {
// DEL '$id'
// ZREM 'ALL_SESSIONS' $id
}

public function gc($maxlifetime) {
$old = time() - $maxlifetime;
// $keys = ZRANGEBYSCORE 'ALL_SESSIONS', 0, '($old'
foreach($keys as $id) {
$this->destroy($id);
}
}
}

$sessionDataHandler = new RedisSessionHandler();
session_set_save_handler($sessionDataHandler, true);

// Send any headers you want to

// Rest of the code

While this will just work for a new application, we wanted to ensure that your current sessions are migrated - so we wrote a compound session handler that wrote to both old session handler and the new session handler and compared the read results to ensure that we weren't seeing anything inconsistent.

Once we ran it for a few cycles of "$maxlifetime" - ours was set at 1440 seconds, we were sure that the new handler was working as expected. Once that happened, we replaced it with code that was just the Redis Handler and we had the entire system shift to Redis.

At the moment, my nascent profiling is showing me that we are saving somewhere close to about 30ms savings in page views that uses Sessions.

Clean up crapware on Android devices

2014-04-08T03:10:00.001-07:00

Android is in pretty bad shape - simply because the OEMs are loading it with crapware apps that are never used by you and I, but will be consuming the phone's resources. After struggling with battery drain issues recently, I sat down to see if there are any alternatives. I found cleanup master app that helped me identify some of the crapware and helped me improve the battery performance significantly. Here are the steps:

Download the app
Turn off auto-updating of Apps in Play Store (read below for the reasons)
Use the App Manager tab in Cleanup Master to find all the pre-loaded apps. There is a good chance you don't use any of them, but go through the ones that you don't seem to be using and disable them. Some of them can't be disabled since they have been updated since your factory settings - so you might have to uninstall the updates before disabling them. (The reason for turning off auto-updating is that if you kept it on, then these pre-installed apps will be auto-updated and enabled - which is where you started off.)
Remove any non-pre-loaded (apps that you installed) apps that you don't use anymore.
Go to junk files tab in Cleanup Master to see if there are any files that you can delete.
Update apps every now and then manually - only those that you seem to be using.

When you are done, reboot your phone and confirm that everything you disabled is still disabled. If you are paranoid about what Cleanup Master app might do, then now is a good time to uninstall that too.

You should see a significant performance improvement.

Redis as a replacement for SNS/SQS

2014-04-02T21:38:00.001-07:00

(Cross posting from http://tech.vocanic.com/2014/04/redis-as-replacement-for-snssqs.html)

We recently launched VSocial-API, a REST based API for our Social-CRM module. We also mandated all our internal uses to be shifted to use this API instead of going through the libraries that we had earlier provisioned for. This API, codenamed internally as Mantle, brought along a set of new architectures, each of which should perhaps be talked about in a separate blog entry, but I would like to talk about one specific feature and how we just sped it up.

For each call to the API, we wanted to split the operations into two parts - one that needs to be done synchronously - like say storing data to Source-of-truth store like MySQL, and those that can be done asynchronously - like caching, or updating analytics etc.

For this break up to work, we would either need multi-threading (such that the asynchronously processing can be forked into a separate thread) or we would need message passing. Since PHP doesn't support multi-threading our first release of this API achieved this using a combination of SNS-SQS.

(Src: https://wacomdpsstorage.blob.core.windows.net/articlesmedia/content-ppe.windowsazure.com/en-us/documentation/articles/cloud-services-dotnet-multi-tier-app-using-service-bus-queues/20140207015810/getting-started-multi-tier-101.png,
Our Architecture can be loosely illustrated like this.)

At the end of synchronous processing, a message would be sent to SNS, which would be connected to a SQS. This SQS would then be scanned regularly by a background process, the asynchronous steps would be assigned and finally deleted from the SQS. SQS comes with mutex-visibility etc and automatic re-insertion if the current process died etc. All in all it was a great functional arrangement.

But this was slow. Very slow. Our profiling indicated that we were spending about 40ms (out of an average of 150ms response time) just sending data to SNS. This was pretty much unacceptable in the long run.

We have been toying with Redis internally for a few different things and this seemed to be a good fit to solve this problem. It is easy to use Redis as a Queue implementation using RPUSH and LPOP or BRPOPLPUSH commands as suggested here.

// Producer
RPUSH message

// Consumer
LPOP message

However, the problem is of mutex. The suggested solution there was to move the message from the primary queue to a processing queue and then if the process dies, then you regularly scan the processing queue and re-insert it back to the primary queue. This sounds a bit complex.

Redis can also be used as a Mutex - SADD for instance won't be successful if run for the second time with the same value. And we can easily expire the key after some time - so the mutex will be automatcially released. The Mutex can also be extended if need be.

// First process
SADD Mutex_message1 message1 // Succeeds
EXPIRE Mutex_message1 10 // Held for 10 seconds

// Second process
SADD Mutex_message1 message1 // Fails - mutex not obtained

// First Process extends the mutex for another 10 seconds
EXPIRE Mutex_message1 10 // Held for 10 more seconds

// First process releases the Mutex
DEL Mutex_message1

So, we combined the two solutions to get a replacement for SNS-SQS. It hasn't deprecated SNS-SQS yet, since the background process is still pushing it to SNS for distribution to other consumers, but going forward we should be deprecating SNS-SQS completely.

True to the word, after our push earlier today, our response time has dropped from an average of about 150ms to about 90ms.

Using PHP Namespaces

2014-01-29T19:27:00.006-08:00

// Let's suppose:

include("/usr/lib/facebook/facebook.php"); // Defines \Facebook\User
include("/usr/lib/facebook/twitter.php"); // Defines \Twitter\User

// And you need to construct both User objects

// Option1:
use \Facebook\User;

$u1 = new User(); // Constructs \Facebook\User
$u2 = new \Twitter\User();

// Option2
use \Facebook\User as FbUser;
use \Twitter\User as TwUser;

$u1 = new FbUser();
$u2 = new TwUser();

Continuous Integration with Phing and Hudson

2013-10-30T23:32:00.001-07:00

I presented at the PHP User Group (Singapore) this week on how to get Continuous Integration working in a PHP development environment using Phing and Hudson.

Check out the talk:

and the slides:

2013 10-28 php ug presentation - ci using phing and hudson from Shreeniwas Iyer

Improving our Central Deployment Architecture

2013-10-04T03:05:00.002-07:00

(Cross posting from: http://tech.vocanic.com/2013/10/improving-our-central-deployment.html)

We have written in the past about how we use a central deployment server to push all our code to our servers. We have three two general category of servers - serving pages to users; web-workers that fetch data from various external API and put them into our content stores.

The second category, web workers, has seen an increase in number proportionate to our on-boarding of clients and eventually hit about 50 servers. Now, the big problem is that if you have about 50 servers hitting our deploy server every minutes, our deploy server is just bombarded with too much and the CPU utilisation maxed out too many times for our comfort.

(Src: http://yankeegunnerblog.files.wordpress.com/2011/08/lions-hippo_1536335i.jpg)

This week, after sitting over enough CPU-Utilisation alarms (over the past 10 days or so), we decided to fix it once and for all. Here is the broad outline of the change we did to improve things:

The deploy server checks the last updated time of our code directories every minute and publishes them over a webserver.
The web workers (and even the page serving servers) have a cron running every minutes that gets this timestamp and compares with a cached copy - if the timestamp is different, then it goes and rsyncs the code over to itself.
To avoid the thundering herd problem, each client will sleep for a time period that it choses randomly (within a fixed limit) so that the clients come to deploy server in a staggered manner.

And that was all it took for us to improve the CPU utilisation on the deploy server. (The hike towards the right is because of certain changes that developers were pushing which caused actual file transfers.)

In fact, the CPU utilisation on the web workers have also gone down enough that we think that we can use them more efficiently. That's for another day though! Have a good weekend.

Compressing all PDF files in a directory

2013-07-27T17:46:00.000-07:00

[UPDATE (2017/09/03): This code is now available at https://github.com/ShreeniwasIyer/PDFCompressor ready to be forked and readily available to run at https://hub.docker.com/r/shreeniwasiyer/pdfcompressor/]

If you are like me, you would have probably accumulated a bunch of PDFs related to travels, bills, receipts, statements, financial stuff etc. These PDFs are generated through a variety of sources - using "Save As" functionality on your browser or elsewhere, downloaded from an attachment in an email, downloaded from a website using the Browser or you probably just scanned them in.

The big trouble with this collection of PDFs is that their sizes could hugely vary - starting at about 30Kb per page for an invoice generated by a well optimized program to upto 500Kb per page for a scanned document. If you want to store these files over the long run, you want to compress them so that you are consuming less space. So, I wanted to compress all of them to an extent that the visible quality doesn't worsen.

(src: http://synapse.files.wordpress.com/2011/06/compress-files1.jpg?w=630)

In Ubuntu, compressing a single PDF is fairly easy (from http://askubuntu.com/a/256449):

gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dNOPAUSE -dQUIET -dBATCH -sOutputFile=output.pdf input.pdf

Depending on the size of the original document, the improvement varied anywhere between 5% to 90%, on a sample test of a few files. So, then I decided to build in a bash script to run it over a folder with subfolders in it.

Firstly, one strange case I encountered was that if the file was small enough (about 100kb in size or smaller), then the above script would create a file sometimes bigger, and most often perceptably worse off than the original. So, it meant that it was best to compress only those files that were bigger than 100Kb in size.


# Get the file sizes=`stat -c%s "$f"`
if [ $s -gt $thresh ]; then
  # Process
else
  echo "Too-Small File $f"
fi

Secondly, if you tried applying this on a password protected file, you will encounter a bunch of errors and hence you don't want to try it on such files. It was best to detect it upfront and not attempt any further operations on password protected files.

#Check if PDF file is encrypted

output=`gs -dBATCH -sNODISPLAY "$f" 2>&1`
gsexit=$?

if [ "$gsexit" == "0" ]; then
  # Normal File
else 
  echo "Encrypted File $f"
fi

Thirdly, if the file was once compressed, I didn't want to compress them again - in case the quality deteriorated too much.

Finally, I wanted this entire script to be idempotent - which meant that I could run this operation on the same directory twice and the second one should retain state from the first run, and I wanted it to be incremental - which meant that if I had a big directory - 10GB for instance, I should run it for a few minutes, let it process a quarter of the files, stop it and restart it later and it should continue from where it left off.

To counter these above requirements, you can always build a stateful script (using files/DB to store state) - but I wanted it quick and dirty, and it should run on bash. I didn't want another programming language to be involved. So, I decided to save the state of files in the file-names itself. I used prefixes like "passprotected-", "compressed-" and "toosmall-" to know I can skip them in following rounds.

Putting it all together, here is what I got:

#!/bin/bash

if [ "$1" == '' ]; then
 echo "Usage <script-name> <directory> <threshhold>";
 exit 0;
fi

if [ "$2" == '' ]; then
 echo "Usage <script-name> <directory> <threshhold>";
 exit 0;
fi

directory=$1;
thresh=$2;

echo "Running the Compressor on $directory and $thresh";

find $directory -iname *.pdf | grep -v -i compressed | grep -v passprotected | grep -v toosmall | while read f
do
    # Looping on all files in the directory which have not been tagged in the past.

 # Get Directory where the file is located
 subdir=`dirname "$f"`
 # Get The filename in that directory. Prefixes can be added to this.
 filename=`basename "$f"`
 # Get size of the file.
 s=`stat -c%s "$f"`
 
 if [ $s -gt $thresh ]; then
  # File is big enough to be considered for Compressing
  
  # Check if the file is password protected
  output=`gs -dBATCH -sNODISPLAY "$f" 2>&1`
  gsexit=$?
  
  # CAUTION:
  # There is a chance that gsexit is non-zero for other reasons too - like format error etc.
  # This script will still mark it as Password Protected.
  
  if [ "$gsexit" == "0" ]; then
   # Normal File
   newfilename="$subdir/compressed-$filename";
   
   # The following command will create a compressed file, then delete the original if the first operation succeeded
   # And echos the fact that both commands were successful.
   
   gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dNOPAUSE -dQUIET -dBATCH -sOutputFile="$newfilename" "$f" && rm "$f" && echo "Compressed into $newfilename";
  else 
   # Tag the password protected file so we can ignore it in future.
   echo "Renaming Encrypted File $f"
   newfilename="$subdir/passprotected-$filename";
   mv "$f" "$newfilename"
  fi
 else
  # Tag files too small so we can ignore it in future
  
  echo "Renaming Too-Small File $f"
  newfilename="$subdir/toosmall-$filename";
  mv "$f" "$newfilename"
 fi
done

There are bunch of noteworthy points:

This is an in-place script which means that the originals will be over-riden. Make a copy and run it there.
The gsexit check isn't the best way to detect password protected files. The gs command could return an error for other reasons - but to keep it simple, I am bundling all errors into one tag.
The command to compress has been chained to the deleting of original using "&&", which, in bash, will happen only if first command succeeds and this could potentially mean that whatever errors, or interruptions, that happen at first command, will ensure that the original is unaffected. This is a simple, but elegant want to ensuring data integrity.

On a run of 307Mb of PDF files, this script brought down the contents down to 130Mb. Totally worth the effort!

If you have an Ubuntu, or an X-OS, give this a shot.

[UPDATE (2017/09/03): This code is now available at https://github.com/ShreeniwasIyer/PDFCompressor ready to be forked and readily available to run at https://hub.docker.com/r/shreeniwasiyer/pdfcompressor/]

AWS as a monopoly threat

2013-07-19T00:07:00.001-07:00

How many of you have heard of IBM SmartCloud? or RedHat Open Shift? or AppHarbour? or Engine Yard? or SoftLayer? or Joyent? Apparently, these are all Platform-as-a-Service (PaaS) or Infrastructure-as-a-Service (IaaS) providers. These are all supposed to be competing with AWS.

As I was sitting at the AWS Summit 2013 yesterday at the MBS, and hearing of one success story after another, which includes my experience at Vocanic, I was left wondering on whether we are seeing the case of monopoly being constructed one service, one quarter, one page view at a time.

(src: http://cdn.ttgtmedia.com/ITKE/uploads/blogs.dir/82/files/2009/07/slide1.jpg)

I have heard of famous monopolistic market captures in the past - Windows in the Desktop market (which continues to today), Google (countered by Yahoo and Bing), iPhone (countered by Android) and so on, but I have only been fortunate to see one right in front of my eyes - and that has been the ever rising tide of AWS.

(Src: http://venturebeat.files.wordpress.com/2010/12/wintel.jpg?w=558&h=9999&crop=0)

In the above instances, the case of Google, iPhone, and if I may add the other case of Facebook whose case is still open, are all cases where their contemporaries fought back hard and created either an equal or at least a significant second. In Google's case, Yahoo and Bing continued to corner at least a third of the market. In case of iPhone, the split is almost even now. In case of Facebook, while the numbers favor Facebook at the moment, significant competitors like Twitter, WhatsApp, Instagram etc are keeping Facebook busy either at competing for market share or at acquiring the market share.

But the instance of Microsoft is somewhat unique. While they were gaining traction in the 1990s, there doesn't seem to be documented evidence of major efforts to counter that in the market. Apple, their competitor on consumer facing compute market, had carved a niche that they seem to be unwilling to expand. Linux, Unix and such, which built compelling operating systems, were building it either as hobby or for the server markets, leaving the consumer market wide open for Windows to capture. Once the takeover of the market was complete, Microsoft could engage in questionable practices leading to multiple anti-trust cases.

(Src: http://insiderlouisville.com/wp-content/uploads/2012/10/MonopolyMan.jpg)

The brief history on popular case of tech competition is important to understand to study what might happen to AWS. Amazon's top brass, Jeff Bezos and Dr. Werner Vogels, are clearly pushing for a vision that isn't shared by others in the market. None of the competitors listed in the first line are fighting out for the big market of cloud infrastructure services. IBM may capture some enterprise market, but AWS is gunning for that too.

There are competitors - Google App Engine, Heroku, Azure and Rackspace that are perhaps worth mentioning (along with similar cases like Linode.) In case of Google App Engine, which I had briefly toyed with, doesn't allow a dev to write code as they like, but to write it in the paradigm chosen by GAE. Same is the case with Heroku, which unfortunately, also runs on AWS making it not much of a real competitor anyway.

Rackspace is too busy just lending standard compute instance and left too far behind in the space of providing managed services to be considered on equal footing. Windows Azure is stuck with a world where people have to use Windows Tech to be developing on their cloud.

Put it all together and you get a case where Amazon Web Services is being let loose to capture the large market for providing tech-agnostic platform and infrastructure services. What happens once their capturing of the market is complete? A monopoly.

As a consumer of AWS, I continued to be amazed by their services and how they continue to be innovating and dropping their prices as they go along. While some experts out there are holding on to the thought of an AWS Monopoly, I feel threatened by the fact that there aren't any real competitors. The consequences of such a scenario are just too frightening to imagine.

Our (selective) approach to Caching

2013-06-04T23:27:00.000-07:00

(Cross posting from http://tech.vocanic.com/2013/06/our-selective-approach-to-caching.html)

At Vocanic, for a long time, we were shirking away from the idea of Caching. It is not that we didn't feel the need, but it was always considered too much of an effort to put in, specially since, any one campaign or module wasn't running at a scale where the cache would have been beneficial.

To me caching involves the following cost:

The actual cache server (this is becoming insignificant since AWS provides one ready made)
Adopting the code for caching the data and cache invalidation across all the pertinent data flows.
All operational work (like going in and changing a data row that is wrong) need to deal with the cache invalidation.

(Image Src:http://www.testically.org/wp-content/uploads/2012/03/caching.jpg)

For most of our campaigns, the scale of data isn't big enough for the above efforts to pay off. Over 2012, that slowly started to change. We were adding on RDS Slaves and adopting our data layers to address the growing amount of traffic.

More importantly, we were also working on a few products that were stretching our data requirements. It seemed that if we can get the caching going, then at the very least, the effort to build the caching and maintaining it would be paid off by the benefits to these products. So, as of late 2012, we started working on introducing caching into a few important modules/products.

Infrastructure Level Caching

The first layer that we attacked was the DB layer. We were using AdoDB as our ORM for PHP. We were quite happy with it and had been maintaining it and adding features to it. For more than 2 years now, more than 95% of our data requirements was going through AdoDB. If we could get AdoDB to be cache aware, then that would ease our implementation efforts considerably. Thankfully, since a lot of our calls were something like:

$user->Load("id = ?", ($id));

$conversation->Load("userId = ?", ($userId));

In such case, it made perfect sense to cache the results against the $id and the $userId params. So, we decided to extend AdoDB_Active_Record (the primary class that maps to a table in DB) and added the logic to cache the results.

The next problem we encountered was that while at one place facebookId could be used to Load the $user object, at other places it could be other primary keys like $id. So, we wanted to make sure that we can cache it first and have the object be returned even in the second case.

Here is how we solved the problem:

Find the primary key of the Class/Table.
When an object is cached, cache is against it's primary key only.
Map the query string (like "facebookId = ?" and $facebookId) into this primary key.
So, when "id = ?" is queried for, we already have the object in cache and we retrieve it.

We excluded caching against queries which were returning multiple objects, like Find queries, in which case we don't know if the same query will result in the same set of objects with confidence. Obviously, we invalidated the cache on every Insert, Update, Save or Delete operations.

Once this extended class was created, we rolled it out to a few modules and made it work. Then we moved on to Application Layer Caching.

Application Layer Caching

The trouble with Infrastructure Level Caching is that it has no clue to the intentions of the calling application/module. Only the module knows the business logic of what makes sense to be cached and what not. Trying to do everything at Infrastructure Level is dangerous - you will end up caching too many objects that may not be looked up before expiry, while you will end up not caching the ones that are important.

So, we started at looking at some of our modules - specially the user warehouse that spans across all campaigns of a client - to add application layer caching. This module had clean entry points (based on the PHP api allowed to the developers) and hence cache invalidation could be carefully controlled. Moreover, this module is called multiple times for the same user while he is interacting with a campaign and the locality of reference (by time) is quite strong.

Moreover, this module was thoroughly tested using automated tests - so we could test the caching easily. So we extended our existing UserManager and added cached implementation of all interfaces - tested them and rolled out the new version slowly to one client after another.

Results

Despite our limitations of not adding caching to vast majority of our code, just attacking the infrastructure layer and one module at the application layer has got us quite a bit of Cache Hit successes. Let's look at the second peak in the red line (at 150,000 mark) and compare the various lines:

Traffic on Loadbalancer 1: ~6,000
Traffic on Loadbalancer 3: ~25,000
Total Traffic: ~31,000
DB Connections: ~12,000
Cache Hits: ~150,000

Since we use s3 for serving out static content, it is safe to assume that much of the traffic through the Loadbalancers are application service requests (ajax + page loads put together.)

Essentially, we were having 5 cache hits per request, while we are at 1 DB connection per 4 requests.

The big dipping in cache hit results mostly happen in night time SGT, when Load balancer 1 also sees a significant dip in traffic. So, our caching is tracking our data requirements quite well. The key lesson for us is that we managed to get a significant caching mechanism going with very little coding practices changes and developer effort.

Medium Size Data Set Processing with PHP + MySQL

2013-03-30T01:20:00.004-07:00

(Cross Post from Vocanic's official blog - http://tech.vocanic.com/2013/03/medium-size-data-set-processing-with.html)

This blog is a bit long, but for engineers trying to solve data processing of non-trivial sets across a MySQL+PHP setup, this is perhaps a read worth the time they spend.

(Img Source: http://www.dreamwizardz.com/images/php_mysql_logo.png)

At Vocanic, we selected to go with PHP + MySQL for a set of reasons that hold true as much as it did when we took the decision 30 months back and we are still quite happy with that choice.

One thorny problem, however, is trying to process reports or other such data processing activities. Let me define the problem first and then explain how we went about solving it:

Processing reasonably small sets of data (at least by current big-data standards): 50k-500k objects/rows at a time.
All objects have primary source of truth in MySQL.
Each object will have secondary linked objects (or joined tables in MySQL terminology) taking the total input size to anywhere between 200k-3million objects/rows.
Synchronous results are not necessary.

MySQL queries, with a bunch of joins and group bys just don't cut it. They are too slow and hog up too much resources that live queries, which are very important to be serviced aren't serviced in time. You can go along a few paths of adding indexes and optimizing the queries, but at some point it breaks down.

(Img Src: http://www.brainstuck.com/wp-content/uploads/2009/02/mysql-or-my-sql.jpg)

The next alternative is to go distributed, as in Map Reduce to solve such a problem. Having been at Yahoo when we were building a Hadoop based solution for exactly this category of problems, it was a natural choice for me to consider. With Amazon offering Elastic Map Reduce, it seemed we wouldn't have to worry about setting up and maintaining our own Hadoop cluster, which was too much for us to handle.

The trouble is - if you source of truth is in MySQL, you have to pump all of that data into S3, which is Amazon's choice of replacement for HDFS and then you started a cluster linked to S3. For me, moving the data from MySQL to S3 was more than half the trouble anyway. In addition to that, whether we went traditional Hadoop or Hive, we would have had to maintain a parallel code base in a different language, just to handle reports and data processing. There is yet no port of PHP based Map Reduce in sight.

(Img Src: http://3.asset.soup.io/asset/3776/2147_ecdd_500.png)

The reason for PHP based distributed systems not being there, seems natural, since PHP doesn't support threads, and while it is easy to imagine building a distributed data processing system with Inter Process Communication, I am gathering that it might be too heavy to make sense. I haven't done enough thinking or researching, but the evidence shows that PHP based distributed data processing is a non-starter in the tech community.

(Img Src: http://cdn.smartphowned.com/2012/11/5/54897fd852dfd829a5fc3535fb7e05d2.jpg)

So, back to Vocanic's problem. We had a perfectly functioning PHP+MySQL setup for all our synchronous processing that seemed to be useless for heavy data processing for asynchronous needs. We already had the architecture to scale up and down a bunch of web worker machines. So, we decided to experiment with building a bridge between the two worlds our self. We started building a PHP solution that would work with all our existing PHP code and still be able to achieve the above requirements.

Data Export

(Img Src: http://d36cz9buwru1tt.cloudfront.net/solutions/backup-storage/Manyways.png)

As a first cut, we adopted the AWS route of storing intermediate data in S3. So, we built a rudimentary data export from MySQL to S3. Turns out that if you have an auto-incrementing id on a table, then incremental exports are super fast - just export the latest objects. But moving a single object from MySQL and into S3, after serializing into JSON was proving to be too slow, when attempted across a sizable table. Moreover, in the first cut, we were first storing it on the local file system and doing a batch copy and I was worried that having one file per row, in the file system would make it too slow, and I was fairly right on that (with hat-tip to Yasin for cracking the exact reason). So, we moved to using batches of 5000 objects each, and we decided to write it straight to S3, since with batches, the call overhead wasn't too much. Also, we switched to PHP serialized data and not JSON due to encoding issues with UTF-8 characters. (I am sure it is solvable, but the fact that PHP serialization works perfectly well cuts down our motivation to fix it. )

Imagine that we had 50000 User objects, on which you are generating a Gender Count report, the S3 location would have looked like this:

<jobid>/input/1-5000.php

<jobid>/input/5001-10000.php

....

<jobid>/input/49001-50000.php

Map & Reduce

We decided to adopt the concept of Map Reduce itself. The simple reason is that we were foreseeing that we will parallelize the maps and the reduces, using PHP processes instead of Java threads. Even if that didn't work, adopting a well known distribution computing paradigm would have helped us if we decided that this whole arrangement was a mess and we moved to Hadoop or something similar.

Essentially, the map and the reduce are no different in this world than what you can see in the Hadoop world. Every object is passed to a Map function, that is allowed to return a group for the object and all objects of the same group are passed to the reduce function for it's processing.

Since we had neatly cut down our objects into slices of 5000 each, we read a batch, kept it in memory, passed each of them to map function and write back n number of files (with n being the distinct groups generated for this batch of 5000 objects) to the groups section of the job in S3 itself.

So, in the example above, it would become:

<jobid>/groups/male/1-5000.php (may not contain all 5000 objects, but contains every object from the batch of 1-5000 which are grouped as males)

<jobid>/groups/male/5001-10000.php

<jobid>/groups/female/1-5000.php

....

And finally, in the reduce phase, each of the files in <jobid>/groups/<group> can be passed to the reduce function for it to do whatever it does - export, summary count or something else if need be.

A summarize function is also called to allow the developers to close out anything that they wish to do after the heavy lifting is all over.

Putting Together as a service

We put together all these ideas into a PHP based framework and we are calling it VocanicMapReduceLite or VMRL in short. It runs as a daemon service on a few select servers. Every time a report is needed, developers call a helper function providing the table, the secondary object mappings and their implementation class with the map and reduce functions and the framework does the rest.

The helper function first puts the request into a SQS Queue.
Then a bunch of VMRL workers keep scanning the queue and pick one when they find it.
Each worker, at the moment, goes through the entire process of data-generation, map, reduce and summarize in sequence.
We have implemented a S3 based locking to make sure that the same dataset (same table, same implementation class, same DB) is not attacked by more than one process leading to race conditions.
The framework provides the option of developer giving a callback location, where callbacks are sent to ensure that the calling module is kept aware or progress or when the job is complete.
While many jobs of non-competing data sets can run in parallel, we haven't implemented intra-job parallelization yet.
Expected functionality like CSV exports, CSV summary tables (equivalent to group by queries) and similar SQLite3 outputs are pre-built into the framework.

It took about 2 weeks for us to develop and about 2 more weeks to troubleshoot the first implementations, but as of now, the entire set up can be considered advanced beta. We have already rolled it out into our flagship product VSocial and it seems to be working fine, on production data sets.

Summary & Road Ahead

Has this worked? Fortunately yes. At least for us.

This solution is allowing us to process a lot of insights reports over the data we contain in a parallel, asynchronous manner. The first lot always takes time, since it has to export all the data till then, but incremental runs are super fast.

In addition to this, it allows us to process the data in PHP during this flow, that wouldn't have been possible with MySQL queries, like geo-tagging, or expanding a JSON text field to extract deeper attributes.

This is also a lot cheaper than the Elastic Map Reduce, since AWS doesn't allow you to run EMR on anything less than medium EC2 node, whereas VMRL works and works perfectly fine on a EC2-micro instance.

We are working on closing out a few kinks in operational issues with VMRL and then we shall be attacking Intra-Job parallelization. I believe that this framework has promise for medium data set processing (of sizes of about a few million objects) for those who have adopted PHP+SQL as the combination for other engineering considerations. We are hoping for a 18-24 month run of this framework before we will have to reconsider it based on the state of the art of both our and our provider's setups at that point of time.

We are also considering whether such a framework would be interesting enough for a wider audience and if so, should we open source it. If you think this is the kind of solution you are looking for, drop a comment here please.

State of Vocanic Infrastructure 2013

2013-03-04T19:42:00.002-08:00

Many months ago, I wrote about why we decided to use Amazon EC2 as our primary Web Hosting mechanism. Here is a quick update on how things are going with us.

From my previous posts on this blog, it is apparent that we are still very much on Amazon EC2. In fact, our being on Amazon EC2 is one of the reasons that we are well ahead of some of our competitors in this space and we can do things that we ourselves would have deemed daunting.

Team:

We continue to be a team without a specialized server admin or DevOps engineers. Between Myself, Chris and Yasin, we share the biggest load of our DevOps work. That has worked out fine till about now, when we finally started looking for a full time person. We have found the right candidates (one full time and one trainee engineer on rotation) and they should be coming on board pretty soon. So, what I am writing below has been achieved without having a full time engineering effort.

(Image Source: http://www.companyweb.com.br/site/wp-content/uploads/2011/10/Imagem1.jpg)

Key AWS Advantages:

The natural advantage of AWS is to spin up new infrastructure elements at the click of a button. Want a new Database server? click a few buttons. Want a slave for that server? Click a few more buttons. Want a new caching server? some more buttons. Want a new Queue? Just a few more buttons.

Just the DevOps convenience isn't enough. SQS is a classic instance of how the availability of easily administered technology infrastructure spurred us to queue up all background tasks and handle it in a structured manner. If we were to build our own queuing system, like the ones I had seen at Yahoo, we would have been too intimidated to even try, given the size of our team and trade-off on such an effort.

Just having the ability to spin up new RDS servers, at multiple scales, have instilled demarcation of what data goes into what farms and hence made the setup a little easier to understand, a bit cleaner and easier to administer. If we were on hard servers, the cost of having to set them up from scratch (and pay extra just for demarcating data) would have made it too costly.

Caching servers are among the easiest things to set up. Install memcached on a server with tons of memory and you are done. Right? Wrong. The rest of the hardware on those machines either go waste when left unused, or tend to interfere with the functioning of the cache if used for other purposes (a restart for a security patch involving an imagemagick library for instance, would have made it mandatory to lose the cache too.) ElastiCache takes the headache away from you.

Summary:

(Image Source: http://www.lasvegasadco.com/draft_lens1996638module9907943photo_1213026232cartoon_geek.png)

The state of our 2013 systems architecture is that we use a lot of Amazon Web Services - EC2, RDS, ELB, ElastiCache, SQS, SNS, CloudWatch, S3 and SES, at last count. And we are very happy with it.

Central Logging Server Using Syslog-Ng @ Vocanic

2012-12-21T05:45:00.001-08:00

As blogged earlier, one of our DevOps goals at Vocanic is to have flexibility in scaling up and down the app hosting environment using EC2 instances. I also blogged about how we set up a central deployment server to ensure that pushing code changes is as easy as possible for the developers.

One of the nagging issues that we wanted to solve is that since we don't know the number of EC2 instances running, how do you troubleshoot issues where the primary way of investigation is log files?

All our apps log a lot and all of them used to use either the PEAR Log class or the native PHP error_log and both these methods, by default, log into the local machine's file system. Both these systems can be configured to separate the logs of one app from another, by specifying the destination at the beginning of the execution of the script.

The challenge then is to have one, well known, centralized logging server where all of these apps would log to, and where they would segregate logs files across app space and eventually end up. This has to have the following properties:

Update near real time: Having a one minute delay, perhaps using crons, is just way too slow for all practical purpose, since you can't tail the logs while trying to re-create the bug.
Automatically scale up when a new instances is spawned - having an arrangement which needs manual steps before the instance can be commissioned is just not acceptable.
Very easy for the developer to adopt, and not have them needing to change the 100s of log statements that they have already written through out the application/module.
Ability to write into the app specific log files so that developer don't step on each other's toes while troubleshooting stuff.

With this mind, the easiest solution to produce is the following - just have a web listener on the central logging server and call it every time somebody calls error_log or PEAR Log.info. This will ensure that the above properties, but will make our code execution incredibly slow, because every time a log has to be written, it will wait for the process to initiate a TCP (or UDP) connection to the logging server. Having a persistent connection may help, but all said and done, logging to a third-party node while processing client's request is going to still be very slow.

The optimization to this, is to have the logs collected in memory during the script's execution and passing it to the central logging server, using perhaps register_shutdown_function. In practice, this is also very slow and it tends to hold up the serving process for a precious few milli seconds more than required and in effect, affects the overall capacity of the front end servers. (Every milli second extra spent per request slows down the freeing up of THAT process/thread to serve the next request.)

The perfect solution, then, is to have something where the logging works quite like it does with a regular error_log or PEAR log, but it in turn sends it off to the destination in it's process space and process time so that the front end processing is not much affected. Turns out, a smart combination of syslog-ng client and server should solve the problem. The genesis for this solution came from Michel Feldheim's answer on StackOverflow, which was then improved upon to achieve the desired result. So, here goes the detailed steps to achieving a central logging server for a PHP web serving environment.

Central Logging Server

In the central logging server, install syslog-ng and configure it thus:

sudo apt-get install syslog-ng

add the following to /etc/syslog-ng/syslog-ng.conf:

destination d_php { file("$PROGRAM" owner(www-data) group(www-data) perm(0644)); };

filter f_php { program("^\/var\/log\/"); };

log { source(s_all); filter(f_php); destination(d_php); flags(final); };

source s_all {

# ....

# .... LET THE PREVIOUS CONTENT STAY - add the following line

tcp(port(5140) keep_alive(yes));

};

restart syslog service:

sudo service syslog-ng restart

On FE Servers

On each of the FE Servers, install syslog-ng and configure it thus:

sudo apt-get install syslog-ng

add the following to /etc/syslog-ng/syslog-ng.conf on each of the FE servers:

destination php { tcp("log.example.com" port(5140)); };

log { source(s_all); filter(f_php); destination(php); };

filter f_php { facility(user); };

restart syslog servers:

sudo service syslog-ng restart

Application Code Changes:

Now, the application code can be changed thus. Suppose each of the application have code like this writing to a separate file and you want the same structure to be reflected in the central log server:

// PREVIOUS CODE: using PEAR Log

include '/usr/share/php/Log.php';

$log = Log::singleton('file', '/var/log/nginx/xxx.log', '', array(), PEAR_LOG_INFO);

// PREVIOUS CODE: Using error_log

ini_set('error_log' , '/var/log/nginx/xxx.log');

The new code should look like:

// NEW CODE: using PEAR Log

include '/usr/share/php/Log.php';

$log = Log::singleton('syslog', LOG_USER, '/var/log/nginx/xxx.log', array(), PEAR_LOG_INFO);

// NEW CODE: Using error_log

ini_set(‘error_log’, ‘syslog’);

openlog('/var/log/nginx/xxx.log', LOG_NDELAY, LOG_USER);

If your FE servers and the Logging servers are all within the same EC2 security group, then there is no need to open the ports, since within the groups, all ports can be accessed freely, so long as a service is listening to it.

This allows your auto-scale environment to expand and contract without worries about your logs going astray.

(ps: Credit for this solution goes to Michel Feldheim's answer and then fine-tuning by Yasin, our de-facto Operation's Lead.)

Central Deployment Server @ Vocanic

2012-11-29T23:49:00.000-08:00

I blogged about how we solved the issue of scaling up our infrastructure through the use of EC2 instances, Elastic Load Balancers and a couple of RDS MySQL farms. In terms of hardware and compute power, this is good enough. There was one big problem that we had to solve, however.

The way Vocanic's Engineering team is structured is that all our engineers work independently on their project-of-the-moment. The project of the moment may be a platform module, a feature in a product, or a social campaign (which in turn is most probably a Facebook App). Unlike a product company, this team doesn't work towards collective, synchronized schedules, but rather each engineer has their own deadline, to which they launch their work, after completing their due diligence (which could be automated testing, self testing or process testing involving multiple people). Because of this, it is too expensive for our team to have a fixed release process (check out code, compile, run tests, push to servers, restart servers etc). Doing that for each file change we small change we want to push will be too slow. The process had to be instant and scalable to individual file level plus it had to allow for anybody and everybody in the team to do so.

So long as we had 2 Front End Servers, this problem is reasonably trivial to solve - ask the engineer to push the files to two of the servers and we are done. Despite the fact that pushing to just two servers is reasonably easy, engineers, being human, make mistakes and two servers end up being out of sync. So, we said, no problem - we will run a nightly script to diff the two servers in the important folders - the ones containing PHP code and PHP libraries and so long as we don't find anything amiss, we are good and if we find anything amiss, we shoot an email each night. Every morning, I would receive an email about an out-of-sync file and forward it to the engineer, and he will go fix it. This worked, at least for some time.

Then we started talking about 3,4,5 servers. When I first brought up more servers in a team meeting, most engineers had this resigned look saying - "f**k, we got to push now to half a dozen servers, and Shreeni will be forwarding diff mails about how I missed pushing it to server 4". Turns out, we needed a better solution.

EC2 allows you to store files either in the machine's own virtual space, or you could load up an Elastic Block Storage to hold your data. EBS allows you to scale your file storage requirement independent of your compute scale requirement and also allows you to detach EBS from one EC2 instance and attach it to another. And EBS attached to all our EC2 instances would solve our problem - just throw in your code there and you are good to go. Turns out AWS doesn't allow that arrangement.

We thought about using Dropbox on our servers or some such tools - but DropBox on Linux servers have always been iffy - even the single Ubuntu server we have at our office for file sharing has issues.

We then looked into the good old rsync to do the file updates from a single location. Turns out rsync is fast, efficient and works for the needs we have.

So we have now deployed a single, central deployment server, unimaginatively called deploy, whose job is to receive the updated code from the developers and hold it till an FE server comes and syncs up the relevant folders. The FE servers themselves can easily be set up - using the SSH keys of the deploy server - to do an rsync over password less SSH. Using the deploy's keys means that once you have set it up, you can replicate the FE servers many times over and you won't have to do any keys setup along the way. And with the right combination of rsync flags you can ensure that you pull only what you need and the deletes are also synchronized. The only throuble with this approach is that symlinks are not preserved, but end up becoming hard files on the FE servers, but it doesn't seem a big enough worry for us.

Since all our code is Plain Old PHP, we don't have to worry about restarting or reloading servers to pick up the updates. Our nginx+apc combination works fine in picking them and up and executing them in the best possible manner.

All transfers within EC2 instances is free from a bandwidth charge perspective - which means that as you scale up the number of servers, this arrangement brings down the cost of your operation, since your file updates are going only to one server, though one could always argue that updated code only constitutes a negligible component of our overall bandwidth bill.

This still doesn't answer the following questions, some of which we haven't sorted out yet, but we are working towards them:
- What about pushing library or config changes to FE servers, still got to be done one by one?
- What about the logs being generated at the servers, can we centralize log collection so that the developers don't have to log into a bunch of machines to troubleshoot?
- What happens if the deploy server itself goes down? What specifically happens to the keys that have been set up already?

I will write more about these aspects in further posts.

Scaling Up Infrastructure @ Vocanic

2012-11-28T02:53:00.001-08:00

I have been meaning to document some of what we have been doing to scale up our infrastructure at Vocanic. When I joined more than 2 years back, Vocanic was running on very limited infrastructure, running off a managed hosting center somewhere in the US. I wanted to move the hosting to Singapore, simply because I wanted to keep the servers close to where our traffic was coming from- which was 99% from Singapore, Malaysia and Indonesia.

Amazon Web Services had just set up a colo in Singapore and t1.micro instances were just launched in EC2. It seemed like a perfect combination for us - host our services in Singapore, put up a bunch of EC2 instances behind an Elastic Load Balancer (ELB) and back it up with MySQL RDS. We plunged into it and the experience was overall satisfactory. Given the kind of campaigns we run, we don't experience regular high traffic, and in fact, traffic at any given time was never too much to bother us. The cost was still well under control and although we took a painful 12+ months to get off from our US hosting provider, we finally did it.

We realized, recently, that our architecture isn't going to cut it too much into the future. One problem was that RDS was getting filled up fast and we were dependent too much on one RDS. We did two things at the same time, hived off our DB into two farms, each with a master and slave. The read traffic were being sent off to the slaves, while all writes to the master. This was enough to get us back to comfort zone, despite a significant increase in the amount of computing and read-write we were encountering. (Master Slave brought in a bunch of issues, but I am going to write about that in a separate blog post.)

Also, we were using one Elastic Load Balancer (ELB) to handle all our FE traffic, only to realize that one ELB can only be configured with only one SSL certificate, which meant that, as our SSL work increased (due to secure nature of recent campaigns), we were left with having to commission a bunch of ELBs. Thankfully, EC2 now allows you to attach a single EC2 instances to a bunch of ELBs, which means you don't have to spawn too many instances.

Recently, we have also realized that having a bunch of ELBs alone isn't enough. In some cases, the instances behind an ELB need to have fixed IPs (since they may be talking to services which are IP locked), while we have a lot of apps which don't have such a restriction, but need to scale up fast.

So, we are now in the process of categorizing our ELBs into two sets - fixed ELBs and flexi ELBs. Fixed ELBs will always a fixed number of nodes which we promise to the developers, which can then be converted into IP locks. These ELBs can't be scaled up by adding more EC2 instances, but only by increasing the size of each instance (from t1.micro to small to medium to large and so on). The flexi ELBs can be scaled up across either dimension - number of nodes of the capacity of each node.

When a new ELB is commissioned, the commissioning team (Dev+PM) needs to inform whether it is going to be flexi or fixed and we will accordingly set up the scripts.

Along the line, we are also working on requesting DB connections on demand, to avoid opening unused connections to DBs; using memcache as default caching for all objects etc to allow us to scale our system across different attributes. We have also commissioned Simple Notification Service (SNS) to improve the speed at which we process asynchronous tasks so that we don't have dependence on crons, which are already proving too slow for us since we can't kick it off before a minute is up. Each of these will need a separate post in itself and I hope to find time to write about all these developments.

Kicking this blog back to life

2012-11-22T04:09:00.000-08:00

Its been a long time since I posted on this forum (with my previous post being motivated by my boss wanting to showcase S3 to a client). The primary reason has been that I have been too busy (at work) and too lazy (at home) to make any meaningful contribution. I am now trying to make a conscious effort to post regularly here at some of the interesting things I have done in the last few months.

Option to access S3

2012-09-30T23:06:00.003-07:00

S3 is becoming the preferred choice of storage for a lot of companies now, and with its security features, auto scaling, and of course, easy access from EC2 instances make it the favored choice of storing files. What if the same had to be used and exposed (through secure credentials) to your clients or other third-parties?

Here is a quick collection of possible options on how your client would access the same:

List of S3 Backup Tools (By Jeremy, the famous techie)

http://jeremy.zawodny.com/blog/archives/007641.html

DragonDisk (for all OSes):

http://www.dragondisk.com/download-amazon-s3-client-google-cloud-storage-client.html

Amazon recommended or supported LIbraries for Android, iOS, Java, .NET, PHP & Ruby:

http://aws.amazon.com/code?_encoding=UTF8&jiveRedirect=1

Firefox Plugin:

http://www.s3fox.net

RIP Dennis M Ritchie

2011-10-12T20:27:00.000-07:00

Dear Dr. Dennis Ritchie,

I just heard of your demise. My condolences to your family.

You have been an inspiration to many engineers like me. I feel it is my honor & privilege to have read the book you co-authored with Dr. Brian Kernigham and set me on the path to becoming a programmer. I am also glad that all my co-students who thought that that book was "difficult to read" have either become managers or salespersons. The fact that I read it many years after I stopped actively programming in C, just so that I strengthen my basics, is a small tribute to that masterpiece.

Your creations - both Unix & C, have had a phenomenal impact on the computer industry spawning dozens of thousands of innovative products and almost single handedly powering many many industries and helping the entire computer engineering manpower to make their livelihood. I find it an honor that every language I programmed in has been either a direct or indirect descendant of C.

I believe you have left a lasting impact on my life.

Rest In Peace.

Kindle - the review

2011-07-07T02:28:00.000-07:00

Its been about 5-6 weeks since I have started using a Kindle and I believe it is long enough to write a review. My Kindle is a 3rd Generation Kindle (with the small screen) with special offers - worth USD 114. For the record, it was a gift from my lovely wife on my just-passed birthday. I picked a good partner.

I will write more about Kindle in the following lines, but it is a device meant only for reading. Don't buy it to do anything else. This is for bibliophiles - not geeks. If you are a geek - buy an iPad or something like that - where you can read books and read emails and browse internet and listen to music and watch movies and video chat and so on. The Kindle is not a general purpose electronic device, and I repeat, it is meant only for reading. Did I tell you, it is only for book lovers?

Right, now that I have put the reason d'etre of Kindle in clear perspective, let's proceed. Here are my quick notes:

Best device to read. With a nice black and white display and fluid options for page navigation, it is easy enough to read and addictive enough to keep you hooked.
Form factor is good - fits nice into your palms, easy to hold with a single hand too and still navigate pages with ease. The only complaint I have is that it too thin and too light that it can easily slip out of your hands or hold (I have dropped it a few times already - unfortunately.)
The hardware seems rigid - and as I said I have dropped it a few times and no major damage done. It also doesn't heat up - so you can hold it in your hands for many hours.
The battery life is just about brilliant. I don't bother with charging. Seriously, I don't. Once in a while, when I plug it into my laptop for copying books, it gets enough charge to last me till the next such occasion. For the record, the Kindle can go 3 weeks without a charge if you don't use Wifi and 2 weeks if you turn your Wifi on.
In terms of supported formats - the primary support is for AZW, amazon's own format and generally available only from amazon, and mobi. In general, this is good enough since calibre can easy convert almost any format into mobi and the experience is promising. PDF reading is a bit of an issue though, because reading it directly means that the screen is too small and if you adjust your zoom level, then you are left with too much scrolling for the experience to be good. Converting PDFs into mobi is ok for PDFs with primarily text content - but if you have PDFs with lots of images and such, skip it.
Getting content for the Kindle is reasonably straight forward - amazon itself is the best source - it has free and paid books. Prices are costly if you compare to Indian retail prices, but not if you compare to Singapore retail prices. There is of course a decent supply of free content around the internet - both legal and illegal - so chose your pick.
The e-Ink technology is just about brilliant - no glare, very easy on the eyes - and the comparable technologies - iPad (too much glare), Computer (again, glare and not easy on the eyes) don't even come close.
No distractions - all you are doing is reading a book - and not getting a popup here and there for an email or a text message or a call. That makes the experience well rounded.
The special offers that comes with this price is totally non-intrusive and not detrimental to the reading experience.
Lastly, it is the price. US$114 (plus shipping to Singapore made it SGD 157) is attractive enough that you should consider buying it. It is cheaper that your mobile (and cheaper than say even 3 months of mobile bills), TV, Computer, A bicycle, a dinner at a costly restaurant with drinks or two dinners at a not-so-costly-restaurant. It is great pricing.

If you are a book lover, who reads at least 10 books a year, Kindle is the device for you.

Buying my second Android Phone

2011-04-23T22:05:00.000-07:00

A few months back, I blogged when I bought my first Android phone - Samsung Galaxy S. I had my fair share of disappointments and had written that the idea that I cannot upgrade to the latest versions of AndroidOS when the updates had been launched was too disappointing to bear. When finally I did get 2.2 on the phone, it was a massive improvement and suddenly the phone started looking a lot more closer to the iPhone experience.

Subsequently, when my iPhone 3G (the one that I had passed to my wife when I got the Galaxy S) contract ended, I decided to continue with the AndroidOS. These are the reasons:

I don't use too many apps - I have a minimal set of apps that are mandatory, then everything else doesn't matter much. To that extent, Apple & Android are about the same - the minimal apps are good in both OSes - Twitter, Browsers, Skype, Email, LinkedIn, FB and the likes.
Google experience - Gmail, Chat, Picassa etc is a lot better on Android than on iPhone.
For people like me who like to read a lot on the screen, the screen size and quality of some Android phones (like the Galaxy S) is a big plus.
Battery life is normally good, and you can replace the batteries.
WiFi hotspot - I can't count the number of times this has been a life saver for both me and my friends/colleagues.

So, I am now the proud owner of a Samsung Nexus S - the Android 2.3 phone (and my wife has moved on to Galaxy S). The biggest plus is that it is a pure Android phone - which means updating stuff should be pretty easy. On the other hand, Samsung had done a good job of creating an OS version with more utilities (video player, file browser and such) than the standard OS and I had to spend some time to get up to speed with that. All in all, I am yet to have any major complaints and am generally quite impressed with the product.

To address the concerns of the previous blog post:

Touch screen becoming unusable while charging - you get used to not using the phone when it is charging. Doesn't seem much of a bummer now.
GMail App gripes: I had a few of those - but as I have been using the app, I feel the flow of usage is a lot more suited to my usage and works better than the Apple Native Email option.
Browser: Both on 2.2 and 2.3, Android's browser is a lot faster and has a better feel to using it. I still have the problem of closing a browser which I have been using for a bit, but then, I am now used to using the task manager to killing the browser. I still feel Google can do something about this.
Multiple calendar works and works precisely as expected in both 2.2 and 2.3 of Android OS.
Contacts usability: Once you get used to searching instead of browsing for contacts, life is back to being good on Android - and so is the case with me.
3g/2g switching - still as bad as it was in 2.1, but with better battery lives, I have stopped worrying about switching to 2g altogether.
Skype VoIP: Skype is still a rather poor experience on Android as compared to iPhone. Call quality is very poor and the phones freeze when trying to use skype. Can't help but hope for a better version of skype on Android - stable & fast.

Google App Engine vs Amazon: Our decision at Vocanic

2010-10-22T02:11:00.000-07:00

When I joined Vocanic about 10 weeks back, one of the key mandates for me was to move our applications to a better infrastructure. We were running a old style hosting environment based out of a colo in the US and it just didn't seem like a scalable solution. There was already an instance where one of the apps brought in heavy traffic and brought down the site - so we wanted something that can be scaled at a short notice.

To just complete the context, we are a social media marketing agency - so we run a lot of social media applications for our clients - facebook apps, apps that interact with Foursquare, Twitter and the likes.

As a fan of the Google App Engine (GAE), I was quick to point out that option. As soon as I joined, I showed the engineers some of the apps I had built on the GAE and spoke passionately about moving into that direction.

Then I started evaluating Amazon's EC2 - specially because Amazon had set shop in Singapore recently and we were hoping to ride the Next Gen NBN wave in Singapore (most of our user base is from the region and if we could avoid sending their traffic across the pacific, it would be a boon for us) and I quickly saw why GAE was so much more a restricted environment in comparison to Amazon's EC2.

We decided to go with Amazon EC2 and having been on it for about 6 weeks now and though we were a skeptical bunch in the beginning, I have to admit that it proved to be a good choice. The specific reasons include:

GAE psupports two domain landings per app - one from appspot.com and one custom domain (say shreeni.info) of your choice. We often ran apps on multiple custom domains - if the client so wishes. In EC2, we could just hack up the apache vhosts file, but there is no such option in GAE.
GAE's data storage engine while being quite well designed for scaling up, is rather a pain to deal with - specially since any single app of ours never scales to a million rows for us, but taken together, they add up to that order. We did not want a store where we could not do arbitrary DB operations - you can't do count(*) operation on GAE, for instance - and we didn't find the trade-off of retraining all our engineers to be GAE-storage-restrictions-aware when we were not going to get any benefits in return. Amazon's RDS storage, running a MySQL InnoDB engine was good enough - and it is fairly trivial to upgrade the storage on that while still retaining all our MySQL-training intact.
We did not want to tie ourselves down to Java or Python - the two programming environments supported on GAE - specifically since we are veering towards using PHP as our default programming language (more on that in a separate post).
We found some of the restrictions a little too er.. restrictive - timing out page requests after a few seconds, for instance. So, if we wanted to run a cron job that took more than a few seconds, we would have to go and redo the whole thing to fit into the GAE restrictions. Considering that we often launch entire apps within a few days and sometime in few hours, having to deal with altering our thought process just because GAE has that restriction seemed too time-consuming and detracting to me.
Of course, serving the traffic from Singapore (one of EC2 colos) was going to make our pages faster than from GAE, which doesn't specify a region from where it will serve traffic. It might as well be doing it from Singapore, but unless it spells it out, we assumed it was from across the oceans.
Amazon is showing a lot more innovation - SNS and RDS Read Replicas being launched within the last month, while GAE hasn't shown enough new development in the last year and half I have been following them.

Starting up, one of the considerations we had was the cost of EC2 - almost ~$100 per instance per month for the lowest instance. But just in time for us, Amazon launched Micro Instances - and at ~$25 per instance per month, that was a perfect deal - our traffic, thankfully, fits perfectly on to the micro-instance-plus-RDS architecture.

We already have a EC2 setup where we have launched a few client applications with higher availability and reliability than in our previous setup and it has so far been a good experience. This is not to say we won't be building GAE apps at all - I still see a role for that kind of environment in our setup, but it didn't fit into what we wanted to be our primary infrastructure base.

In conclusion, GAE takes upon a lot of responsibility vis-a-vis Amazon EC2, but cuts down on a lot of a developer's authority. We chose to have a higher authority in exchange of accepting a higher responsibility.

Musings on Android

2010-10-03T05:58:00.000-07:00

I am hardly alone in the thoughts that Android is spreading itself too thin (thanks to @getanis who reached the same thought as I did). It seems that between Google, the manufacturers and the telecom operators, there are way too many parties involved, each of whom have the ability to screw up the final product. In the good old iPhone world, Apple took most of the responsibility - they designed, manufactured it (through FoxConn of course, but the specifications were Apple's), tested it, ran the apps, sold it and even did a significant share of customer service. And to me, that seems like a working model.

When the flash issue came up a few months back, some thought of it as a repeat of the Mac vs PC battle that was clearly won by PC in terms of market share. Yet, I believe this round of the battle is different - it is more like Windows vs Linux with Apple taking on the monopoly part and Android taking on the Linux part. Linux was fundamentally sound - open, and with lots of technical merits and yet, it fought a hard and mostly losing battle to win over desktop users. I always felt that Linux had way too many versions of the products flying around with little or no hand holding for the vast majority of the desktop users who needed a consistent experience across machines and over a reasonable period of time. Something similar might happen to Android with every body from Google to Verizon to Amazon wanting to run market places, and somewhere along the line, the experience might get diluted. Some feel Android is too open (to the detriment of end-users) and probably too supportive of operators - the parties who have killed innovation for too long.

The more pertinent thought for me, however, is that the economics don't seem to add up in favor of Android. When Apple alone is working on one product that can sell as much as a dozen Android phone, the the number of $s it can invest in making that model perfect is always going to be more and hence the product management, engineering and testing resources will always be more available for an Apple team to build a better product vis-a-vis Android products - which will be split between Google (who isn't going to be making much money per activation), manufacturers (like Samsung or HTC who will never boast of the best product/software-engineering teams) or the operators (let's just forget them as being the gatekeepers of phone quality - the best they can do is to provide good network coverage.)

(Observers of news on Yahoo will be wise to remember the peanut butter manifesto which covered the same issue - spreading yourself too thin.)

This is not to say that Apple won't mess things up - we have already seen Antennagate - but it seems that the dice is loaded in their favor of producing one good smartphone a year that can sell more profitably and can satisfy consumers a lot more than all the Android models put together in the same year.

What do you think?

My Humble Take on Android vs iPhone

2010-09-05T19:24:00.000-07:00

[Update: Read follow up post written after buying a second Android phone]

I recently shifted from an iPhone 3G to Samsung Galaxy S. Sort of, because I actually gave away the Nokia E63 that my wife held in exchange for the Galaxy S, while my old iPhone 3G was taken up by her. I have used it for about 10 days now and done about same kind of things I used to do with my iPhone - so I believe this review is well-placed.

To start off with, one of the primary motivations to buy the Galaxy S was because of the Android appeal. Android is one of the most talked about OSes out there and purported to be the real competitor to iPhone OS4. Secondly Android was often whispered to solve some of the thorny issues in iPhone - push Gmail, support for multiple exchange accounts, ability to run widgets and stuff, radio and in general openness (Oh, the tyranny of iPhone app store!).

The secondary motivation was that it was cheaper. After trading in my Nokia E63 and signing up a new contract for 2 years, the Galaxy S was delivered without me paying a cent. To compare, the iPhone 4 would have costed me $110 on top of that. Nothing much, but still the allure of moving to a more open OS coupled with it being cheap was too much to let pass. So, the Galaxy S.

Now the actual review. (Green means positive, red means negative, if you did not guess it already):

Hardware:
[Compared is a brand new Samsung Galaxy S with 18 month old iPhone 3G.]

The screen size is bigger and it doesn't lose out of clarity w.r.t iPhone 3G. For that bigger screen, it doesn't become any difficult to handle or keep in your pocket. In general, a plus.
It is so much lighter than the iPhone 3G. After handling it for a few days, when I pick up the iPhone 3G, it feels like lifting a brick.
The batteries can be replaced. One of my big gripes is that smart phones in general are a drain on batteries - they must be since they run so many radios and a powerful display - and hence the ability to change batteries is a must - for two reasons, one so that you can keep a spare battery fully charged up if you are a power user, and two to replace the battery a year later so that you can prolong the life of the phone itself (most phones degrade mostly on the battery front). On the other hand, the display on this phone consumes so much battery that it goes from 100%-4% within 12 hours with about 2 hours of actual usage. Turning off 3G doesn't help, as it did on iPhone, because the display is the real culprit here. More so, charging the phone from say 5% to 100% takes nothing less than 3-4 hours. iPhone 3G would do that in just over an hour.
Galaxy S is so slim that I am often afraid that it will drop it or it will slip from my hands.
For some strange reason (I am guessing that the electronics majors would throw some light here), the touch-screen becomes completely unusable when charging. The sensitivity is all thrown off and it assumes taps all the time, even when I didn't actually tap. In general, I disconnect it from the charger before using it.

Software:
[Compared is Android OS 2.1 to iOS 3.1.3. I don't care if Android fans respond to these arguments with Android 2.2 features because iOS 4.0 also has its own set of rather impressive victories.]

Swype. It is the way forward on inputting words on a phone. Typing just feels so 2000ish after using swype.
Gmail App is OK. I know, everybody seems to love the app, but I will give it passing marks at best. In general, it is good to have full web functionality, but iPhone always had that support through the browser while the native client covered 90% of my use cases. My main gripe is that the new mail alert disappears once I open the app, but not actually read the unread messages. On a busy day, I often open the app, see that the mail is not important and come back later, but with the alert gone, I am often deceived into believing I have nothing to read. The iPhone native mail client got it right - keep the number of unread messages always consistent with the actual unread messages inside, neatly updated every time you read something through the web browser (if you used IMAP/Exchange). On a minor note, usability hasn't been thought through - if you have a back button and you are in the inbox of one account, with other accounts also configured, what do you expect to happen when you click on back button? On iPhone it goes to the page listing all your accounts so you can go to one of the other accounts - intuitive, very intuitive. On Galaxy S, it exits the app itself. To go to one of the other accounts, click menu, then accounts and then you get list of accounts.
Usability of browser. Imagine you have been clicking around links and now you want to close the browser. How do you do it? On the iPhone you clicked home and you are done. On Android, you click back (which closes other apps) and you go to the previous page. So, you need to track back your entire browsing history to close it. Or you click home, leaving the browser running in the background, then go to app manager and then mark it closed. This was a big bummer.
Mutiple Calendar fail. One of the reasons I wanted Android was because of the full exchange support for multiple exchange accounts. Turns out they can't handle multiple calendar accounts, even if those calendars are all hosted on Gmail. This was again, a rather big bummer.
Widget/Calendar usability. You would imagine that a widget holding weather and stock information which comes default packaged with the phone would show you the current weather and stock information, right? Wrong. The widget sits right there with the information from the day you last refreshed it. If you want to update it, you have to click refresh. Now, who the hell was smoking cheap pot when designing this feature? Also, what is it with Calendar not notifying me when a meeting or appointment is due? iPhone did it neatly with an alert, but Android seems to do nothing of the sort. There is a tiny little message on the top, but if you didn't open the phone, you wouldn't get to know.
Contacts usability. I was using the iPhone to sync my contacts to Gmail. And on the web interface, I had filtered the contacts using groups such that only the most important and relevant would come on the phone (using "My Contacts"). In fact, in anticipation of buying an Android, I had spent hours organizing the groups to my best convenience - and guess what? The OS built by Google doesn't support the groups running on Google's gmail interface. In fact, the entire concept of groups is thrown off on Android's contacts app. If that wasn't bad enough, try updating an existing contact with a number from which you just received a call. On that interface alone, "search" is disabled. So, Android expects me to browse from 1000s (which is actually only 100s if the phone respected the groups that I created on Gmail) of contacts to update the contact. Absolute FAIL. It is so bad (and error prone) that I have stopped updating contacts using the phone and instead do it through the laptop when I get the chance.
3G/2G switching. You would think that you, as a techie, knowing that 3G meant "faster internet but more battery consuming" and 2G, aliased as GPRS, meant "slower internet but less battery consumption", were learned enough to switch between the two on the Android, right? Wrong. You also need to know that 2G is "GSM only" and 3G is "WCDMA" and that "GSM/WCDMA (auto)" means auto switching between the two. Unfortunately, I did not. So, I couldn't figure out how to switch off my 3G to save battery. And after clicking on the GSM only mode (because I thought my phone was GSM and not a CDMA one), I couldn't figure out why my phone wouldn't run 3G. It took some wikipediaing to figure out that WCDMA meant 3G. Big big FAIL.
VoIP. For some strange reason, Skype is listed on Android market place, but only for phones sold on Verizon network in the US. Tried it on my phone's market app. Nothing turns up. Tried a few other methods. Then eventually tried installing it through back door entry (download APK, put it on dropbox and then install, as suggested by @jasonong), but it was a skype lite feature which wants me to pay for receiving calls on the phone while my Skype app is running. It is rather sad to read articles which talk about the dirty little secret of Android and realizing that it could all be true (and that I am contributing to it) and in the name of openness, we might be handing the power back to the very same companies who couldn't innovate a proper smart phone.
Background Apps. It is good to have the feature. At least half the world has been singing a positive song about it - except that I can't seem to figure out why to use it? I was really happy with my iPhone usage and never felt the need to have background apps except for VoIP. Now that VoIP support isn't state-of-the-art (without skype), background apps looks like a superfluous feature. I am yet to use it. (In the meanwhile, I am seething that iPhone 4 has both background apps and state-of-the-art VoIP.)
The whole marketplace tyranny issue. If Apple has been criticized from running a tightly controlled App Store, then Google should also be criticized for running a haphazard app store. There are apps for one provider but not for others (i.e. skype). There are also apps whose reviews will tell you how it works on one device and not other. The democrat in me (wanting an open marketplace) is losing left right and center to the user in me (one who wants the apps to just be available and work).

In summary, Galaxy S is reasonably good hardware, but with Android's usability, and the phone's usability in general, being average at best, specially if you are coming from the iPhone experience (even an old iPhone OS experience) you might feel disappointed. The "open" Android isn't as usable as the "closed" iPhone. And in general, the $110 you are bound to save is just not worth the money.

[Update: Read follow up post written after buying a second Android phone]