Brave For Free

Dancing in the Cloud

2023-01-07T02:57:00.006+05:30

I mentioned in my previous post that there are only five reasons to move to the cloud. Number one on that list is flexibility.

(Image generated as usual by DALL-E-2)

Everyone tells you the cloud will give us a lot of flexibility; that you can dance there in a way it cannot do on-premise (and not just because you’re a paunchy middle-aged man). Scale up, scale down, clone, snapshot, change your mind whenever you like, no long-term contracts, no fear of irrevocable errors — flexibility, what’s not to love? While you’re getting caught up in all the sexiness though, it is important to ask one key question.

Why can’t your elephants dance? (By this, I mean to ask, of course, why your on-premise is not flexible -no actual elephants are involved)

It’s not a trivial question. For a while now, most enterprises have virtualised infrastructure not very dissimilar from the cloud, and hence many of the technological barriers to flexibility (attached storage disks, physical servers etc) have long since been demolished. Most have invested in tools that allow for big phrases like “single pane of glass” or “intelligent management”. Yet we pursue flexibility elsewhere?

This is what my experience has been…

Buying Beware

A huge reason for on-premise inflexibility is the way we buy IT capacity.

First, there’s the whole procurement process — usually involving committees and operating processes that stretch into the far horizon. Companies talk big about smart and agile and whatnot but let's face it, procurement processes are largely about paperwork and price negotiation. As for agility, let's say this; I’m proudly touching toes against other middle-aged men whilst studiously ignoring the effortless backflips of my teenage niece.

On the cloud suddenly I face no contracts, no long-term commitments, simple, immutable terms and conditions — maybe I should be doing those backflips. And don’t talk to me about sticker shock; I used to have international roaming on my mobile before Airtel offered packages.

Frustratingly, I’ve seen company after company make procurement harder thus pushing people into the clouds as a bypass. We keep making fun of how badly the government buys anything, yet we keep emulating the same exact things. A bit of an Indian phenomenon too — globally, procurements are faster, easier and often driven by rate contracts. CIOs will take flexibility where they can get it but it would serve companies much better to fix the needless rigidities on ground.

Second, IT capacity is traditionally purchased against specific budgets provided either under a project or against a division or line of business. This capacity is then “owned” by the owner of that budget, and woe betides anyone trying to cross this Laxmanrekha and use even an inch of it anywhere else. The consequence is inflexible pools of unused resources even while there are shortages elsewhere, not to mention all the costs incurred defending the (quite thoroughly virtual) fences. It's not unusual to find a virtual platform hard-sliced into many many smaller virtual pools which are then managed as individual universes. And because of issue number one (long procurement cycles), hoarding rather than scrounging is the norm — getting the size wrong means starving for months waiting for fresh supply.

The cloud rather conveniently makes these things flexible — no capacity lock-ins, no restrictions on reducing or increasing, no Laxmans to complicate things, no owners to press ownership, what’s not to like?

I’ve seen a few small fixes help — for instance, buying capacity overall under a CIO budget collated across the company and then charged back to projects usually leads to better utilization and less wastage (plus, the CIO is a convenient single neck responsible for aggregate capacity). Also, in today’s world, it is not very efficient to manually assign and manage capacity — better to go the cloud way and create a single large pool which is then parcelled off dynamically based on need.

Worst of all is if you move to the cloud but maintain the fixed allocations and rigid boundaries. Yes, you can do it if you want and unfortunately, many people do indeed want. A verticalised business structure leads to a verticalised and inflexible IT setup, regardless of technology, elephants or clouds.

Heroes of the Dark

A big driver of flexibility on the cloud is the ability to see what's going on — while enterprise CIOs stumble about their infrastructure like a blind man with a stick, teenagers are doing an eyes-wide-open Simon Biles on the cloud.

What do I mean by this? Traditionally, we of the doddering legacy systems world are used to monitoring only a few things about our infrastructure and leaving the rest to the many gods India is blessed with. We do CPU, we do memory, maybe some network, a bit of storage (only utilization, usually) and well, nothing much more. This is partly because not that much was available when we were growing up, and partly because we were too busy buying to do the running. Review of on-premise infrastructure is rarely continuous; even weekly or monthly is rare — corrective actions are therefore similarly slow. Thresholds are generous and hitting them initiates SOPs that take days or weeks to execute.

The cloud, on the other hand, came from the e-commerce world where every tiny inch of everything was monitored — which application sneezed in the last four seconds, which one acts funny every Saturday, who is pushing how hard on the network, and so on. Cloud natives are used to seeing things in (near) real-time and then using that vision to jump about in similarly short timeframes, usually automatically.

What kind of jumping about? Auto-scaling for load management (up and down). Resilience and failure recovery in minutes. Reallocation of capacity in hours. Bandwidth optimization in real-time. Adjusting dynamically to external failure incidents. A technology system that feels living and responsive, not lying on the couch playing dead.

Yes, you can get this on-premise too, through free tools like Nagios, Prometheus or ELK, alternately by shelling out for the incredible more details provided by Solarwinds, Dynatrace, Appdynamics and the like. Automation tools — today’s virtualization setups all offer competent options. The cloud provides still more — many events that can be trapped and monitored and automated and acted upon (including billing events that prevent sticker shock).

The trick, of course, is to use this information. IT has managed operations in a slow, plodding way for decades; it takes a very different operations structure (and lots of automation) to really use all this information to any benefit. Otherwise, well, you get really nice dashboards and all the flexibility of a blind man in a movie theatre.

Don’t underestimate the inertia to change; I’ve personally found it very hard to convince traditional IT operations setups to change their ways even while startups run rings around us. A pervasive distrust of automation, manual processes that no one wants to rethink, generous layers of approvals for each action, no understanding of the new tools, all make for a high inflexibility on-premise.

Worse, companies when moving to the cloud move the same practices instead of rethinking what's possible — so you pay for a teenage gymnast and get a geriatric cloud setup.

Bend it like Beckham

The biggest attraction of the cloud’s flexibility is that it comes pre-digested and fully tested, on tap at the tap of a keyboard. No pesky utilities to implement, no complicated processes to navigate, no integration tangles to unwind. However, it can lead to a few challenges. The obvious one is sticker shock — overdoing this can be quite costly.

There’s one greater danger — rushing into a flexible world with rigid people and processes and a poor understanding of how to extract the benefits. The cloud merely offers the tools of flexibility — it does not magically make you flexible. Promise needs work and some very open eyes to convert into reality.

Do become cloud-native — taking benefit of all the new technology on offer. If you are able to extract value, it becomes a question of deciding where the best price-performance is — on-premise or cloud. Otherwise, you’re going to the cloud on a fad; wearing the same sunglasses as Tom Cruise hasn’t landed me any movie offers.

Flying into the Clouds

2022-12-25T03:07:00.004+05:30

A few days ago I gave sage commentary on leaving the cloud. Today, it's about going into the cloud instead.

Image generated by MidJourney

I’ve been living with my head in the clouds for a long time now. As a fledgling startup from before AWS, I could only look on in envy as a cohort of youngsters flashed a credit card and pulled in computing power I had to spend months and millions with IBM to get just a handful of years ago, This got me deeply into the cloud, and my next few startup-ish attempts made me no money, but did teach me a lot about how to ignore IBM.

A History Lesson

I’ve worked with companies that were not startups but benefited greatly from the move to the cloud in selected areas. The first was Indiainfoline, a decade ago India’s largest broker and struggling mightily with its hyper-growth. We put — not the core, but a very essential and troublesome service on the cloud — email. Remember, a decade ago email was the collaboration lifeblood of a company, and constant problems with it were sapping a lot of energy. We wanted to get it off the table, Google met Nirmal and suddenly I was told — you have a month to move these 20,000 emails up there. This was well before such things became routine — we were one of the top ten largest migrations in the world. Googlers came down from San Francisco to help (though sometimes just to shake their head in disbelief). But it happened, and it was easier than we thought it would be, and what was worse — email just stopped failing. My best excuse for skipping meetings (I never got my invite) was zapped into the cloud in thirty days.

This was my first saas-bahu serial but more was to follow. My procurement approval workflows were saassed into existence within days when the pandemic removed all possibility of papers and peons. A loan origination system jumped into Salesforce in days. Cloud desktops bloomed like flowers when laptops were all sold out.

Then there was RBL’s disaster recovery. The argument took longer than the delivery and a handful of junior people faced the Nirmal in me. We went from nearly zero to full platinum drills in about three months. In a Bank. With lots of critical servers, apps, processes, data…

There were other stories too. Every time we had a messy complicated on-premise system we’d shove it onto the cloud and clean it up there. Legacy is notoriously hard to fix on-premise — turns out the cloud makes a lot of things easier. And along the way I kept realising, I was not the leader. People were using the cloud better than me. Getting more returns. Getting better value. So I did what you do when you can’t practice — you study and then you preach.

Staying Clear

Do CIOs have the choice today to steer entirely clear of the cloud? Of course, they do but it has to be a considered choice rather than the usual knee-jerk of security or regulation or cost fearmongering. The cloud is a powerful weapon so a CIO would be a fool to ignore or fear it. You could steer clear, but you should be very clear about the benefits of disavowing such a powerful tool.

Clear about the Cloud

The first thing about moving to the cloud is to be clear about why. Remember I wrote in the last post about not being scared of on-premise? I’ve met lots of people moving to the cloud precisely because they are. Some move because it's a fad like skinny jeans or green tea. Some don’t know why they move — they inherited someone else’s strategy and just went ahead.

Don’t move till you know why. No one is kicking you upwards yet; you can stay where you are for at least another decade (though I predict that eventually, you will have to move — data centres and server manufacturers will eventually just ignore you if your order basket isn’t a million servers). You thus have a decade to do the move well.

What are the reasons to move? Well, today only five things:

Flexibility. On-premise has always been bound by a few rigidities. A few years ago it was normal to wait months to provision new servers or storage. Today, Businesses are changing, paradigms are shifting and suddenly flexibility is a crying need. On-premise is still largely cemented into the same place. Hence cloud.
Reliability. When your grandmother’s foodblog has better availability than your expensive business website, CIOs feel worse than foolish — they feel incompetent. Better to join the bandwagon.
Capacity. Egged on by these consumer tech companies, businesses are being forced to do things like analytics and AI properly, not using overgrown spreadsheets. Petabyte and microsecond no longer have the power to intimidate businesses like they used to; how is the poor CIO expected to produce these things on-premise?
Cost. Yes, rent can be cheaper than buy. The catch? You need to be reasonably cloud-native to benefit (or if using a SaaS service, cloud-native in usage, which means eschewing premise-y things like printing). Huge benefits if the use case is correct, otherwise get back off now!
Security. Everyone asks if the cloud is secure, whereas you should be asking if not being on the cloud is secure. Properly done, the cloud is far more secure and difficult to hack into than anything the average (or even above average) CIO can produce in his or her backyard.

I can see CIOs yawning already. This is all well-known stuff; spouted by every brainiac in the last decade so why is this worth anyone’s time anymore? Well, for one I’m saying these are the ONLY five things worth thinking about here; if you have a longer list you’re wasting your time with diversions or an overpaid consultant. Sure, there are lots of other benefits possible but — if you’re at a river full of gold nuggets, pick up the nuggets and don’t get distracted fishing trout.

Go Big or Go Home

These five (FRCCS, if you want to sound like a doctor) are the lodestones of a successful move to the cloud — where I define success as better outcome and not just uneventful migration. I want each move to the cloud to achieve at least one of these outcomes, and I want that improvement in outcome to be a noticeable, substantial, Messi-winning-the-cup kind of achievement.

Why do I insist on this? The cloud is not an improved version of your on-premise setup; it's a substantial shift in paradigm, and paradigm shifts are hard, costly, stressful, risky to get right. If you achieve only marginal gains you have either not really shifted paradigms or all those huge efforts were not necessary in the first place. When we moved our first lot of servers into the cloud as is, shifting IP address rather than paradigms, costs increased by 30%. We then went and cleaned up, did the actual shift needed and costs came down dramatically, by some 80%.

Why did we move as is in the first place? It's quick and low risk; much better to move paradigms once the move is done (though the homework should have been done well beforehand). Think of it this way; If you’re planning to move apartments would you decorate first, then box it up and move or move as is, then redecorate?

Summary

So that's my quickie summary of the advice. Decide which of the five matters in your shift, plan well, move quickly as is and then paradigm-shift immediately after. Then sit back and sip coffee at the substantial improvement (or move back if you discover you’ve made a mistake).

Leaving the Cloud

2022-12-21T14:18:00.016+05:30

There’s much talk now about how the cloud is passé, and just an expensive way to send people into space really. A blog post by David Heinemeier Hansson posted some time ago about how he thinks the cloud is largely new coats of paint on old stuff and thus wants to leave it behind at once. As a practitioner who’s spent time on and off the cloud with companies large and small, these are my comments.

(I got this image created by DALL-E-2)

Now David is no lightweight. He’s the man behind of Basecamp that millions of people use. He’s also the man behind the widely used Ruby on Rails framework that tens of millions of people use. He’s a bestselling author. He’s even an actual, certified race car driver. I’ve never written a major open-source project, and only ride bicycles past racetracks so lets be clear — I’m sniping at giants. My only defence — I’ve done some of this stuff at all kinds of scale. I’ve been born once on the cloud, spent time purely on the ground and been back and forth a few times and this hopefully gives me a view worth looking at.

Davi’s core argument, that one should not rent if one can buy, is correct — but only under some circumstances. David himself points out two of them; a new company with no customers yet or an established company with highly volatile growth where he says the cloud is better. However, his argument is that these are the ONLY two cases where cloud is better, which is where I think he stumbles over his racing car.

First, let's get some background. David — one has to understand his vantage point. The company 37Signals he is CTO of is a small company — $7.9m in revenues, and about 95 employees. It has about 14 million users per month across some 150,000 paid customers. It’s a profitable, steadily growing company that is not VC funded and, unlike most other tech companies values slow profitable growth over rapid scaling. They have repeatedly talked about this: their ideal is grocery store rather than Slack or Netflix.

This makes 37Signals quite the outlier, and because of this what works for them is not always what works for others. We need more nuance.

Scaling without the Cloud

The article says Basecamp is already running a large and complex service but that’s not really correct. As far as I can make out from Basecamp’s own site and from googling other sources, the company has 8–10m visitors a day and 130,000 accounts. This is not very large; even RBL Bank where I work currently is about 7–8m visitors a month. On top of that Basecamp has no real-time transactions, relatively simple back-end processing and much easier security requirements.

David himself says “We have a business model that’s incredibly compatible with owning hardware and writing it off over many years. Growth trajectories that are mostly predictable. Expert staff who might as well employ their talents operating our own machines”

This difference is crucial. Basecamp’s loads are probably managed by a handful of servers, compared to over 3,000 servers for RBL Bank running 40+ lines of business. Running a full infrastructure operation with hundreds of racks and thousands of servers, not to mention stacks of routers and switches. And, there’s all this need for 24x7 support. Can a handful of people manage this?

For most company CTOs, the answer is an immediate no.- So much hardware usually spells. There’s the server team with its Linux experts, the network team with its Cisco experts, the storage team with its EMC experts, the virtualisation team with its VMWare experts, the database team with Oracle experts and so on. Lots of experts, lots of supervisors, lots of complexity to face up to.

However, I agree with David; this is not today’s reality.

Today’s infrastructure even on-premise is usually already virtualised and mature tools exist to manage every aspect in a largely automated fashion so a small (not particularly expert) team can do — with some help from OEMs -everything that used to require an army. Remember there was a time when a Xerox machine used to require a certified operator — today a toddler can press the button and get perfect answers because the technology has progressed, interfaces have been simplified and moving parts reduced. Something similar has happened to servers and networks; most CIOs just haven’t noticed. All the easy integrated management stuff that’s available on the cloud today is also easily (and reliably) available on-premise as well.

Amazon and Google have themselves helpfully open-sourced many of the tools they use, plus the VMWares and Nutanixes and Dells of the world have copied and matured. The hyper-scalars today achieve server-to-people ratios of the order of 10,000 — meaning one person manages 10,000 servers. At the somewhat smaller company scale, it's still easily possible to manage a few thousand servers with no more than a cricket team. Remember we’re not talking about supporting application code — that’s the same effort cloud or on-premise — but rather about managing the underlying compute storage networking infrastructure.

There’s no magic — a simplified, virtualised hardware setup (everything of the same kind) and multi-skilled people and excellent management tools — all of which most companies have within easy reach if only they tried. NSE does it in IFSC Gift City, Zerodha does it, startups do it but established companies struggle to get out of existing paradigms to do it this way.

A caveat, though (and it might be a big one). Your hardware stack needs to be uniform and relatively new for this to work (which incidentally is indeed the reality in most companies). Most of the tooling and skills become complicated or impossible unless you’ve committed to simplified, modernised stacks. A bit like Southwest Airlines having only one kind of plane — a key strategy Southwest uses to literally take to the clouds.

Spiking without the Cloud

David’s statement that businesses are either steady or volatile is too simple. Any complex business entity always has multiple lines of business and multiple products — and thus a mix of steady and volatile. New businesses may launch small and scale quickly, or launch big and scale even faster, or have steady growth at either scale. Further, it’s hard to predict spikiness in the future.

Basecamp has the luxury of both steady growth and predictability of future spikiness. After years in the same business with the same product, it’s unlikely to get any unexpected spikes. This is not true of most complex businesses, where constant new launches and introductions make things more volatile. Even here, though, David himself talks about how much of a boon the cloud was when they launched Hey — 300,000 users in hours rather than the projected 30,000 in months.

Can this be achieved without the cloud? Here the answer is a no (but with some escape clauses). On-premise scaling for a single company essentially means leaving capacity idle; there’s no other way to get capacity on demand. Some of the burdens can be eased by dynamically reallocating resources from idle applications to busy ones (an ability that most modern on-premise stacks have today) but the capacity on-premise is finite and eventually relatively limited.

What can be done today is bursting into the cloud. Even traditional businesses have started doing this in limited ways, running massive data management jobs on the cloud instead of purchasing the hardware for it. Oracle and many other hardware OEMs provide a way to expand into the cloud with pay as you go beyond the core on-premise load — RBL Bank made great use of this during the first few days of the core banking upgrade when many jobs were taking much more compute while optimisation was on. All applications don’t support this, but as applications get containerised this will increasingly become the norm.

Of course, if you’re in hypergrowth mode all this burstability won’t help; you’d rather stay on the cloud.

Shiny New Coats of Paint

David spends a lot of time talking about how people are fooled into the magic of AWS. This seems to contradict the fact that companies that are very technologically sophisticated (such as Capital One, Nasdaq, Slack or Netflix) continue to be on the cloud and continue to commit to it. Netflix spends about $12m per month today and has publicly committed to 3x the number by 2025. Slack has committed to spending $400m. It is unlikely that these companies have not done the math or are dazzled by new coats of paint.

So what does a Netflix know that a Basecamp does not? Nothing really, it's just a difference of viewpoint. Whatever you do on the cloud you can do on-premise, and it's not as difficult or people-intensive as it used to be — but it is more work than doing it on the cloud. Basecamp prioritises cost savings and is willing to put a few extra people into managing the service. Capital One (much bigger and also more complex) probably prioritises other tasks over the cost savings of the cloud; they may, for instance, choose to deploy people into product development over infrastructure management. There’s also a question of scale — grow to hundreds of thousands of servers and it's no longer a few open-source tools and some automation.

And then there are the services. The cloud can offer many services fully predigested; just take the service and don’t worry about either application or infrastructure. Analytics, logging, database, mobile backend, object storage, serverless web serving, and messaging are all available on tap, on demand, and with extreme reliability. Of course, they come at a premium but even people who own cars and houses rent from Hertz and Marriott. The trick is to use it appropriately and wisely and to know when to leave the rental.

Summary

Adoption of the cloud is about many things — cost is one of them. Businesses that need agility need to keep a foot in the cloud. Handling non-core concerns with non-core skills eventually bloat an organization’s workforce. Cloud is not a silver bullet — it’s one more powerful choice in an armoury of weapons.

Key Lessons:

Don’t blindly put things on the cloud. Consider continuously — for each product launch — if you’re more Basecamp or more Netflix. New applications or highly volatile applications are obvious candidates but there are others too. And things change, review this periodically to see where to move what.
Be less worried about not being able to hire or retain on-premise skills. You don’t need those expert armies anymore, so you can train across your entire workforce much more widely and more frequently. DevOps absorbs much of your expert workloads (requires a significant cultural shift).
Clean up and simplify your stack. Most on-premise complexity and cost come from purchasing multiple variants, vendors and versions.- Be Southwest
Move in-house for the priciest bits — in AWS today it is RDS (managed Postgres databases) so I would strongly suggest moving away from it. Build skills in-house for Postgres management and stop being so scared of databases.
The time when IAAS was the goal is long gone. Move to the cloud only if you’re ready to be cloud native — containerised, DevOps, horizontally scaled etc.
Where it makes sense, make use of some of the extra value adds that the cloud offers for free, such as much better monitoring, autoscaled AI Operations, much finer-grained cost control (Netflix has done some amazing stuff here). If you’re not ready to do all that, replace your CIO rather than your infrastructure.

Banana Peels and Getting Things Done

2019-07-15T03:46:00.000+05:30

I am sure all of you must have experienced this. A team of smart driven people get together in a room, all focused on a project and eager to progress. Two months of meetings later, the project is still drifting along, not quite getting to closure. Have you wondered why this happens? Why seemingly smart people with the best of intentions slip on the banana peel just as they are about to raise the curtain? My theory is that one (or god help us, more) of the following syndromes is in play.

1. The HYTO or "Have you thought of..." syndrome. This is when, after a decision to act has been finalised, someone pipes up with "but have you thought about ". This of course leads to discussion about whether that has really been thought about, often causing action to be deferred till the thinking about is done. Of course, the next meeting to close has another "have you thought of..." moment and the cycle continues. With each round, the HYTO concerns get increasingly more extreme and trivial, yet the delays continue as teams worry about not having thought of everything.

The HYTO syndrome is always about last minute interjections, not really about brainstorming concerns and risks. My observation is that there are two kinds of habitual HYTO interjectors to watch out for. One kind is the Cynic, always convinced things are going wrong and raising every trivial concern - usually in scary-sounding areas such as compliance or security. The other kind is the Value-adder, who feels that if he can point something out that apparently no one else has thought of, his value to the team increases (of course, it applies to women too). One way to deal with this is to identify and get rid of both kinds from the team - they're net distractions rather than net additions to a team. The other way is to make them owners, with the interjector given a mandate to close such concerns by a deadline.

2. The SWAD or "Shouldn't we also do ..." syndrome. Just as you're about to jump into action, you have your SWADist type say "shouldn't we also do ". The result is often that instead of simple decisive steps, suddenly you're faced with building a Taj Mahal with all its plannings and delays - sometimes even the eventual demise of the project. I see team after team repeatedly adding to their charter with interjections like this, and getting nothing done.

The SWAD syndrome is also about two kinds of people - the Dreamers and the Scuttlers. The Dreamers genuinely want to achieve the everything in the world - wipe out world hunger, not just plan a dinner party. The Scuttlers, on the other hand, are invested in preventing change so want to push for the change to be big, expensive and thus as far away as possible.

3. The WSET or "We should ensure that ..." syndrome. Plans are in place and you're going to go ahead when someone pipes up with "We should ensure that uptime, security or availability all expressed at 30,000ft level>". While non-functional requirements are legitimate, they need to be proportionate and appropriate. Most such genuine concerns are usually fleshed out in the beginning. All of us would have faced situations where even small rollouts get tagged at the last minute with ultra-stringent uptime, availability or other non-functional needs that are quite disproportionate or irrelevant to the rollout. These are usually accompanied by sanctimonious phrases such as "nothing less than the best" or "our reputation cannot be risked".

The WSET interjectors are usually the persons on the team not actually responsible for delivering the project or the NFR, and consequently feel no hesitation in being sanctimonious. The intent is usually demonstrate how mature and "enterprise-thinking" he or she is. You should ensure they are heard only on debate panels.

4. The LDTF or "Lets discuss this further..." syndrome. This is the last resort of the procrastinator. Usually wielded by the person who really does want status quo preserved but has run out of other tactics, this is invariably accompanied by looks of deep concern for the well-being of the project owner.

By elongating the discussions with LDTM, he or she hopes to preserve status quo as long as possible, either because of a vested interest or simply because change is scary. Such a person will obviously never get anything done and must be encouraged to discuss elsewhere.

These interjectors are pernicious because they often express real concerns, just not at the right time or in the right proportion. Their intent is not always malicious - even the well-meaning can waste time perfectly competently - but it is always to distract and delay for reasons unrelated to the project itself. A good project manager needs the maturity to detect the syndromes, and the skill to manoeuver past them.

Backing up is for Dumbos

2019-05-10T11:45:00.001+05:30

Sunanda recently lost some photos that she'd clicked on her holiday, and I asked what I thought was the obvious: "where did you back them up"?

A freezing glare followed. "Why", she asked me "should I have to back it up?".

Putting my foot firmly in my mouth, I persisted "its dumb to not back up! Technology 101!!".

"Pretty stupid technology, if you ask me" she said, marching out of the room.

This exchange got me thinking. In all my life as a technology manager (not to mention high priest of family photographs), backup has been are core mantra, but also a colossal the pain in the neck. The process takes too long, is never as current as you want and is invariably missing for the very day that your most important data was being written. Storage did indeed seem more than a bit stupid if so much time and effort was required to protect it from losing data. Technology should be smart enough to do invisibly what is essentially a core maintenance job; why should we need to allocate so much extra time, effort and money towards it? And it is much - about a fifth of every day, a bundle of licenses, a few operators and oodles of IOPS are consumed by the average company just for this - and its been that way since we left punch cards behind.

Lets go back to the basics. Why is backup needed at all? If you have a passive (let's not say dumb) storage system, backup does two things the storage cannot do for itself

Protect against failures such as disk crash
Undo accidental changes to the data by going back in a few days in time.

Protection against failure was important - especially because disk crash used to be pretty common once upon a time (even a misplaced sneeze could do bad things). Today's disks, however, don't fail quite as much and there's plenty of recovery mechanisms when they do. Nutanix has built its core around keeping multiple copies of each block on multiple separate hardware nodes, ensuring a near-perfect availability. The creating and restoring of copies (pretty much what a backup is supposed to do) is continuous, invisible and instantaneous (no chasing IT with a service request). The only thing Nutanix won't cover is if the whole cluster or datacenter goes up in flames, which is not an entirely trivial problem to solve but I suspect no worse than what their Xi-Leap is anyway trying to do. Why then are we breaking our necks backing up something that isn't going to fail?

DISCLAIMER: Nutanix does not run on the pendrives storing Sunanda's photos.

This is, of course, going to be deeply uncomfortable for a lot of people. Backups are such an integral way of life for a generation of IT operators that no matter how rational, it is a hard habit to give up. Not having a recovery you can touch is scary. Then there are regulators to convince, some of whom mandate explicit backup requirements. As a basic principle, though - its a compelling one. Storage should handle all its backing up and recovering invisibly and continuously - and to the user should appear as never failing.

Accidental deletion or corruption is a more complicated matter; consistency, quiesence and all kinds of other tricky issues come charging in. These are, however, hardly unsolved problems. Snapshots have been around for a while and are quite effective at this - its just not invisible or automatic. Think of how life would be if you could just tell your storage to just snapshot itself every hour for upto thirty days - and be able to recover time-machine style back to ten minutes before the ransomware hit you. Why should this not be a basic feature of enterprise storage, given that I don't know of any enterprise that does not need it?

There's also the debate about where such versioning and snapshots should reside - at the OS level (file or object versioning) or at the storage level (block versioning).

Here's my opinion. Versioning at file or object level is good for individuals but enterprises often need to restore whole snapshots; smart storage should offer a time machine at block level. Operating systems or application software can still offer file and object versioning but the big restores should come from the storage. Again, hardly a new idea - storage vendors have been peddling snapshot and restore capacity as an addon for ages; just that it is neither automatic nor invisible.

Every enterprise needs this, and spends enormous time and energy today obtaining a poorer version of this need. Thats why I argue this is a core function of smart storage, not an optional add-on.

There's also another use backups are put to - long-term archive needs. This is where data are stored for years, even decades for forensic reasons. These are meant to aid investigations, disputes and forensic analysis rather than restoring for operational recovery. Should this become part of smart storage? I personally don't think so - archiving is a different need, generally requires a different kind of media and is meant for particular files and data sets rather than for the underlying storage.

The Great Privacy Elephant

2018-09-17T00:12:00.004+05:30

Privacy is all the rage now, and it struck me just recently that privacy protection has, in fact, changed worryingly without being noticed.

Recently RBI insisted that all copies of all financial transaction data be stored inside the country, for reasons of "unfettered supervisory access". Part of the justification for this extra-draconian measure is to protect the privacy of Indians, ensuring that no foreign power can snoop into such data. Many other data directives followed, all in the name of protecting data privacy.

Today all of us are heated up about how Facebook and Google are compromising our privacy by knowing where we are and what we do. We insist on the right to be forgotten by these and other entities, convinced that they are stealing what really belongs to us. We insist that if they use information they learn about our habits, they should compensate us from their profits and not just by providing free services such as door-to-door directions and free email. So much noise about it all these days, one can hardly focus on clicking a decent selfie.

At the same time, the government seems very keen to ensure that every action of its citizens, especially financial actions, are tracked with identities. Anonymous transactions are basically forbidden, and WhatsApp is being asked to bend to the same pressures that Blackberry once submitted to. Money cannot be laundered, terrorists cannot be financed, and if that means checking everyone's everything all the time with "unfettered" supervisory access, so be it. I ordered a bicycle seat online from Spain the other day; even that required me to put KYC information. Demands like this are steadily increasing.

Here's what struck me. Generally, the key privacy concern that people have always had is with governments misusing privacy rights- traditionally called "search and seizure". When USA passed its constitutional strictures against "unreasonable search and seizure" it was explicitly government misconduct that they had in mind. When debating laws to prevent intercepting phones without a court order, it is the authorities and not Airtel that we are trying to check.

Privacy precedents in India has indeed historically been used to prevent anyone, but especially the state from snooping into an individual. Indeed, the Indian supreme court has always considered privacy as a key part of the right to "personal liberty" (Article 21 of the constitution), and the government of the day has often argued against the right. We the citizens should have freedom of personal action, not needing to explain to anyone or ask permission for conducting our lives - basically the right to tell anyone including the government of the day - "meri marzi, tere baap ko kya jaata hai?". Historically, the sarkars of the day have been unhappy at that but Indian jurisprudence has since evolved significantly more towards the individual in recent times.

This check on state power is required largely because private companies have very limited ability to misuse. Sure, they can fool you into buying the huge TV you did not need, but there are strong limits to what they can really do. Governments, on the other hand, can do (and have historically done) great harm with unfettered access to a person's information. While we worry about the 87 million people analysed by Cambridge Analytica, incumbent political parties all over the world - from Obama to Prashant Kishor - use substantial access to data to target, influence, gerrymander or distort elections involving significantly greater numbers of people.

The new Privacy Bill under consideration apparently gives citizens four key rights:

The right to confirm whether and how their data is being used;
The ability to correct misleading or false data;
Data portability rights
A "right to be forgotten," or the authority to restrict companies from using data they previously shared.

Apparently these data fiduciary responsibilities extend to the government as well. The problem is, skeptical as I am that private companies will always follow the rules, I am much more skeptical that babudom will be honest, incorruptible and by-the-book. Examples abound of the Indian state having trampled over other rights, why should those about personal information be any different. The concern is not so much that the government itself is malicious in the way that, say, Soviet Russia was; the worry is that there are enough bad actors in positions of power who, with all this data added to the power of their posts, can misuse it for personal benefit to the detriment of us normal people.

I keep hearing - why worry if I have nothing to hide? That's being unable to distinguish between privacy and secrecy. I'm sure Madhuri Dixit didn't have anything to hide under that famous choli of hers, but I don't think she would agree to walk around in public without it.

While I worry at Facebook and how much it knows about where we are or what we ate for dinner, India is quickly becoming a state that gathers unprecedented data about its citizens. More worryingly, unlike Facebook it is gathering mandatory traceable individual transaction data - not voluntary metadata meant for statistical predictions. Facebook can only guess if your daughter is pregnant; the Indian government may actually know. In all the noise about "foreign" social media grabbing and using our data, the bigger dangers are being silently forgotten. Aadhar data may be leaked to private operators in all sorts of corrupt ways, but think of the power we have given unnamed bureaucrats who can legitimately access the data.

I've suddenly realised that this is the elephant in the room, with real data about things that matter - and more worryingly real power to act - and this elephant is proving to be increasingly unchecked.

Rethinking Disaster Recovery

2018-08-29T02:33:00.001+05:30

Disaster Recovery has been on the minds of companies ever since the early days of commercially available computing. Today's world of DR revolves around four acronyms - BIA (business impact analysis), RPO (recovery point objective), RTO (recovery time objective) and BCP (business continuity plan). The acronyms appear in a disaster recovery plan in roughly in that order, the thinking being that you first analyse the impact to business of systems being down, then figure out how far back in the past are you willing to turn the dial back to recover from (last day, last hour, last millisecond). Next focus on how long you can afford to be down. Finally - buy a boatload of hardware, software and services to convert all this into action.

Setting up a DR is a hugely expensive affair that takes a significant amount planning and effort, not to mention all those drills and tests every now and then. CTOs have followed this prescription since the late seventies (apparently the first hot site was set up in 1978). But should they? Forty years ago computers were the size of houses, Amazon was just a river and mobiles visible only in Star Trek. Computing has undergone seismic change - are we sure these disaster theories are still valid?

The Business of Impact

As they say, the best place to start is the very beginning; the much vaunted "business impact analysis". This is supposedly the foundation of DR - studying what impact the non-availabilty of computers is going to have on which business line, and what that expected financial loss is likely to be under various scenarios. The idea of course is that the cost of downtime should be known, so that mitigation can be sensibly budgeted for. Consultants take a lot of time and make a lot of money answering this question - usually with a hundred pages of details - but I've rarely found the details used anywhere except in presentations.

Here's the bottom line; any well-managed company can tell you the most critical bits of all its businesses in a minute. Even a decade ago most companies had computers for only some lines of business; for other workflows they were not critical and basically supported processes that could be done manually (if slower). Today, computers run the show itself, not just track the data. If all your servers are down, impact will be big - is it really important to know whether you will lose a hundred crores or a hundred and twenty? And if you think your HR systems are not critical, just hope that disaster knows to avoid salary day. At best, you can focus on the order in which they are recovered. There too, be prepared for the unexpected - deadlines can loom anywhere just when most inconvenient, throwing the best laid sequences into disarray.

The other fallacy of a business impact analysis is to pretend that the disaster that struck the primary can be overcome in a few days - maybe a week. In practice, no major disaster gets over so quickly; if a serious fire or earthquake or flood has happened, you way well see your primary data center unavailable for months, not days - meaning that all those "non-critical" bits suddenly start to bite. Another reason not to indulge the fantasy that only some applications are worth moving to DR.

The Disastrous Definition of Disaster

Its important to understand what we mean by disaster, to understand what we mean by recovering from it. Traditionally, this is divided into three buckets:

Localized disasters such as equipment failure or accidental damage that is local to the computers or networks.
Site disasters that cover the entire site the computers are stored at, such as fires, explosions, riots and vandalism, building collapse, etc.
Wide area disasters that cover a large geographic area, such as earthquakes or floods.

The easiest way to handle all these disasters, conventional wisdom goes, is to put computers in a different building (thereby covering both local and site) in a different part of the world (thereby covering wide area disasters). The core reason for this is that prevention of these disasters is difficult, so it makes sense to create a backup.

All sounds firmly logical and reasonable; where's the problem? Here's where I have issues. First, localized and site disasters were quite a worry when servers were concentrated in office buildings, but a custom-designed Tier 4 datacenter is not particularly prone to the first two, and even the third - wide area disaster - is reduced considerably by choice of location. My offices have to be in cities prone to tsunamis or earthquakes but data centers can be located any distance away and in safer zones. Wal-mart apparently uses a remote nuclear bunker as their data center. The requirement that a second data center be in a "second seismic zone" is not particularly meaningful if the primary is not in danger of earthquakes.

Then there's software damage. Many of the most likely catastrophic disruptions we face today are not physical. NotPetya caused Maersk worldwide a $300m loss by wiping out nearly 4,000 servers. Before this, Backorifice, Wannacry and a host of other malware have caused large scale outages. These are now far more likely and far more destructive that the fire or earthquake fears of decades past. And most DR strategies focus only peripherally on them.

The final problem is the assumption that everything is (and should be) concentrated into one "primary" data center. Modern businesses can be everywhere; hundreds of servers in hundreds of locations. Businesses like Facebook or Google are already like that, but so are many less visible businesses such as Zara. High speed interconnectivity allows distant locations to be as "near" as the next computer used to be a decade ago so centralization is a design choice, not a necessity. When tech says "it can't be done", they usually mean it can't be done without some redesign - usually a redesign of the way they use their relational databases.

Yes, today most businesses have "primary" data centers. I put to you that its possibly cheaper (and probably no more time consuming) for most businesses to re-architect a system into a distributed model than to create and run a single full-fledged far DR site.

The Objectives of Recovery

DR strategies are dominated by the conversation around two objectives :

Recovery Point Objective - the elapsed time (or data points or transactions) before the start of the disaster that you target your recovery up to, post which all transactions and data are lost. an RPO of 10 minutes would mean that only transactions completed ten minutes before the disaster can be recovered.
Recovery Time Objective - what are you targeting as the maximum time between declaration of disaster and resumption of operations. An RTO of 4 hours would mean four hours from start of DR activity to going live again. Not start of incident mind you, a fire may rage for some time before DR activity is initiated.

Why am I calling it a trap? These two sensible objectives are sensible only if the underlying assumptions hold, and these are again invisibly tied to the mainframe-centric view of computing that we CIOs grew up with.

One: The theory of a recovery point is a viable objective only if all transaction points fall along a single timeline. If multiple parallel interconnected transactions are going on, you're in deep trouble. A decade ago, most businesses had one single thread of processing, activities were slow and linear (mostly batch) - and an RPO was easily defined. Today, businesses are different: a bank may have overlapping internet banking and funds transfer and debit card activity all done by the same set of clients - a perfect RPO would require the exact same cutoff on all the different activities or lead to inconsistencies that will take hours to resolve. A manufacturing company has manufacturing processes, goods receiving processes, finance and accounting processes all running in parallel. If you have many different points of recovery for these many different systems (even if each individually has the same RPO), a long, difficult, treacherous reconciliation process awaits.

Indeed, in any such system, there is only one single valid recovery point - zero - that is, no transaction loss at all. This is theoretically impossible to achieve in a modern system with overlapping parts unless there's an orderly shutdown; the best a business can do is to take advantage of shutdown windows and guarantee the quiescence required. It also means that most RPO targets are not actually achievable outside drills.

Two: Recovery point is assumes a business is operating in isolation. I can only recover to a point in the past if the word around me has not moved. If the ERP of a manufacturing plant shut down at 10am, and came back at (say) noon - then an RPO of 1 hr would mean ERP has no record of goods received from 9am. However, goods were received, and the suppliers of those goods do have proof that they delivered. Whats more, goods continued to pile up even after 10 am. A bank, a stock exchange, an e-commerce business will have similar problems; the rest of the world interacted with it in the hours or minutes or even microseconds that were lost; a disaster in one place can't wipe the records of those other companies away. Even compensating them for lost transactions (through, say, insurance) would require some amount of reconciliation.

Today, in an age where people get upset over losing Facebook photos, why should any business tolerate or plan for anything but zero RPO? This means businesses may lose the odd uncommitted transaction but no committed transaction should be lost or in an inconsistent state. Given the challenges of quiescence, this means that the "fail and recover" model just won't work.

Recovering in Time

Recovery Time Objective is less of a trap, since there's no theoretical barrier to having a reasonable recovery time objective. My problem, ironically, is with more time rather than less. The point of RTO is predictability - it allows a business to estimate the quantum of lost business hours and plan, insure or do other mitigations against it. If your train is not at the station, its much better to know how long the delay is, so that one can come back rather than just wait around.

However, the longer the RTO, I put to you the less predictable it is.

If you took an RTO of minutes or seconds, this would rely on technology that can be tested even in real conditions, and would probably work. We have all had long experience of failing systems over from one to the other in near real time, and we routinely test it while in production - yanking a cable to watch a router fail over, or forcing a server to fail over to another in the cluster. Netflix's famous chaos monkeys are probably the most extreme example of just that, and it all works quite reasonably.

Once things gets into hours or days, however, such predicability goes out of the window. Firstly you can't test such big outages in real conditions, since no business will risk hours of downtime to prove some reliability engineer's design. In the absence of repeated testing, I find it hard to believe that someone has designed a few hours worth of workflow and taken everything into account, not a single mistake all at the first shot. Drills are well and dandy, but everything there is neatly planned, no stress, no crisis, no unpredictable behaviour. A real disaster, like a battle plan, rarely survives the first gunshot. And businesses know this; one will find quite easily examples of companies dithering about switching over to their DR sites for as long as possible because of the fear that the process may go disastrously wrong.

Matters get even more complicated when taking into account the issues with synchronising states because of the RPO points described earlier. This is typically not part of the RTO calculation, since that one is purely about technical system availability. In my experience, reconciling business transactions can take quite a while, and businesses cannot really start till this is done. DR drills don't take this part into account, any more than fire drills take into account the time it takes for you to get your family photographs reprinted.

It is thus more reliable and realistic to plan for continuous availability rather than limited downtime - especially in conjunction with zero RPO. Remember, most modern businesses think of all significant downtime as disastrous, so continuous availability has great business value. If you're planning for limited downtime as a disaster response, be prepared for great disappointment when a real disaster occurs. Can (near) zero RTO really happen - it already is. Some companies have indeed into the act - China Construction Bank, apparently the world's second largest bank by market capitalization, claims an RTO of 10 seconds or less. Newer businesses like Amazon or Facebook have no concept of RTO; instead they constantly recover automatically and unnoticeably from repeated partial failures. Its not that they have no outages, but the outages are all partial and not complete shutdowns. The most extreme example I know of is the Netflix Simian Army - a series of automated programmes called "chaos monkeys" that deliberately disrupt the live Netflix setup as a way to test the resilience of Netflix's systems.

Mismatched Expectations

My primary issue with disaster recovery plans is that companies often have a very different expectation from DR plans. Almost without exception, the CEO thinks of a disaster recovery plan as covering mitigations against any major business interruption. Traditional DR is quite emphatically NOT that - it only covers catastrophic physical disasters and says nothing about outages such as software failures, deployment problems and a host of other downtimes unrelated to physical disasters. New York Stock Exchange did not invoke disaster recovery on July 8, 2015 even though all trading was suspended for over three hours - it was a software glitch. Neither did ICICI Bank in 2013 or NSE in 2017, when their respective systems went offline for hours. Maersk suffered a catastrophic cyberattack that wiped out all its computers and networks, and their DR plan was of little help then. This wasn't a mistake - it was because the current definition of DR is not targeted for these situations.

The overwhelming majority of loss due to IT failure in a company is generally not caused by the low-probability/high-consequence catastrophes but by high-probability/low-consequence events that occur frequently, such as software bugs, local hardware failures or security breaches. Worse, as applications become more complex, involving an ever-larger tangle of codes, data nodes, systems and networks, the exposure to these "smaller" events becomes more frequent and their impact more costly.

Indeed, substantial system outages have plagued every major company at some time or another and the solution has never been DR. The difference is - technology teams treat (external) disasters as one cause of outage among many - and DR as a response to that particular outage. Business, however, thinks outage is the disaster, regardless of cause. This to them implies that DR should be invoked to solve the problem, and there's deep frustration when it isn't. Even regulators can get into this act - companies have received notices from regulators asking them to explain extended downtimes, as NYSE or DBS found out.

There's also the constant battle about business having to ask for a certain level of resilience as part of their requirements. This is another hangover from the days that resilience was hard to do, and therefore was not done unless asked for. While it is more resource-efficient to do resilience etc at the application level, it is also far more difficult and often leads to gaps. For a couple of core applications where it may be worth the time and effort to carefully craft resilience into the application itself; for others its better to take resilience at the infrastructure and platform level - all hardware is redundant, all databases highly available, all web servers clustered for resilience and so on. This is why the public cloud is so attractive in this regard - pretty much every regular VM on AWS shares the same resilience characteristics as every other, obviating the need for long study and detailed contracts for each individual deployment.

Also something to remember - for most companies, partial loss of business, even if extended a bit, is much better than complete failure for any period. Banks can easily survive the failure of a branch or two, or the odd ATM being down - even if for extended periods. Individual flights get delayed or cancelled all the time. Business continuity and customer experience become more in line with expectations.

Then there are regulatory restrictions. I've ignored them here, since the primary objective of disaster recovery is business exigency, not regulatory compliance. If the outcomes needed by a company are achieved, regulators can generally be convinced - or some amount of work allocated explicitly towards regulatory compliance while focusing on what's good for the business. I'm reminded of the simple chain and Godrej lock that a public sector bank put around its servers in an ultra-modern, access-controlled, biometric-protected data center, as a way to meet regulations that were obviously written some time ago.

My New World DR prescriptions

Let me sum up what I've said so far:

Business impact analysis isn't all that useful. All parts of a modern business is dependent on computers, and you should look at being able to recover everything.
In today's world, the only worthwhile RPO is zero. Pay a consultant to tell you that if it helps you sleep at night
RTO plans promising just a few hours of downtime look great on paper and work well in drills, but melt away at the first sign of real trouble.
What business thinks it is getting is insurance against all significant downtime, not "downtime only if caused by an earthquake".

My prescriptions are quite simple, though there is a mountain of work underlying their implementation. I put to you however, that this mountain is no bigger than the effort consumed by a traditional DR implementation, and probably not much more costly given how expensive and wasteful traditional DR is. So here goes (roughly in order):

Build the final defence first: A bulletproof, remote bunker site that stores all data, configurations, permissions and logs. This becomes your last line of defence against the most catastrophic of events such as devastating ransomware attacks. For all my talk of resilience, we should not ignore the black swan doomsday scenario. When in real trouble, reboot everything.
Throw away RPO, RTO: Each individual failure should have zero transaction loss and seamless, sub-30-second restart, which is well within the capabilities of current off-the-shelf products. Indeed, with a little bit of design effort sub-second recovery is possible for many applications. If regulations force you RPO and RTO on you, Zero and 4 hrs is perfectly respectable and does not need a huge study to support it.
Availability rather than Recovery: Forget recovery - aim for availability where another instance of a process takes the load if one instance fails. Plan to keep everything under all possible failure scenarios, not just disasters. 99.99% availability for core applications and 99.9% for all consumer interfaces will generally give you great results.
Resilience is the best defence: re-architect systems into multiple distributed processing clusters with no single points of failure. Most of today's systems and platforms are already cluster aware; its just a matter of taking advantage of the implementations. A significant amount of resilience can be achieved even with relatively low numbers - even a cluster of as little as four, each with a very moderate 90% reliability can give you better than four-nine availability overall.
Prevention is better than cure: Use reliable cloud hosting when you can. Host on-premise applications in top tier data centers that are extremely well protected against site and wide area failures. Multiple highly reliable data centers connected with high speed data links to each other and possibly to one or two clouds can mitigate failure at incredible levels.
Separate Operations from Servers: Its surprising how many companies concentrate operations and servers in one location. Moving people is much harder than moving applications when buildings catch fire.
Focus on Infra and Platform: Make underlying infra and platforms resilient, so that any application deployed on it automatically becomes resilient. Externalise resilience.
Design for Incremental: Traditional DR designs are very stepwise, while today's technology allows things to progressively become more and more resilient. Clusters can get larger, alternate redundant paths can keep getting introduced, things should keep getting better.

Today's DR model is not like insurance; almost everyone knows someone who claimed insurance but look about you and ponder; how many companies do you know that have invoked a move to their disaster recovery site under an actual outage (not a drill)? Outages come and go, drills move along nicely but no one seems to have used it in a real life situation. It's like an expensive temple to the gods - we haven't seen it work but why take the risk of angering any celestial beings?

Disaster Recovery as we know it today is a little behind the times; lets think of better ways.

The VoIP horse that has long since bolted

2018-08-27T11:35:00.000+05:30

In June 2018, DoT announced with some fanfare that VoIP calls would now be “untethered”. Is this time for a celebration?

The truth is, VoIP calling became mainstream a long time ago. The original restrictions were designed to make life difficult in a world where phones were dumb and PCs smart; the world moved on and the restriction that both sides needed the same app installed became a fairly insignificant one. To add ketchup to the fries, smartphones smartly built deep integration to the point where the VoIP applications themselves did not need to be fiddled about with. Calls could be initiated from phonebooks directly and missed or received calls showed up along with “regular” phonecalls. And Indians caught on quickly — indeed, even while the ban was operational in full in 2017, (according to data from Warp Speed Reads, a telecom and technology research firm) some 28% of all voice calls in India were VoIP calls made through WhatsApp and Skype and the like. That’s over 80 billion call minutes; one might say more than a few horses have bolted.

Why have such a regulation at all, especially since we seem to be quite at odds here with the rest of the world? The original reason was protection of telco revenues — especially those eye-watering rates that telcos charged you for long distance and international calling (in the year 2000, calling 1,000km away would cost over Rs 30/min) supposedly to compensate for their investments in networks. Then came 2003 and Skype, which threatened to derail the party by charging next to nothing whatever the distance. Now the powers-that-be could have rewarded services that seemingly was good for consumers while punishing overcharging incumbents but such was not to be. VoIP was duly banned, at least where ever it touched anything that the Telcos thought as their domain. Computer-to-computer calling purely over data networks was exempt; but try to link any bit of a voice call to the telecom network (“VoIP-to-PSTN bridging” in the lingo) and someone would pounce menacingly. Telcos themselves moved conveniently to VoIP backbones to carry traffic across states and countries, but any whiff of innovation from anyone outside the charmed circle was quickly snuffed out.

To bolster their case the telcos had the support of the security establishment who proclaimed loudly the problems of caller ID and intercepting of VoIP calls of drug dealers and terrorists while they were chatting about the weather and whatnot. No one told the terrorists of course; the 26/11 perpetrators seem not to have bothered about the ban. It created a nice absurdity; the only kind of VoIP that was banned was the kind that started or ended in a regular phone line — easily tapped with conventional means on the telco side. The app-to-app variety that was indeed difficult to intercept —ah, well not only was it allowed (maybe a bit grey, that area there) it was widely, widely, widely used.

You may ask, why bother anyway. So somethings were banned, some were not, a bit of absurdity helped spice up things but no harm was done. Mothers got to talk to their NRI sons without much fuss, Nanu, Ringo and a few upstarts griped a bit but all was overall fine. The truth is, this policy did indeed cost the Indian industry a goodly bit — modern call centres process voice calls through lots of software to stay competitive, and in India that meant a complex, expensive dance of multiple telecom lines and PRI cards, all tied together with unnecessary, frustratingly difficult customized integrations. Every BPO, every tech multinational, every company that had anything substantial by way of a call centre had to run mini telecom empires to stay competitive while their global counterparts failed to understand what the fuss was about. And we kicked out or harassed a generation of entrepreneurs — people like Bhavin Turakhia or Ambarish Gupta moved focus to other countries and left India alone. Even a giant like Reliance Infocomm ran into walls in trying to make international calls cheaper using a VoIP backhaul.

What changed? In a word — Jio. The license that the bigger Reliance acquired on the way to telecom supremacy was only for data, so becoming a voice operator required some gymnastics and a change in the rulings. Luckily, big brother had the required charm and things happened; not as fast or as fully as people would have liked but better than before. The clarification, even as it has come, leaves enough unsaid ambiguity to make Samuel Beckett smile. If things do go the way of the rest of the world, though, we can expect a few price drops, service offerings and entrepreneurial wealth creators to follow.

So to that extent, yes — it is time to celebrate. Just don’t expect the fireworks to be very loud.

Secure on the Cloud: a Keynote

2017-11-11T14:41:00.000+05:30

Some time ago, I gave a keynote at a CloudSec 2016 Conference in Mumbai. I was pleased to learn recently that the organizers put it on Youtube. I've presented this point of view before - with slides; this was my first attempt to break away from the oppression of Powerpoint and be slidefree.

Myths and the art of iPhone maintenance

2017-10-30T18:03:00.000+05:30

A group of friends pulled me into an interesting debate. Two events happened a couple of days ago; Starbucks announced a promotion for their hundredth store in India, and iPhone X pre-order sold out within minutes. Hotly debated was the sense in these events - why are people lining up for a coffee that's just a bit cheaper than usual, or a phone that is no longer much of the revolutionary product it once was. There was talk of cheaper CCD coffee and better Samsung phones, of price elasticity, Veblen goods, artificial shortages or just plain moronic behaviour.

I felt that all these arguments were wrong. Economists should not be involved; this is really about storytelling.

Starbucks has, over the years, gathered a loyal following of people willing to pay high prices for their coffee through a combination of ambience and aspiration. The hundredth-store sale is actually aimed at their regulars, not at new clientele choosing between CCD and Starbucks. It encourages the already committed regulars to come back that particular day and be rewarded for their loyalty. Starbucks gains because people who participate become a little more loyal (myths are made "I was there when the hundredth store promo happened") Increased sales, more experimentation with products they would not otherwise try, these are all side benefits to the main loyalty one. Indeed Starbucks would probably have lost a bit of money in the process, what with added promo costs and getting only Rs 100 from someone who would probably have paid Rs 150. Sports and rockstars have known how to exploit this for ages but many brands are quite good at this too. To create a religion, you have to build rituals and myths. Just price elasticity wont do.

And of course iPhone is very much about the myth. 95% of iPhone users say they will buy another iPhone, compared to only 53% for Samsung - certainly not because iPhone is better but because it is much better at the mythmaking than Samsung. People buy the latest iPhone minutes after pre-order starts for the same reason that people queued up for Rajnikant first day first show tickets - they belong to a fan club. You may think that a phone or a coffee is just a phone or a coffee, and that people buying into this myth are morons wasting their money, but thats because those myths are not ours. I obsess over bicycles and food, my friend Shuvam over cameras and stereos, my uncle over football; in all these cases the functional needs can be quite adequately met very quickly and with little emotional involvement. However, the emotional involvement is a big reason why we buy anything beyond our necessities. The value of bonding with Starbucks regulars over long lines on the special day is no different from attending an alumni meet and chatting with people who you otherwise share so little with that you haven't called them in twenty years. It is also no more useful than cycling 500km over desolate Himalayan roads. We all need and look for more than the direct technical benefit a product provides us; we need myths and memories from the pedestrian products that populate our lives. The best brands know how to latch on to that.

Once, before the advent of the material explosion, religion and warfare filled this need. I personally am glad we have moved on to iPhones instead.

Dodging Battle

2017-10-29T22:32:00.002+05:30

Security is obviously on everyone’s mind these days with the media gleefully broadcasting with unfailing regularity yet another marquee name falling victim to a breach. Much energy goes into discussing the latest attacks and protections and responses, peppered with generous doses of doomsday scenarios. The talk is all of battle, of fighting off the evil hacker enemy. It may not be the losing battle that it looks like at the moment -remember the war against viruses once looked similarly bleak - but its certainly not looking easy.

As the great Sun Tsu probably said, the greatest victory is avoiding battle. And even in this cyber-battle, there are a few ways to do that.

Tokenize internally

Dealing with sensitive information is a necessity, but there are ways to make it less sensitive. The most effective - mask sensitive data at source and keep it masked all the time. We’ve been doing this to passwords for years but have never gone beyond it. If you store sensitive information in the clear, if applications can read and manipulate sensitive data then its only a matter of time before someone figures out how to extract it by getting past your defences. For years, our most sensitive data were passwords; we learned to converted it into a one-way-hashed token once and then allow both people and applications access only to the hash because some bright brain figure out that knowledge of the actual value of the password was not needed, only a way to compare the hashes reliably. The “one-way” part of the hash ensures that one cannot reverse the process and extract the original password value from the hash. With cloud applications and OAuth we went even further, distancing the tokenisation itself from internal applications. Yes, we still need to protect the hash, but its a lot harder to extract useful information from a compromised one-way hash, making it less valuable and hence less likely to be targeted. It can even be sent abroad without setting off any legal issues of cross-border privacy, since the token isn’t the actual protected data.

Take, for instance, a person’s PAN number. Today our internal databases store the PAN, and applications use it. Yet, we can replace the PAN with any ten-character hash and let systems store and use that hash instead. Most applications will not need any change - they only need to compare PAN numbers with other PAN numbers to see if there are matches, and that can work just as well with the hash. Only some will (such as those that try and validate the PAN, for instance); those applications can be reworked. As a first stage, the tokenisation can happen inside the boundary of a company - the first time the company receives an input - but can later move to an external service (even the IT department itself, maybe). This reduces the number of systems that need bulletproof protection, allowing a company to focus (always finite) resources into fewer places.

If you don’t have Helen, people won’t be so interested in invading Troy.

Encrypt with the receiver’s key

Every company sends lots of stuff to other companies; either message streams or files. The standard procedure is to encrypt them at rest as part of the underlying file system, and in transit by encrypting the channel. However, the file or message stream is nevertheless readable by applications and people within the company itself, and thus remains a target for theft. Companies spend a lot of time and effort getting over this weakness, such as private connections, multiple levels of access, etc.

Lets think of it another way. For years, we have had PKI - a way to encrypt something that can only be decrypted by the receiver. Anyone intercepting the message or file would not be able to do anything with it. We use SSL to encrypt the transmission channel, which uses the very same PKI. Now lets apply the same principle, not to the channel but to the messages themselves. We encrypt each message with the receiver’s public key; in that case only the receiver can decrypt the message. So if Company A is sending it to Company B, A encrypts the message using the public key of B, and B decrypts it using his private key. This is done for each message. The same can be done with files.

The effect of this is quite interesting. The encrypted packets or files can now not be read by anyone - hacker, middleman, even by employees and systems of Company A (so no point bribing anyone). If the encryption happens at the time of generating the packet or file itself, then the bare data is never on Company A’s system. Company A does not have to go to any great lengths to protect the encrypted packet or file from being stolen, since there is little value in it. One can even store it openly on public networks. This is very different from encrypting the channel, where hacking in at either end or as a man in the middle can yield huge dividends. The only way to hack this is to steal Company B’s keys, which is security that Company A is not responsible for.

The reverse of this technique is used quite often - that of signed requests. Every Aadhar request, for instance, has to be signed by the sender. The encryption process and load is exactly the same; only the intent is non-repudiation rather than security. And the technology is obviously quite feasible.

No one will battle to steal a locked box if one cannot open it.

Move your perimeter

The big challenge of perimeter security is that you have to let legitimate outsiders in while at the same time preventing malicious ones. The more the need for porosity, the bigger the chance that some bad actor will slip in. What if we did not have that problem? What if we could move all external interaction to a border far away, so that even if it was breached there was no way back from the border to the core? Chinese Emperors survived assassinations for centuries by allowing no one into the forbidden city - all external interaction was with interlocutors.

This is the easiest implementation of the idea - trusted intermediaries who handle all external interactions. The intermediaries are known beforehand, and will typically be much smaller than the total number of external actors a company has to deal with, allowing for concentration of security procedures. Data interactions - both messages and files - can go through this intermediary which acts as a protocol break. The air gap between the real perimeter of the company and the perimeter exposed to the outside participants is what keeps the company’s perimeter safe. The company sends the data to the intermediary, and whoever is supposed to receive it is told to approach the intermediary and collect it. There’s a matter of managing the access rights of the external partner to the intermediary, but this is no more than managing the same rights on one’s own perimeter.

In today’s world, the most effective outside perimeter is the public cloud. Done properly, public cloud is quite difficult to attack, but is also completely isolated from a company’s own network. The company should use API calls rather than networking protocols to copy data onto the cloud, thereby ensuring that only one way communication happens. In addition, companies can use private, direct lines onto the cloud so that even the API calls (to - lets say - Amazon S3 storage) does not require a company’s servers to connect to the Internet. Of course, it need not be a public cloud provider - it can be any intermediary that implements a one-way protocol.

In conjunction with the previous two techniques, this shrinks the attack surface even more dramatically and makes the company a very unattractive target. There’s no point attacking the perimeter if the actual defender is somewhere else.

History Channel

2017-08-11T15:46:00.000+05:30

Analytics, we are repeatedly told, will enable us to have these wonderful insights that will change our business and transform society, probably abolish world hunger at the same time. And the only reason they're not doing so is because we have not been able to implement it well.

All true, but to my mind a bit of bollocks. Its a bit like Nike advertising the same shoe that Michael Jordan has, and the only reason we're not in the NBA Basketball Hall of Fame yet is that we just haven't put in the hours of practice required.

Here's the problem with insight in analytics. One, insightful analytics is very very hard. It requires a lot of investment; to go back to the Jordan analogy it takes thousands of hours of practice to get anywhere near the NBA. One basically sacrifices everything else in life - something the average person is just not going to be able to do. A company whose job is analytics will put in the investment (an investment research firm, for instance) but a business that is busy doing some other core activity (as most companies are) is unlikely to.

Second, it takes more than investment and effort. 10,000 hours of effort will produce a very good basketballer, but it cannot guarantee a Jordan. Insights are rare, and in addition to investment and effort takes talented analysts and luck. Yes, luck - something that product companies don't like telling you because it isn't on the pricelist. Newton's brain wasn't enough - he also had to be under the apple tree at the right time. This puts a bit of a dent in that ROI thing, and probably explains why the extensive investments in analytics dont really lead to daily game-changing insights.

So am I advocating tossing out the analytics baby and getting on with other things? Not at all. I'm just advocating getting out of the insight steamboat, where the promise is of untold riches if only one buys a ticket to the magic kingdom. Forget using analytics for insight, but don't by any stretch forget analytics, because...

Analytics are about hindsight.

The real value of analytics to a company is the ability to see what has gone before, and learn from that; not to change the game but play the same game better. Hindsight is not 20-20 unless you study it (which is why history is so often repeated). If one has to learn the lessons of history, one needs data and analysis that describes well what exactly was happening - and what is likely to indicate its recurrence. Why my sales fell last quarter is a question of insight - but where they fell, by how much they fell, what else was happening the day they fell - these are all questions of hindsight. And these questions are much easier to answer predictably and reliably than the why.

Once you look at analytics through the hindsight goggles a lot more starts to make sense. Gathering more data points gives you better detail on what happened. Gathering more sources gives you breath of visibility on all the different things that may have happened. Correlation helps you identify what happened before, after or along with what other things. And every time you have a theory on something, analytics tells you (often quite accurately) if the data so far supports your theory or not.

This is quite analogous to the scientific process. One does not carry out experiments hoping to stumble on an insight; instead one tries to analyse experimental data to see if a potential insight is indeed backed by fact or merely a misguided fancy. Computers are not well suited to mull out insights from half-formed ideas - that is the domain of human brains. Computer software, on the other hand, is supremely good at crunching data to validate or junk an intuition.

Humans have since ancient times hoped for a way to reveal what is going to happen, and have repeatedly come away disappointed. Figuring out what has happened, and repeating the successes while side-stepping the failures may be more pedestrian but it is ultimately more rewarding. And that's not an insight; after years of looking for patters in the sand that's hindsight.

Layer Cake

2017-08-10T10:35:00.000+05:30

The three-tier architecture has dominated application deployment for decades. Presentation Layer, Logic Layer, Data Layer - every application designer is fed this bag of chips again and again, till every other chip is forgotten. Most companies have rigid policies that ensure that this holy trinity of layers is hammered into every policy document and SOP.

But...

As is true of so many religions, this one is also a bit outdated (Microsoft has even discontinued that page in their architecture). The theory was that many presentation layers could reuse the same business logic (encapsulated neatly in a application layer) but in my many years of slaving at these things I've rarely seen an app layer used by anything more than a single presentation layer as part of a single application. Indeed, this model was created for the client-server world and should have been thrown out when the world moved on but it was what everyone knew in those days so it somehow made it to the web world. A whole rationale was created to retrofit that design. Presentation became Web, Logic became App, Data remained Data and three tier became three-layer. The roles changed a bit too - Web layer was to stick to static web serving, App became the big daddy doing business logic but now also UI interactions, while Database remained where it was.

As long as things were small in scope there was not much pain involved, but seams started stretching quickly as sites started doing really big things. The first big casualty was scalability. Lets see how.

Web applications are supposed to have horizontal scale with hundreds of small front-end servers; this sounds fine in the three-tier till you start working out the details. Lets say the standard farm started with 2 web, 2 App and 2 Database servers. Each Web server connects to each App server, leading to a total of four connections. The same four connections are required between App and DB, so a grand total of eight connections. However, these 8 connections are actually giving you a horizontal scale of only 2 - since the consumer has only two entry points to be used with a load balancer. Now say you want a hundred entry points to be load balanced - that's a hundred web servers. If you limit yourself to two app servers, that's nevertheless 200 connections. Of course a hundred web servers cant be serviced by 2 app servers - say 10 are required. That's 1,000 connections. And on the database side, if say 4 DBs are required thats 400 connections there too, a grand total of 1,400 connections to create, maintain, firewall and secure. As you can see, this becomes quickly unmanageable.

An option is to scale in full modules - ie a hundred web servers connected one-on-one to a hundred app and database servers. This solves your connection problem but pushes your server requirements through the roof, not to mention making stateful applications a nightmare. Of course, given that web servers serving static requests use very little CPU, we can merge Web and App servers (especially since App server also runs a web server anyway) but this will certainly fall afoul of your security policies. You need three firewalls, they will say. Why, because three layers need three firewalls. Sounds a bit circular, but that's what you'll face.

Can security be managed with less than three layers? Of course it can - why should the number of firewalls be dependent on the number of layers. Put all three firewalls on the public edge (or two before the web+app server and one before the database) if you want. The insistence of a firewall between web and app comes from the old separation of presentation and logic servers, where only the presentation layer talked to the logic layer and no direct access was allowed to the app layer. Also, security tended to be layered into zones - all databases in one zone, etc - because firewalls were expensive and limited resources. Today web servers do static things and app servers do dynamic things (using a web server) so the same ports and protocol are open on both web and app servers. The web server merely passes the dynamic request along. If someone can compromise the web server then he can by definition also compromise the web server on the "app" server. Firewalls, meanwhile, are no longer expensive resources; micro-segmentation allows even a single database to have its own personal firewall.

Then there's the small matter of changing things. The three-layer architecture is notoriously difficult to do tiny changes to. Everything requires a full build and thus a full round of testing for the entire app. Want to change a comma? Run a full build and test cycle. Microservices? Sure, just create an application instance each time (did I tell you about the build and test process already?). Look a bit closer, and you can see a clear shadow of the earlier practices of building, compiling and deploying EXE applications. Modern applications are closer to document editing than traditional build-compile-deploy - edit, preview, publish, version - but the shift in paradigms is not an easy one.

Modern web applications are more akin to a network of interconnected processes, some interacting with each other and some with human users. They are not monoliths with a single neat front, middle and back. This makes them far more flexible and scaleable than the 3-layer model. Indeed, the "network of components" model an incorporate some very complex behaviour, where one service comes from one provider in one domain and another from another provider in a different domain. It is not unusual today to have applications where authentication services are provided by Google, storage by Amazon S3, analytics by an on-premise Tableau, maps by OpenStreetMaps, database by Zoho, ticketing from Zendesk, monitoring from and log management by Splunk. Such an app - impossible under the 3-layer model - will usually outperform and outscale traditional models and be cheaper and faster to build to boot.

A few big changes in web applications really push against this three-layer model. The first is the emergence of markup languages where all the work of the application (presentation and logic) are handled by the web server. The original CGI model of the web server handing control to another EXE for logic processing and then receiving the output was thrown out of the window with PHP, which even today runs the bulk of the web (including Facebook). The second was JavaScript and the emergence of the Single Page Architecture, where the entire UI shifted to the client outside the boundary of the company's data center, and the servers became mere vehicles for feeding static objects and JSON data to the client.

The final blow to the coffin are serverless architectures, where micro-services run directly on a PAAS container with neither web nor app nor DB layers visible. Each of these movement have huge advantages over traditional design - markup languages make comma changes a breeze and allow for true DevOps. Single page architecture is often the true decoupling of presentation and app; it has made app development dramatically easier and more democratic (think Facebook apps). Serverless architecture is by far the easiest to scale massively on the cheap. Isolation and micro-segmentation, something public clouds do in their sleep, obviates the need for the expensive zone-based 3-firewall approach companies continue to favour.

For all the proven scale and benefits, only a handful of companies that are not startups make use of these alternatives internally. Why? Because they're still baking the layer cake.

Changing a Password

2015-09-26T15:56:00.004+05:30

It's that time of the month.

My corporate email account has started warning me of dire consequences if I don't change my password. I'm going to have to start thinking again of a hard-to-guess easy-to-remember never-used-before nonsense string that will be my companion till thirty days do us part.

Now security experts are always telling me "industry best practice" dictates that I change my password ever so often. I never quite believed it, because it just did not add up. There just didn't seem to be a feasible attack that could take advantage of this kind of hole. After all, password changes matter only if someone is already accessing your account. If your password did get compromised, what kind of hacker would wait thirty days to take advantage of it? There are indeed some stalker scenarios where this kind of thing can be useful, where I might want to see all the emails being exchanged without doing anything for as long as that window is open but in other scenarios, you want to be in and out quickly, not wait thirty or ninety days. This seems like a case for a password that's hard to guess in the first place rather than one that changes often.

What is the real world analogy? Its not unlike asking a user to change his or her door locks every 90 days. A key, like a password, is easily copied. Both are suspect to simple attacks that anyone with simple skills and access to either the user and a bit of social engineering, or the lock itself, can gain access and retain it till the lock is changed. Yet no one recommends to homeowners that they change their locks every so many days, and burglars seem very reluctant to take advantage of all those millions of unchanged locks.

On the other hand, the problems of frequent password changes are well documented - written down passwords, predictable patterns, high support costs and so on. Indeed, there is strong evidence that frequent password changes reduces security by making it easier to guess the password (in which case most safeguards are out of the window). To prevent the harm, more rules have been added making passwords less repeatable, and therefore even more inconvenient.

How this policy came to be "industry standard" is not clear, which is surprising given all the debate surrounding it. The best I can figure out from literature is that passwords were originally set to expire before a brute force attack could reasonably be expected to crack the hashes. However, it seems to me that that is an argument for making passwords stronger, not changing them which we know leads to weaker passwords. I can come up with one or two strong but easy to remember passwords without killing too may grey cells, but asking me to do so every thirty days is like Sachin to hit a six every innings (oh wait, did he do that...?). For instance, if the minimum password length was increased to 12 characters (which gives you some 2,000 years of brute force cracking with your iPhone) could we do away with password changes? And of course, there's the itsy bitsy matter of supercomputers and massive bot clusters, which can chomp through even the longer passwords like candy - but if you're against one of those no frequency of password change will help. Also, the danger here is that the password hashes are compromised, which allows the attacker to take time cracking the hashes. Again, there are more effective steps to prevent that from happening than password change policies.

Is my rant backed up with anything? As it turns out, it is. There are lots of arguments (I've listed some links at the end) and some academic discussions too and, as far as I can make out, the overwhelming consensus is to get rid of it. Particularly illuminating was a Microsoft Research paper on the topic. Even NIST, the fount of much of best practice lore, gives no ringing endorsement that industry standard bearers would have hoped in its policy draft and is indeed strikes a very worried note in its study of actual password usage and attitudes. In the ACM, Zhang et al of UNC also decries the ineffectiveness of password expiry (albeit in a math-filled, tough-read kind of way), with the authors concluding "We believe our study calls into question the merit of continuing the practice of password expiration". Possibly the strongest indictment comes from Gene Spafford at Purdue's CERIAS who slams expiry policy as a "myth".

Then there's real world evidence - Citibank globally does not have a password expiration policy, neither does Amex or, closer at home, ICICI Bank. Yet, in spite of a fair amount of incentive to hack these accounts, the banks seem to have survived millions of customers for at least a decade without issue. Nor does GMail, Apple, Facebook or Yahoo with their hundreds of millions of accounts ask such expiry menstruations of you, and they are probably attacked more than Amex. When both motive and opportunity are so plentiful, maybe one wonders if this is really an open window at all.

So, known damage or uncertain theoretical protection? All security is a compromise, but this is one compromise that does not seem worth asking for. Instead, there are steps one can take to make passwords themselves stronger, or protecting the password hashes that evidence says actually works. Finally, MOVE AWAY from passwords.

Articles

Academic Papers

iPhone 6: two weeks on

2014-10-12T17:19:00.001+05:30

Two weeks ago, in a journey that brought back memories of hunting down Casio organizers, I trudged the corridoors of Funan Digital Mall in Singapore till I got my hands on an iPhone 6. Here's my impression of it two weeks later.

Right off the bat, I must say I love it. Yes, I'm an Apple fanboy who has had an iPhone for many years, (though I've used an Android on and off and before that lived on a Blackberry). I also have a couple of iPads and iPods, though never a Mac. That bias aside, however, the iPhone 6 is a seriously nice phone - lovely build quality, beautiful screen, wonderful to hold in your hand, blazngly fast and as loaded as you can want. Mine is a 128GB, though I'm still wondering what I to do with the 80GB still left after I put in every song and photo I had. As phone goes there is little to quibble about with what it has.

First, the size. Yes, Jobs was right in that it is a bit of a stretch - a few things require finger gymnastics. However, he was wrong too - visually a screen of that size is gorgeous enough to put up wth a bit of jugglery. A couple of days ago, I restarted my Four, and was immediately struck by how inadequate and small it looked. The six plus has an even bigger screen, but that is clearly a two-handed phone. This one is fine for one-handed, except for the occassional twisting. Its also one of the sharest and brightest screens around - quite comparable to the best of Samsung's offerings. Its also far more oleophobic than the Four was; indeed nearly smudge free even in the stickiest of Mumbai weather.

(Not ditching my iPad Mini though - for many things size matters more than resolution)

Then there's the new look, which does unfortunately look a bit like a Samsung (especially the white version that I bought). Not that I care very much; the rounded seamless glass-metal look is understated but beautiful. And before you say Jobs again, the original iPhone 2 (the one that changed the world) looked like this. Does it bend? Haven't tried it, but it certainly does not feel bendable. Its very light, but looks quite solid. All the other stuff - its fast, the fingerprint scanner is very fuss free, the camera is great, the apps run frictionlessly and all that. Battery life I found to be about as much as before, maybe slightly better, certainly no worse.

Bottom line, its as great a phone as you can find even though the top-end price point is a bit eye-watering (though I find the cost comparions with, say, Xiaomi, similar to matching a Rolex with a Titan). As an upgrade - do it blindly especially if you're on the Four or even the Five (though I'd leave the 5S alone). As a new phone, I give it a more nuanced but still very strong recommendation. Yes, there are worthy Android competitors, but the iPhone 6 is definitely up there among the top few.

Two thoughts, however, pop into my head.

The first is that Apple is peaking out on how much more perfect it can make the phone - the technology leaps have been getting smaller. The iPhone 2 started the smartphone revolution, the Four introduced the eyepoppine retina display, the Five S gave us the reasonably magical fingerprint scanner, the Six is merely bigger and faster; improved implementations of technologies already in the market. As the PC world shows, bigger and faster does continue to sell for a while but eventually starts peaking out. The problem is, there aren't that many tech leaps left - I would think something like texture-enabled displays or an incredible battery that charges in minutes and lasts days - the iPhone 7 would need one of those.

The second thought is that while Samsung Sony HTC Xiaomi go in for device gimmickery or price, Apple continues to build out an incredible eco-system. Its hard to move away from that world, especially when a few clicks gets you every song you ever liked - English, Bollywood or Bengali in my case; even the Tamil stuff I started listening to in hostel. Apps, podcasts, movies, books, a wonderful rich, beautifully integrated eco-system that one never needs to leave and Android has yet to match. I'm hoping FlipKart does integrated Apple Pay soon. The first wave of magical phones created the mass adoption needed for this eco-system, but it now has its own momentum. No doubt this, like everything in the world, this will eventually die but I would give it another decade at least. Till then, Apple just needs to produce great phones, not necessarily phones that are technological magic like the first few were.

Meanwhile, let me get back to crushing even nicer candy.

The Song of Socialism II: Penetration

2014-05-31T23:24:00.000+05:30

Social is the new buzzword, and what a loud buzz that is nowadays.

In the last post I described my theory of levels of social media - Presence, Participation, Penetration and Platform. Presence was passive, Participation was active entry into the world of social media. Penetration and Platform are quite a bit more complex. This part two of the series gets into the next P – Penetration. I’ll need another post for Platform.

To understand where I’m going with this, lets see what a social network really is.

The web started off as a way to connect content together. Any piece of content could easily and simply refer to another piece of content, even if it was on a different server in a different country. A browser allowed people to move from content item to content item, thus making the universe of the world wide web a wonderful store of content. People soon realized that this could be extended to all kinds of transactions – one could pass not just content but messages and information, and not just between humans but between humans and computers or even between two computers. The simple text foundations of the world wide web were thus abstracted into a complete transaction system – a multilocation multiserver computing universe we call the Internet. Facebook, like any other web application, sits on these foundations but it’s more. Facebook is a platform, a whole layer of paradigm on top of the foundations of the web that provides an abstraction for your real life friendships and social connections – a big chunk of the people on your Facebook friend list are people you knows in real life. People are still getting used to the web itself, so they often miss this fact. Its not just about content, its about content and transactions in the context of your own friends.

Presence and Participation is still at the level of the web paradigm – that of content and message sharing. Facebook improves the distribution and sharing, but in this context its just a better mousetrap. Presence is an improved version of the “Rate This” button, while participation, though an improvement over emailing an interesting link to your friends, is not that different.

Facebook, however, offers an extra layer - that of a model of interactions between actual people. The web has no concept of persons– only content - and websites have therefore to work hard at creating this extra overlay through sessions, registrations, logins etc. If you live inside Facebook rather than on the raw Internet, you have a readymade environment where every participant has an identity and you need not worry either about registration or about login.

Presence and Participation are useful tools, of course, and depending on the business needs you have can yield wonderful results when exploited properly. Facebook is like an airport – a business setting up there needs to exploit more than just walls and a roof that could be anywhere – they need to exploit the fact that they are in a space that people congregate and interact in a particular fashion. A duty-free store, for instance, knows the nature of travellers flowing through their space much more precisely and in addition the exact identity of every shopper from the scan of the ticket; this allows them to exploit some very precise optimizations. The product mixes and offers in these stores reflect this knowledge, eliminate clearance sales and make these stores some of the most profitable in the world

That’s where the next step comes in – penetrating the social network, injecting yourself into other people’s networks using the reach that Facebook offers you

Penetration

The first step to penetrate into the world of social is a Facebook App.

Technologically, this Facebook App is a web application that sits inside Facebook. Doesn’t sound like much, and plenty of Facebook Apps indeed don’t do much more that a web app. What is possible, however, is much more and the first people to take advantage of this were game developers.

Video games, the kind you played on PCs (and increasingly on mobiles and tablets) has been multiplayer since the days of dialup, but to do so you had to explicitly join a multiplayer gaming network – either local ones or big boys such as XBox Live, PlayStation Online or Battle.Net. These networks weren’t trivial – game publishers went through considerable effort and expense to create and maintain these networks. Then came Farmville.

Farmville was a game for the Facebook world. Nominally playable in standalone more, in practice playing Farmville needed lots of help from friends. And those friends came not from some custom or generic game player network, but from Facebook. Zynga’s genius was realizing that Facebook is more than a simple hosting platform with a lot of users – it’s a readymade network that is similar to gaming netoworks, but better. Why better? because all kinds people are already on it. The game playing networks only had game players – indeed each game had a different network and you had to build friends and allies from scratch for every game. Farmville, on the other hand, is for playing with your real life friends. This is how games work in real life; you play golf or bridge or poker with your existing friends, not with strangers at a golf or bridge club.The secret to Farmville’s success was in enabling interaction with existing friends. You can visit their farms, see how their crops are progressing, help them out with feed or fertilizer if they need it – this kind of interaction is hardly exciting with strangers.

The challenge with using networks has, till the advent of Facebook, been that you had to first create a network before you could make use of networking effects. Anyone who joined had to encourage enough friends to join, and there was no way for the network to know proactively who to invite. Facebook provides companies an easy and powerful way to bypass this problem; a network friends is ready for your consumption and only some inviting and asking for permission is needed.

Zynga became an overnight sensation on the back of Farmville, and Farmville became this viral phenomenon by using Facebook to encourage all the friends of players to play and to help. Earlier, if you had 10,000 fans, you could get your message out to 10,000 people. Now, if each player had 100 friends, your message was suddenly reaching not 10,000 but 1,000,000 people! Farmville had no existence outside the Facebook network – you logged in to Facebook to play it, and Farmville used this login to connect to your friends, send them a stead stream of messages on your farming progress, encourage them to play and help you along while at it. This helped it reach 10 million players within six weeks of its July 2009 launch. Its interesting to compare it with Farm Town, a very similar game launched months before Farmville – its also very successful, but never achieved Farmville’s levels of glory, primarily because it was focused on interactions with strangers while Farmville was resolutely about interacting with your own friends.

Scrabble, Words with Friends and Draw Something are more games that rode to success on the backs of the social features of Facebook. Scrabble needs at least two players, and computer Scrabble isn’t much fun; two Kolkata engineers caught on early, and created Scrabulous on Facebook in 2007 for playing with – you guessed it – your friends; it quickly became the top game on Facebook. When Scrabble owners Hasbro eventually shut them down, they replaced it with a game that still ran on Facebook but used a different network – a game network called Origin – for multiplayer gaming; Hasbro believed in the traditional, not the new fangled Facebook way. Of course, this wasn’t a resounding success – it still supports Origin but Facebook is by far its key channel (for copyright reasons, there are two versions totaling 1.7million users monthly). Words with Friends launched in 2009, and is still hugely popular – 8 million users monthly– while social drawing app Draw Something is chugging along at 2 million.

Here’s a couple of key things they all do, that only Facebook allows them to do:

They don’t build separate networks of their own; they gather details from your Facebook friend lists, like names and pictures, and give a very real verisimilitude of interacting with friends. They all allow you to see which of your friends are playing, play with them and invite people not already playing.
They all post on your behalf to your Facebook feed, telling even your friends with no interest in those games how you’re doing on them. Even Scrabble-haters applauded when I hit 101 points on a single word (and Scrabble helpfully posted it on my wall).
They analyze and report how you’re doing within your peer group, how far ahead or behind your friends on various criteria.

And these features, easily accessed through the Facebook API, are available to any App. Game developers are not the only ones making use of it; birthday reminders, activity planners, social shoppers – many apps take advantage of this. The trick is not so much the technology as figuring out the use case that makes sense. For instance, paying your bills is usually a solo activity; clicking photos or planning a party are not.

I must point out here that the other mega phenomenon, WhatsApp, solves the network problem in a different way. It uses your mobile phone book as your social network (which, if you think about it, is perfectly true). It does not ask you to give a bunch of information during registration, nor does it ask you to invite anyone or “select your friends” like, say, Yahoo Messenger had to do. Your address book already tells WhatsApp who you’re connected with.

That’s Penetration, but we’re not done yet – really good companies use Facebook as a full-fledged platform, an engine of business rather than just an enabler. That’s for the next post.

The Song of Socialism

2014-05-26T18:41:00.000+05:30

Social is the new buzzword, and what a loud buzz that is nowadays.

Everyone from sedate bank to hippy dippy start-up wants to be "social". Being unsocial nowadays is like admitting to a fondness for bellbottoms - hopelessly anachronistic and just a little worrying. Its not unlike the early days of the Internet where everybody wanted to "be on the web" and "have a domain". What does social mean, though – does it mean putting up a Facebook page? Does it mean obtaining a twitter handle? Does it mean encouraging a bunch of people to like you on foursquare? And what does good this do my business anyway?

This rush to keep up with the social Joneses lead to some absurd decisions. Companies put up, sometimes at considerable expense, a "Facebook presence" and then proceed to be puzzled when no benefit accrues from all this. As with many technologies in their early days, people glimpse vast potential but simply cannot figure out how to convert this potential into actual gain. I spend a lot of time my time using social media, and it set me thinking on how it works, how companies can take advantage of it. Not unexpectedly, I find the most effective uses of social networking comes from game developers (at least that's my official excuse for playing Scrabble on company time). I will use games - especially the hugely successful Candy Crush - as my examples (assuming of course that you play games - what kind of a social person are you otherwise?)

There are, I feel, four levels of social media involvement (and it took my consultant avatar only a short while to come up with the relevant four Ps) providing different levels of value to organizations. They are, in increasing order of value:

Presence
Participation
Penetration
Platform

Consider a real life analogy: Presence is like a speaker in the park. People come up to the speaker to listen, and possibly take away a brochure or fill a form. Participation can be thought of as the next step, where the listeners include the speaker in their conversations, invite him to parties, engage in dialogues, willingly sharing his or her identity with the speaker and may even assist in enlarging the conversation by encouraging his or her friends to also connect with the speaker. If this is successful and grows, it becomes a campaign (Penetration), or even a movement (Platform).

I’m sticking to Facebook for the examples, but you can apply the stuff to other social sites too.

Presence

The first level - and the simplest - is about presence. The quick way to social is to put a Facebook Social Plug-ins on your own page – “Like” and “Share” buttons or “Activity Feeds”. These are simple javascript widgets, small snippets of code easily included into your own pages.

Presence is just a small extension of your existing web presence into the social world. It gets you the tick mark, it makes your page look social and it does allow for sharing and commenting. It does not hurt to have a flavour of Facebook on your existing pages, it makes sharing content a little bit easier and does encourage a bit of virality; things like blog posts and event announcements benefit from easy sharing. As a company though, your upside is limited.

Presence gives you two things. One – a popularity proxy that is a little bit better than the old proxy – visitor numbers. Two – it makes sharing the content on your page a whole lot easier – and thus gives you a chance at virality. The companies that benefit most from these are companies that want the maximum number of people to read their content, such as a newspaper or an e-commerce site; these numbers are much less useful, for instance, if you’re selling machine parts to power plants, or other businesses that do not really care about popularity or virality.

Participation

The next level of engagement (and currently the most popular flavor) is to put up a Facebook Page. It’s a noticeable step for many companies, since it involves creating a new web presence, not part of any existing web presence. The technology is not challenging – indeed this is little more than brochureware hosted within Facebook. Ten years ago, having a website was the latest in cool, today it's a Facebook page.

Participation is more than presence, which was all passive. Here, you have to constant contribute to your Facebook page to keep the beast alive. Pages that have more frequent contributions by the page owner usually have more fans. More than a website, a Facebook page (or twitter handle or google+ page) is all about fresh, frequent updates. And no, comments or replies to comments do not count.

How does a company gain from participation – from putting that extra effort into creating and continually posting to a Facebook page? What does a Facebook page give you that adding those widgets to your existing web page does not? The gain is easy to see in entrepreneurships and small companies who can’t go about creating fancy web presences or splurge on SEO budgets, but why large companies? You will see a lot of statistics on how many users there are on Facebook (a hundred million in India), how they spend a lot of time there, how you should be there because everyone – are those good reasons? Think of it this way - if population was all that important, India would be a bigger market that America; what decides a company’s market share is much more than just population.

Presence made it easy for a Facebook user to share content, but how do you persuade them to share? Also, each share is relatively shallow - if that person has a few hundred friends, that’s the number seeing the share. You have to get a lot of shares before numbers start to tally up. Participation, on the other hand, pushes this content out to all your fans - it gives a company much more influence, a louder voice. Everything you post on your company’s Facebook page appears as a wall post on your fan’s pages, to be seen by all their friends. If you have – say ten thousand fans, every post on your Facebook page or Twitter feed is straightaway seen by ten thousand people. No waiting for people to lift their lazy fingers and press the share button. On average, a Facebook Page has 4,596 fans; Barack Obama has nearly 7 million – that’s a very loud voice. CEOs of some companies would do well to have that kind of voice (of course, as mentioned earlier, you’d better have something to say). And these fans can also talk to you, or even each other.

There’s another benefit. One of the challenges of the web always has been that it is essentially that people don’t have any identity. You can easily find out how many people visited your website, but who they are, what sex, what age, all that is invisible. Websites try many things to get a better idea – registration drives, cookie analytics, loyalty schemes – but the great thing about Facebook is that everyone there already has an identity. Even better, the identities are mostly real; a big percentage of the data is correct. Names, faces, birthdays, education, jobs– people usually don’t lie or fake it on Facebook since it would otherwise spoil the social experience. You can learn a lot of real life stuff about your fans on Facebook; far more than almost any other way.

Two Ps down, Two to go. Its in the next post.

Flipping Myntra

2014-05-25T12:48:00.001+05:30

E-Commerce is close to my heart, and I have been asked many times in the recent past what I think of Flipkart buying Myntra. And really, the one thing I feel is dread. I really do like both Flipkart and Myntra. Both have good products, reliable service, convenient sites and great customer support. I have been frequent purchasers from both; everything from shoes to mobile phones. Both are businesses I hope will survive, as I hope e-commerce in general will.

Why the dread, then?

Its because I don't understand the deal, the reason FlipKart was willing to shell out a generous chunk of change to buy a good but not great brand. And what it seems to portend for the etailing category as a whole. People have pointed me to Facebook's purchase of Instagram or WhatsApp as the template, but I see more Compaq than WhatsApp in this deal. Instagram and WhatsApp were both global leaders, dominant in markets that Facebook was lagging in but hoping to penetrate. Myntra offers FlipKart neither - not a dominant brand, nor a new customer base, not even a category that cannot be organically established at a much smaller cost equally quickly. HP bought Compaq more in an attempt to consolidate in a falling market that continues to fall, where growth in marketshare could only be achieved with these kinds of stunts - and destroyed considerable value in the process. This one has the same smell. Will the combined entity have better fundamentals - more profitable, more competitive, more attractive to customers? I have a hard time seeing how.

I find it hard to believe this is a reaction to Amazon, which has never done much in apparel globally. The most cynical explanation is that this is an accounting trick - allowing the venture funds that own so much of each company to book a big exit and bolster their own record while basically transferring money from one pocket to another. If youre a little less cynical, you will think of synergies, of marketshare, of category dominance and jargon like that; all valid stuff but very short on specifics. And forget about the fact that both companies are burning money like petrol in a Ferrari.

Everyone has heard of the Amazon lesson - grow big fast. What people forget that Amazon's enormous hunger for money was to power its investments. It did not spend anything significant on advertising in its first few years, and even now mas a relatively muted marketing drive. Instead, it spends on servers, on technology, on warehouses, on supply chains and on people - assets that have helped it grow to a $60bn business. It is also the sole e-retail venture in the top ten that is not an offshoot of a retail chain while other aggressive cash burners like Boo.com, Buy.com or Pets.com flamed out entirely or survive as shadows of their former glory. Some digging into Amazon's balance sheet seems to indicate that it is operational quite profitable. Amazon offers nothing for free - extra conveniences are either charged for, or you are locked into a subscription. All that extra money goes into investments such as cloud or Kindle or drones delivering packages, not customer subsidies.

When I heading FutureBazaar in 2006-7, we soon realised that the natural, profitable size of this business was about Rs 65 crore a year based on the number of customers we could attract who would purchase reasonable amounts and pay for things like shipping. We, however, wanted to excite venture capital, raise money and become Amazon - this 'excitement' number was Rs 300 crore a year - not much money was available below this. The remaining sales could not come from profitable customers; we could only generate this kind of topline by impossible discounts, sweetheart delivery terms and that biggest money-burner of all - cash on delivery. And we could only do this if there was a steady tap of money flowing in on the other side. Its a kind of reverse pyramid - customers are doled out liberal subsidies as the companies engage in a beggar-thy-neighbour valuation game. Its good strategy, and worked well for Flipkart, Jabong, Myntra et al till the really big boys came along. Its hard to see how beggar-my-neighbour will succeed against Amazon, or the inevitable entry of WalMart.com and Target.com. Or even a Reliance Retail, if it chooses to do e-commerce correctly and with enough patience.

Venture money backing these businesses are, of course, not fools; they eventually intend to cash out on the growth and marketshare these companies represent either by reversing the pyramid again and doing an IPO, or finding an invester with far deeper pockets such as, say, BlackRock. In my experience, also, many venture capitalists just believe that scale will eventually become profitable; the details of the business dont always interest them. In FutureBazaar, I was pushed repeatedly to offer incredible delivery timelines or cash on delivery - a business model patently financially unviable - because it would lead to growth that they were sure they could encash. There was no concern about how that growth could be sustained, and of course the money that should otherwise have gone into building capacity - warehouse, supply chains and the like that bosltered the unique proposition of e-commerce but had only long term returns - those were not valued much. Remember, the average venture capital investor has an attention span of between three and seven years by which time he has either made money on you or abandoned you; the long long long term does not really incentivise him.

Think of what Amazon is known for. Its e-commerce platform has spun off new directions in cloud and analytics, its warehouses represent an entirely new approach to real time delivery; its ad campaigns are hardly memorable. Unlike Pepsi, which sells an idea, Amazon's success lies in massive investments in a new way to source and supply. Flipkart's success is also built on replicating some of this in India - they built the warehouses and delivery infrastructure that Futurebazaar wanted to build while it was the leader - but the bulk of its humungous money chest goes not into world-beating (or Amazon-beating) innovation but into customer subsidies.

Back to the deal. I fear its a deal for the venture capitalists, not for fundamental business reasons. And for what it says about Indian e-commerce in general. India has gone through three waves of e-commerce plays so far - the early, heady Rediff Shopping, Fabmart and Bazee days, the later, equally promising world of Indiaplaza, EBay and FutureBazaar and now the glory days of Flipkart, Jabong and Myntra. The thing to note is - every player in each previous wave is extinct or nearly so - that points strongly to fundamental business model flaws rather than poor execution, which would have led to some losers and some winners, not the extinction of an entire wave. I fear we are in the same situation again - the wave is bigger than the one before, but it is about to crash. Unfortunately, customers like us who genuinely find e-commerce useful will be left stranded, a market of demand without supply.

Security: A Model Less Simple

2012-08-21T00:19:00.001+05:30

When passing through a full body scan, a pat-down search and a baggage x-ray on your way to buying a coffee in your local mall, you know you live in the era of Security.

It seems to me, however, that enterprise security operates under the sole premise that mal-intention can be prevented. Password policies, network restrictions, biometric sensors, intrusion detection, firewalls, access tokens, truckloads of technology all geared to one single goal - keep the bad guys out. Driven by fear of doomsday scenarios fed into you at numerous conferences, we treat every potential breach as a nuclear explosion and try to prevent it. I doubt anyone other than Jet Li can bring a jet down with a pocket knife (and by definition he's a rare talent), yet we – at great cost and inconvenience - make sure every pocket on a plane is empty of a knife. Even China does not have that many Jet Li's so for that kind of success ratio this is kinda expensive.

Ignored here is the strategy of deterrence - make them afraid of its consequences. This strategy can be seen everywhere the real world, but its oddly absent in the virtual world of information security. People don't steal just because a door is open; more often its because they don't want to go to jail. They know that jail is a possibility because newspapers and politicians and others give ample coverage to crime and its consequences. In societies where punishment is unlikely (such as war-torn Afghanistan) people rarely worry about consequences and crime is undeterred. Companies should similarly publicize penalties and visibly take severe action in case of breaches. An ounce of deterrent is often worth many kilos of prevention, but I've yet to see a company security policy (usually running to hundreds of pages) specify punitive actions. CISOs should get the authority to swing a big stick, and make sure that it's visibly swung when the need arises.

Its simple. Publicise the punishments (make sure they're substantial). One fine sunny day, pick a few violators and make prominent, public examples of them. Repeat every quarter, or as frequently as needed. Even the good old Chanakya recommends it - "saam, dhaam, dand, bhed" - where "dand" is the thing to note here

Prevention, of course, can hardly be neglected. Locked doors and access controls are important, but in the absence of a deterrent this leads to rapidly diminishing utility. Think of a country with no effective police; the bill for private security and electric fences will start climbing rather quickly. This is fine if you live in Wasseypur, but in other cases a few ounces of deterrence is worth more than a few kilos of prevention.

Changing Channels

2012-07-18T09:29:00.001+05:30

Multi-channel – store, phone, online, mail-order - is finally maturing. For the longest time, retailers promised and customers expected this muti-headed hydra of a retailer present seamlessly across every channel, moving between channels with similar lack of seam, permitting mix & match of all kinds. For the longest time, reality was disappointingly different.

Retailers spent the last decade launching anaemic web stores; proudly announcing that they were multi-channel and hence so much better than pure-play e-retailers. Unfortunately those feeble attempts remained the stuff of boardroom backslapping and upbeat press conferences - neither actual shoppers nor pure play e-tailers took much notice. Staples, John Lewis and a few others remained rare exceptions.

What took so much time? Retailers aren't exactly clueless idiots, why didn't this get figured out a long time ago? Futurebazaar was exactly such an attempt; indeed many customers assumed automatically that we were multichannel and land up at a store asking for online or called us asking for store info. Yet for all that goodwill we didn't quite succeed. Let me try and describe what the complications were.

What's in a name?

The first thing that comes to mind when one hears multi-channel is that all (or at least most) products available on any one channel should be available on other channels too. This expectation turns out to be ridiculously hard to meet. The core of the issue is product metadata – information such as pictures, description, sizes, etc. Traditional retail will call a product (say) "40CTWHTFS" – only a merchandizer knows that this is a 40" cotton white full sleeve shirt. Store shoppers don't need barcode labels to know that the shirt is white, but go multichannel and a proper product name is gold. Stores, similarly, do not need pictures or specsheets. Apparel is the greatest challenge – there are more variants of T-Shirts than scientists chasing the Higgs boson.

A key step to multi-channel is an integrated product master. Early in my stint at Futurebazaar, I met Staples and saw the efforts they had put in to make this possible, and they dealt with a mere 20,000 products or so. Future Group had over 500,000 products across all formats, not even counting food – we got somewhere, but the road was hard and very, very long.

Availability

Once you have the data, you can create the listing but to make it available for sale requires another step – deliverability. This leads us to two questions – is the product available, and can it be shipped? Retail chains often have good ERP systems, and one would think the first question is easy to answer, but as usual life is more difficult than it seems. Retail is the king of the bulk supply chain, but will often not track individual variants – you may know how many of some model of clock is in stock but have no idea how many of those are a particular colour. Many products, further, are delivered direct to store from supplier – there you need hooks into the supplier's inventory. Finally, you might not have a good idea of how many pieces in the inventory are unsold – most stores update inventory information only at the end of the day.

Deliverability is another issue; the product may be in stock but the shopper needs a reliable shipping promise. Products further need to be packed into shipping friendly packaging; this takes staff, space, packing materials and the facility may not be available at every store. Multiple possible pickup locations, packing locations, consolidations, inventory movements – the large number of variables make deliverability a huge challenge. Contrast this with a pure e-tailer, who's probably relaxing on the sofa with stock only in one or two centralised, custom-built, heavily automated warehouses.

Instore and Out

Most shoppers, we saw in Futurebazaar, do not respect the boundaries between online and off. For them, multichannel a single company with multiple channels, not multiple sunsidiaries one per channel. This shows up in behaviour like returns – they want to return goods bought online to the nearest store (in an environment where we don't even allow return of goods bought in another store). They want instant inventory checks across channels, prices and promotions to match, inventory to move from here to there. They want to pay anywhere and pick up anywhere else. Worse, they get frustrated when they cannot get it, and turn back to pure-play e-tail with its simpler promise.

Retailers have never faced customers with so much imagination. Banks have barely adopted anywhere banking; this surge of customers demanding anywhere retail is quite unnerving. And customers are demanding – Staples, Wal-Mart, Best Buy all have success stories around this. The problem isn't demand, its cost-effective execution. Retailers have become very good at the existing supply chains; they find it hard to accommodate all these new demands. Stores (especially stores in India) have no space to designate for packing and shipping products bought online. They also need to clear out some aisles to put in kiosks.

Headbangers ball

The real challenge was getting the various divisions of the retail company to participate. Multichannel only works if the traditional channels participate as enthusiastically as the new ones, but sometimes it becomes a case of the ant irritating the elephant. The new channels generate small revenues relative to the established ones while demanding a lot of time and attention – this often creates a lot of friction and requires some headbanging to align interests.

The core issue: the head of the multi-channel initiative is rarely powerful enough in the organization to do this headbanging. I may have been a CEO with a loud voice, but the big boys at Big Bazaar often heard just a whiny, reedy-voiced baby. The department heads of the traditional side do not like being told by the new cool kid that things have to change. Merchandizers, suppliers, logistics people are all amused at the fancy new multi-channel's sales volumes – even Jhumritalayia store does more, you're told sniffily. The result is often an uneven, disconnected experience – some parts of the organization help out, some are more reluctant. Eventually, shoppers get tired of waiting.

The DNA catch

If multichannel is to work, the entire company must be willing to wade into deep waters. Multi-channel isn't another initiative; its an evolution of the entire retail model. This key idea (fashionable to call it DNA) must flow through the entire organization and everything must start becoming multi-channel. Create a multi-channel initiative and you'll either get confusion or a dotcom me-too with few of those promised synergies. Create a multi-channel culture, and you might just start worrying an Amazon.

Pick Up and Cash

2012-07-17T00:44:00.001+05:30

In my last post, I talked about how giving up some of the trappings of physical retail is not without its costs. This post is about the flip side of that coin – in a way.

What caught my eye was Wal-Mart's new facility allowing customers to pay cash for online purchases. The announcement (made a little over two months ago) comes with a twist unfamiliar to those Indians happy with cash on delivery – this one requires you to visit a Wal-Mart store to pay cash for what you bought on the Internet. After same-day pickups, this seems Wal-Marts latest attempt at that famous silver bullet that was supposed to defeat Amazon – multi-channel.

Online shopping is still a small percentage of worldwide retail, but it is expanding far faster than the boring brick kinds. Profits may not be stellar, but growth certainly is - Amazon is climbing the ladder faster than an MBA on steroids. And physical retailers are feeling – if not the heat, then at least the smell of smoke from the flames. After years of stumbling around, trying without much success to beat the virtuals at their own game, they are finally starting to define a new game.

The real multi-channel game.

(A little disclaimer here. Wal-Mart's cash attempt may well turn out to be a fizzy attempt and not, after all, that great new game. Its just what started my train of thought. Stay with me for a bit).

So many channels, so little choice

What is multi-channel?

The holy grail of this idea is for retailers to build relationships with customers through multiple different ways – so that you buy from Wal-Mart through whatever channel is convenient to you at that moment. Its easy to dream up innumerable possibilities, where all the channels interact with each other, allowing the customer to pick the any combination that works best for any particular situation. If you're on the couch, for instance, online probably works better. If you're going out anyway for a beer, it may be easier to pop into the store next door for your baby's nappies. Some products require research and are best done online, some require touch and feel and are best experienced in stores – most are some combination of both. Some things people are willing to wait for (and wouldn't you rather wait at home than at a long, brat-infested checkout); other needs, like toothpaste or toilet paper are more immediate and best bought on the way home. Multichannel promised to make all these different conveniences possible with the same retailer. Order online or off, pay online or off, pickup or deliver – anything you want. Wake up, its till a dream.

Retailers have trotted out multi-channel for years, every time anything riverine is mentioned. This, however, has been little more than lip service to prevent clicks felling the bricks. All the big boys launched e-commerce versions, but they were just clones of the original e-tailer (and at least in once case actually run by the original till wiser counsel prevailed). For a long while, the e-tail arm of a store remained a silo separate from the retail arm, meant magically to stem the tide, or somehow lure shoppers back into stores. Both, obviously, failed; ungrateful shoppers put their own convenience ahead of the retailers'.

The populace, meanwhile, figured their own multi-channel strategies out, combining online and off in ways that retailers do not necessarily benefit from. People often researching on Amazon to decide on the options, then go to a store to close the purchase. There's the opposite problem too – "showrooming", where people look at products in showrooms, only to go back home and look online for better deals.

Mobiles play a big role in this spontaneous multi-channeling. With smartphones, the Internet is no longer restricted to couches; you can scan barcodes, check prices and reviews online – all without leaving the comfort of your store. This does not always benefit e-tailers; prices may be higher online, or if the difference is small I'd rather check the reviews out and get instant gratification at the counter. Preventing it (Best Buy changes bar codes, Target stocks exclusives) may actually turn customers off – especially on electronics a certain amount of research is a must, and it seems a little silly to send me back to that dreaded couch when I'm already in your store.

Something old, something new

The problem with the old was, it wasn't really thought through - I want to say "holistically" here. The new wave of initiatives are smarter; instead of reacting to some unbricked adversary, the latest attempts use a store's virtual and physical presence to make the customer's overall experience better. Retail isn't about bricks or clicks, its about persuading customers to buy and making it convenient for them to act; that's what the focus has finally moved to.

Cash for online purchases makes online shopping available to unbanked or underbanked shoppers – of which there are plenty in the USA. Its a multi-channel experience, since you combine store visits with online purchases and home delivery. Its about making life easier for the unbanked, not somehow fooling them into walking into a store - indeed the counters are set up some distance away from the regular aisles. So far, the initiative (a first in the USA) seems to have succeeded though a definitive answer will have to wait. Interestingly, Wal-Mart tells us some 40% of the people coming to pay end up using a credit card or something other than cash – they're just not comfortable using it online for fear of identity theft. In India, the impetus for COD is somewhat similar.

Take the first round of in-store pickup initiatives. The initial impetus was again a desperate measure to persuade online shoppers off their couches and into the stores; customers didn't quite see the point. Finally someone thought through the experience and made store pickup a cheaper and faster way for shoppers to get their products – and it worked. More than 50% of Walmart.com's purchases are now picked up in-store, and in a large number of cases retailers offer same day pick-up (or even faster). Instead of doing something that was good for the retailer (get couch potatoes into stores) the program started doing something good for shopper (no waiting for delivery, no missed-you notes). Forget going into stores - car crazy USA even has drive-in pickups and kerbside delivery, in case you don't want to get out of your fully accessorized motor vehicle.

Circuit City, helped by Indian-origin product company Yantra, was one of the pioneers of this (it did prolong but could not prevent their eventual demise). Circuit City had in-store pickup, in-store inventory check (so that you could go to the correct store for what you wanted) and even same-day delivery (their famous 24/24 guarantee – pickup in 24 minutes or get $24 back). Best Buy followed, and then so did Container Store and many others. Barnes & Noble offers same day delivery in Manhattan. Tesco's uses their ubiquitous presence in England to offer same-day pickup. In all cases, the ability to allow shoppers to see which store carried the product they wanted is a crucial element, and one that requires some fancy tech dancing to enable.

Keeping it real

I think of these initiatives as real multichannel, because in-store pickups, cash at the counter and same day delivery all combine more than one channel to provide distinct benefits to shoppers, and are at the moment out of the reach of pure-play e-tailers (do, however, take a look at PayNearMe).

This kind of stuff isn't easy to do at all – fancy technology, brand new methods, processes, thinking all go in. I've argued earlier about separate supply chains for e-tail arms of retailers, and this isn't a change of tone. For those 24/24 guarantee deliveries Circuit City wasn't just picking things off the shelves like people imagine – they had a parallel setup – staff, inventory, pickup areas, even trucking; it was more like a separate e-tailer sharing the same premises. Tesco's, Wal-Mart, Sears, Best Buy, all have similar parallel operations.

How much does it threaten Amazon and its kin? I suspect, not much. Pure couch shopping will co-exist with store visits, each catering to a different set of customer convenience choices. The me-toos fell in between, and hence struggled to inglorious deaths, but the new wave will cement retailers positions, while leaving enough room for the pure potatoes to thrive. And, of course, there's the pure digital stuff – movies, music, ebooks, all of which have neither kerbside nor drive-through.

Real multichannel – coming soon to a store near you.

Opening Windows

2012-07-08T00:11:00.000+05:30

Walls between work and life have broken down. Companies have not noticed.

Work and leisure used to be quite distinct once upon a time (and that wasn't so long ago either). Work was carried out at designated hours in the workplace, with tools that the employer provided; home was spouses, kids and paid vacations. Even where you carried work home, it usually meant a temporary exile to the kitchen table. Once, only artists and rockstars lived without such boundaries.

This, of course, is long gone. Laptops and telecommuting started the blurring process a couple of decades ago, but things really went south with the advent of the smartphone. Uniforms gave way to business formals, yielded to business casuals before finally jumping off the roof entirely when the flip-flopped dotcommers took over. Work texts were shoved between bites of dinner, treadmills served as venues for conference calls, angry birds flirted with corporate emails and social networking finally nailed all those coffins firmly shut.

That's the new reality. Work and life have been smashed into tiny, tiny bits, then tossed all together into the bhelpuri of "worklife" (we dont even bother with the hyphen any longer). People have been taking work home for years – today's real change is the invasion of home into work. Family photos on the desk have been swamped in a flood of bermudas, bean bags and BYOD, and with this has come a new thinking – that of cooperation rather than control. Homes rarely run well purely on command - even Von Trapp was finally unable to stop the singing – and managing in those situations requires different approaches.

Managements, even those thoroughly progressive American ones, are still mostly conceptualised around command and control. Company policies are imperative "this is what you must do" statements – social cooperation plays no role in enforcement. They're enforced by coercive prevention, and suffer horribly when the control required to implement the coercion isn't present.

Security is possibly the first casualty of this lack of control. Out in the open, we're re-learning to secure by social means rather than authoritarian. What used to be simple take on new complications. Consider an employee who is about leave the company, and corporate data needs to be removed from his personal smartphone. The phone device is personal, company contacts mingled with private ones, communication a jumble of work tweets and kardasshian posts – what exactly is owned by the company and can be forcibly wiped?

Most people avoid stealing because they think it is wrong – not because everything is locked up (its not). In most societies this soft sanction works quite well, some locking up, some arresting but mostly cooperation and just the occasional theft. Companies, however, shudder to do security in this way – and it may not even be legally possible. Can a company trust employees to voluntarily wipe corporate data off their smartphones when they leave? Probably - its quite likely that the vast majority of employees will not betray this trust but can that prevent lawsuits and liability in those rare cases of misuse?

I don't know if the invasion of homestyle cooking is better than the earlier management model; it is, however, an inevitable side-effect of this blurring of boundaries. Feudalism used to do fine till democracy came along and changed a lot of the furniture - expect something similar to happen in workplaces.

The Economics of 'E'

2012-07-05T17:54:00.000+05:30

Mass market retailing is an expensive business. Rents, staff, inventory – the average brick and mortar retailer struggles along with barely visible net margins (spontaneous dancing is known to happen at 5%). With thousands of stores, hundreds of warehouses and over two million employees, Wal-Mart has in the last five years managed a profit margin of just 3.5%. The story is no different for any other major brick & mortar retailer, American or desi. Cool-kid-on-block Internet retail, on the other hand, thumbs a nose at the old-fashioned ways and gives the distinct impression that it can do much better.

There's just one small problem.

The bellweather Amazon, for all its buzz, seems unfortunately to have done much the same (indeed, a little less at 2.48% over the same period); nor has any other sizeable virtual retailer done much different. What gives?

The law of unintended consequences, that's what.

Lets take two of the most discussed items – rent and inventory. Mind you, this post isn't to discuss the benefits of not having inventory or rent– there are many of those. This is to point out, to all ye budding armchair Amazonians, that every cost you reduce often has side effects that are – well – unintended.

Location, Location, Location

A store has to be in the right place; unfortunately landlords usually know this too. High rents per square foot are a cost for sure, but they are usually strongly correlated to more customers, higher footfalls and better sales. Reduce rent costs too far, and you could be diddling your thumbs in the more unvisited bits of a desert. Location plays many roles – footfalls, segmentation, branding – and a rent is an easy proxy for all of that. A railway station, an upmarket mall, an airport – decide where you want to be and a whole package of segmented customers with known spend patterns will land up at your doorstep.

The Internet has no rent; it also has no location. Any part of the Internet is pretty much as good as any other; the person watching your latest Victoria Secret collection may come from Sweden or Somalia. This boundary-less nature has plenty of advantages, but having no catchment to rely on is also a serious drawback for an e-retailer. Most would pay rent if they could; location a more reliable way of acquiring customers than the stuff online guys are forced into – mass market advertising, search keywords, link tracking and such other esoteric nonsense. What you save on rent you lose (and more) on customer acquisition and segmentation.

Products Products Products

Then there's inventory. Physical retail has to go through the tedious process of loading up the shelves with all manner of products. This costs money, and can be even more troublesome to the retailer than rent; stocks have a nasty habit of piling up, going bad, getting stolen, being wasted or simply being of the wrong kind. Words like markdown and clearance nip nastily at the heels of store managers.

Like rent, however, inventory is more than just a cost. It turns out that dangling an actual apple under the consumers nose works better than a picture of the same apple. Modern retail works by tempting users into buying more and higher; if people only bought the minimum things would be quite bleak for all concerned. E-retailers have to spend to overcome this, with better product detail or droolworthy photography – and sometimes that costs more than the product itself. Online, apparel sells best when photographed on real models but as you can imagine, that is not inexpensive. Then, there's the cost of poetically describing the stuff, getting its sizing charts right, putting on your website all the stuff that customers in stores would read off the label for free.

Brick & mortar stores have learned over the years to use inventory as bait; the extra cost of abundant stocks being made up for by the increased conversion and larger spends. Its not unusual for a supermarket to have conversion ratios (people walking in to people actually buying) in the upwards of sixty percent or even the nineties. Many e-retailers, on the other hand, would praise the heavens if they saw 5%. Worse, Low conversion rates exaggerate those aforementioned customer acquisition costs.

The Bottom (traditionally a line)

This scenario happens far more often than you would think for a business model that is finally in danger of becoming older than its CEOs (Amazon, after all, is nearing twenty). Blinded by the obvious benefits of not paying rent or stocking stores, e-retailers sometimes forget to account for the much higher spends on related costs – acquisition, conversion, product information. E-retail cost structures are different from brick and mortar, but not much more generous.

Learn to live with it, and look carefully before funding anything that tells you otherwise.

A Couple of Funerals

2012-06-16T14:45:00.000+05:30

The last rites of two promising startups made news a few days ago, and not just because they happened together. Flipkart shuttered Letsbuy on the same day that Snapdeal gave up on Esportsbuy, both had been acquired barely months ago, so it does seem that the acquisitions were just sugar coating on the euthanasia pills.

This by itself is not particularly troubling. Natural processes of evolution dictate that many will launch and many of those will die. 2010 saw a spate of launches, one should not be surprised that many are running out of steam. Indeed, in any new business model it is quite likely that no more than a couple of firms will survive the initial spawning rush and become adults. Not all slaughter, however, presages the survival of the fittest - sometimes it is just an indicator of illness.

Many people ask me why Flipkart is doing so well while others struggle. I think the trouble with e-commerce in India - and every pundit seems to know this - comes from two fundamental issues.

One, the model is financially unviable - operational profits are nowhere in sight. Indian e-commerce has dug itself into a hole of deep discounts, cash on delivery and free shipping, and no site has a viable model other than last man standing. COD in particular erases most of the financial benefits of the virtual model. And I find automatic free shipping on everything mystifying - TV shopping, a much much larger business in India, happily charges very fat shipping fees (occasionally more even than the cost of the product) yet continues to prosper. The final nail in the coffin? The high cost of customer acquisition has not come down - loyalty is non-existent so the only way to drive sales is through high volume advertising.

And all this boils down to a key factor – a weak value proposition. The businesses seem more enamoured with the 'e' part of the model than on what differentiated value being provided to the customers. Why do I say that? TV shopping manages to charge generous shipping and handling because people feel that even after that it is worth the cost. Look at other business online – Bookmyshow and Indian Railways, both confident in their value proposition, happily charge "convenience" fees. Amazon too has a decent-sized business in India, maintaining loyalty without ad-spends, all the while charging fat amounts for international shipping. If a business is unable to persuade customers to pay for its services enough to turn a profit, I suspect the business does not have a convincing customer proposition in the first place.

Second, customer service – by and large – sucks. Even tiny startups have service people who talk like big business ("my shipping department will get back to you" or "sorry company policy"). Imagine my local grocer using language like that (and this is Bandra, his turnover probably exceeds some of these upstarts). Companies that feel themselves above customers even at startup stage are not my cup of tea.

What would I look for in an e-commerce business? Here are my three bullets, after three start-ups and plenty of bite marks to show for them.

Conviction about the 'E' part. The online bit must matter to customer, not just to valuation-seeking investors and glory-seeking managers. We started that way in Futurebazaar, but soon it was more about turnover than value proposition.
A tight, committed team. The team must be committed to the business rather than to the potential to buy a Porsche. I do believe it is better to have a committed idiot that a competent cynic, but it turns out to be surprisingly hard to hold the ground on this particular belief.
Willingness to serve the customer. If the CEO is doing strategy while some junior employee 'handles' customer service, something is wrong in the essence of a startup. Again, something we did well in the early days of FB but got badly distracted from in later years.

That's it. No bullet point about revenue model or profitability. Nothing about business plans. No market share talk or first mover advantage. And especially, no valuation conversations. Those things are important, but they are the details, not the core. Get the core right, and you will usually be rewarded handsomely anyway. I want to participate not in a startup that wants to make money but in one that wants to change the world - or at the very least, the world around them.

Looking Ahead

2012-04-26T00:33:00.000+05:30

I just sat through a presentation about the four big trends in IT that even the polite struggle to describe as dull, but it did get me thinking. Incumbent technology vendors (pretty much like incumbents everywhere) have too much invested in the present and will hence sell incremental as disruptive. IBM once believed sincerely in the future of mainframes, big studios still insist on the glorious road ahead for DVDs now that the laser is blue, predicting the future is always fraught with problems.

Armchair sniping is all well and good, but shouldn't I be in a position to predict better? Time to get my somewhat ample behind off the couch then, and stick my neck into the future. Here, I go - my four predictions for IT trends, at least as far as careful, conservative corporates are concerned.

Cloud will vaporise IT

Today's businesses are all concerned about the technology underlying the cloud, and how to adopt it. However, cloud isn't really a technology - its a method of delivery of computing power. Cheap, reliable, ubiquitous - thats the key. Unit costs must be very low, it should work without issue all the time, and it must be everywhere. These three words are key to the revolution. It happened in electric power, in telephone, in data connections, its now happening to computing power and storage.

A large part of the IT's attention today is on building and tuning infrastructure - server farms, storage arrays, firewalls and networks. This is going to vaporise. Think of what will go away - buying stuff like servers, storage, backup, devices, administering the stuff you bought, managing security, disaster recovery - something like 75% of the typical IT department's activities. We used to have fourteen email servers, a storage network, a spam server, antivirus, firewall, backups, local mail routing servers, three administrators, backup schedules, recovery plans - all vaporized when we moved to cloud email at a single fee per mailbox.

Startups are already tapping this on a large scale. Most own no servers - a credit card and a few taps of the keyboard will get you services like Heroku for computing, Amazon cloud for storage, Google for email, Salesforce for CRM, Zendesk for helpdesk and public internet connections. Each will scale and perform far beyond most corporate IT systems. These and similar services; you buy them like furniture - designed, assembled and ready to use, sturdy and reliable, to be arranged as needed. Much of a company's IT is a support service that is neither strategic nor related to the firm's core competence; ripe for the vanishing act.

Your clueless desktop support person and underpaid genius of a CIO will remain, though.

The Berlin Wall will fall

There was a time not so long ago when work and home life were clearly divided. You may have spent long hours at work, even brought work home sometimes, but it was nevertheless clearly separate from the activities of home. That line between work and home is no longer visible. People work in thin slices, between bites of dinner or commercial breaks on TV, while crossing the road or running a marathon. Work and play are not distinct buckets, they're slices of attention in a continuum. The devices have also merged - facebook and work email, spreadsheets and angry birds, videos and Word documents all coexist on the same boat.

This is not a prediction - this has already happened. The consequence is the prediction.

As long as work and home were separate buckets, companies had the luxury of control. The work bucket was the company's to control as seen fit. You could specify attendance, hours, measure productivity, analyse behavior and the CIO could bully you into using a boring black brick of a laptop if that's what the standard was. Today,CIOs are pulling their bald heads hairless over BYOD concerns but its not just a device issue. Users are increasingly impatient of boundaries between services erected only to make corporate IT's job easier. People are interacting with business associates everywhere - not just on channels approved by IT. Put all the controls you like - the conversation will merely move entirely, maybe onto the Facebook or Yahoo Chat. Corporate IT, long the dictator in a private republic, is being forced to change this mindset and adapt to a lack of walls. Some have already done it, many many more are reluctantly following.

Big is going to get bigger

Data will grow explosively – that's hardly a prediction worth making. I do, however, think that the nature of data that corporates store, and how they store it, will change dramatically.

The data that companies store has been growing over the years (I've seen it roughly double every year in the last decade) but I predict we are in for much faster growth in the next decade, maybe as much as tenfold every year. However, both what we store and how we store it is in for a big shakeup.

First the question of what we store. The focus used to be transactional and analytical data, and this grew as prices became cheaper because many of the compromises made earlier to pack things tight were no longer required. However, there is a limit to how much of this kind of data a company generates; after a while this grows only as fast as company operations do. What really causes the explosion is the new predilection to store historical data – to never purge anything. GMail started by encouraging us never to delete email, and we are now in the digital hoarding generation. Every file, every database transaction, every log - companies now want to store everything for ever.

Storage will thus have to go in the opposite direction – slower and simpler rather than faster and more feature-rich. Storage management will disappear – reliability, versioning and search performance already come built into the newer storage clouds. Assets will come with unlimited undo capability – every change can be undone. Thin provisioning and de-duplication married to tumbling costs per terabyte will make (and is already making) all this magic possible. There will still be some super-performance disk arrays for some applications but distributed storage, search and access is the way things will go.

Tapering off

The era of tapes is over.

Backup was a big part of corporate strategy. Tapes, their use, their movement, their rotation, people put lots of thought into it all. For three decades, corporate IT was all taped up; from those massive old noisy spools to the modern cigarette-pack versions.

Tapes survived on three key advantages – cost per byte, huge storage and portability. Storing data on tapes was really cheap, and they could be moved away from the original data. Optical was somewhat cheaper per byte and more portable, but the maximum capacity was one hundredth of a tape and thus never became practical for large storage arrays. However hard disk costs started dropping, capacities increasing a few years ago – and only portability was left.

What has really started putting the nail in the tape coffin is the requirement to store things for years. Tape is good as a backup for weeks or months – decades bring all kinds of other challenges – the tape may deteriorate, one piece of a set may get lost or damaged, the label may become unreadable and worst of all – you may not find a compatible drive to use for restoration a decade later.

With the fading of tapes will face the concept of backing up. Services like Facebook or Dropbox already automatic create multiple internal replicas across different storage pools. Storage providers provide automatic versioning and undelete protection – you undo any change and go back to any previous version; this will eventually be the norm. Storage reliability will become service provider responsibility – backups are headed the way of paper printouts.

Last words

So there are my four attempts at sticking my neck out. What does everyone else think?