Programming fun at startup

Preparing for an interview after being employed 11 years at a startup

2021-03-21T17:36:00.001-05:00

I would say I didn't prepared a hell lot but I did 2 hours in night every day and every weekend around 8 hours for 2-3 months.

I did 20-30 leetcode medium problems from this list https://leetcode.com/explore/interview/card/top-interview-questions-medium/.
I watched the first 12 videos of Lecture Videos | Introduction to Algorithms | Electrical Engineering and Computer Science | MIT OpenCourseWare
I did this course https://www.educative.io/courses/grokking-the-system-design-interview
I researched on topics from https://www.educative.io/courses/java-multithreading-for-senior-engineering-interviews and leetcode had around 10 multithreading questions so I did those
I watched some 10-20 videos from this channel https://www.youtube.com/channel/UCn1XnDWhsLS5URXTi5wtFTA

Better Tools Happy Engineers

2017-04-29T13:12:00.003-05:00

"The only thing constant in a Startup is Change"

If you aren't changing fast enough then order and complacency sets in which leads to mediocrity and you can soon become obsolete. We do biweekly releases and want to move to weekly and then daily releases. You can increase the release cadence, if you are able to automate most of the testing. But automation can't detect everything and things may break and the intent is to establish a culture of how can you prevent it but if you cant prevent it then how fast can diagnose and recover from it.

For a long time I had been pushing hard for a working centralized logging and after almost an year of trying to scale the ELK framework finally our team has been able to put a fast centralized logging system that's ingesting Terabytes of log per minute from thousands of nodes in production. This weekend we deployed a major architecture change by creating swimlanes in our cloud by directing different types of traffic to different swimlane and we also added more capacity to our Memcached servers. I was training my lablab bean plants on this Saturday morning when I received a call from India Operations team and I was like this doesn't look good. The operations engineer told me there were 4-5 support tickets opened in last 4 hours and they were all related to the datacenter zones where we added the swimlanes. I am like this doesn't look good. Luckily in one of the escalation ticket I found a distributed tracing identifier called RequestId and I quickly did a Kibana search and found that the request ran into 503 due to no server found at HAProxy. I took the String "NOSRV" and fired a Kibana query for last 12 hours and found a pattern that it was happening during the deployment and isn't happening anymore for last 4 hours. This relaxed me and I can then think critically as to what might have happened.

I did some more queries and root caused it and send it to deployment team and also confirmed that for the customers things are working fine. We would do a postmortem to takes steps so it doesn't happen again. Thanks to a working centralized logging solution I was able to ROOT cause it end to end within 20 minutes and I was back to my gardening.

Previous weekend was also similar things happened where I had to take my son to a birthday party and a customer was complaining that his account is being locked out due to Invalid Auth attempts for 3 weeks and no one is able to chase which device it is and he escalated to top management. The customer ticket was in Ashville data centre and everyone was checking there but in one of the email exchange customer told me his Active Directory account is getting lot of invalid attempts from San jose data centre. I was baffled if its a security issue but I was like what the heck I can just query his email "xxx@xxx.xom" in the San Jose centralized kibana and indeed he was right. He had one more account in San Jose data centre where an old device from one of his office was making the calls. I sent customer initial details and I took my son to birthday party and came back and root caused the issue within 1 hour all due to centralized logging and I was back to enjoy working on things I love.

I am lazy by nature and I hate doing mundane tasks like grepping logs, for a long time I was pushing team to get centralized logging working. The problems with grepping logs were:-

New machines were added but the log mounts weren't there so I had to chase people to mount them in middle of debugging session which can sometime takes hours on a weekend.
Mounts would become bad.
Grepping logs for simple things take hours if done over a day worth of logs over 200 machines.
You need to understand the entire architecture of all 20-30 services and 3-4 layers of load balancers to grep in a localized area to save time.
"The only thing constant in a Startup is change" and architecture changes every month so you need to keep up to speed in all of them.
Support or customer success team doesn't know architecture and then ticket is given to engineers who isn't happy to chase issues sometimes only to explain customer that its a feature not a bug.
Product management team also doesn't know architecture enough to grep things.
With Grepping you cant see trends like above unless you are techy enough in linux to do graph on command lines.

Centralized logging solves all of this and best of all you can send a link to the issue or even put it in ticketing system.

Providing better tools and pushing hard for them should be the job of Engineering leads as the technology landscape keeps changing rapidly and whoever can move fast and recover from failures quickly will survive. Also better tools create Happy engineers as they can do mundane things quickly and move back to doing what they love (in my case programming, gardening and spending time with family).

Learning from mistakes

2017-01-31T16:01:00.001-06:00

We should not repeat mistakes and if its the end user that is committing same mistake again and again then the UX should be changed to make it difficult for user to perform such actions. Make the user apply some cognitive thinking before he can do a detrimental action and make the action obscure.

For e.g. we are a filesystem company and "Trash delete" is a detrimental action, over past few years I had several tickets where user accidentally clicked Empty trash and deleted all his files or he clicked Delete Permanently instead of Restore. He was able to do all this because in the UX the actions are close to each other

There are users who did this and immediately created tickets under panic and some even blaming that system did delete instead of restore. I know in UI it may look bad but I have seen several companies solving this by making user think twice before invoking detrimental action like asking a captcha or make the delete and restore flow different like send him an email with confirmation like before he can do Empty trash.

An easier thing would be to just move the actions far apart and put some filler actions between them and change color of action buttons, like make it dumb for user to know which he is clicking.

History remembers finishers

2016-12-18T10:56:00.003-06:00

This week I saw the movie Dangal trailer and the actor says "If you win silver medal, today or tomorrow people will forget you but if you win Gold medal you will become a source of inspiration". Also today morning I saw K L Rahul got out on 199 runs and the commentators are complaining about his loss of focus on one ball and he got out http://timesofindia.indiatimes.com/sports/cricket/news/india-v-england-5th-test-day-3-rahuls-199-drives-india-to-commanding-position/articleshow/56045818.cms

This reminds me of Startups and Finishers, give up too close to the finish line and you wont be remembered but if you finish it you may become an inspiration.

Avoiding Burnout

2016-12-16T19:11:00.001-06:00

Burnout is very common in startup and it hits you more often when you are at the same startup for 7+ years. Much of burnout is felt when you are working hard but progress is slow and you have high expectations of yourself and the team. I read this excellent article http://andrewdumont.me/avoiding-burnout/ and it felt almost Deja vu to me. I work from home so it hits even hard to me as I have less people to share it with. Here are the ways I have been trying to curtail burnout and hope it help others too:

Workplace:Keep your workplace organized and stick to a routine. I have an office room at home and I am in work mode when I am in that office.
Morning:Keep 1 hour in mornings on something outside work. I read a lot in morning and don't come online or check emails until that 1 hour is done.
Workout: rain, heat or winter, I try to walk for 30 min in afternoon as much as possible. This gives you some time for meta thinking and de stresses you from the morning chaos. Many solutions to tough problems come on this walk. I don’t hear music or podcast on this, I tried but then it felt like more work, I avoid it.
Silence room:Do some grunt work on weekends, be it doing vaccum in house or mowing your lawn, you need 1-2 hours of "Your own time" withot distractions.
Email: fight this monster, I sent most notifications to trash and assume people will call me if they need me. Also when startup becomes big people will copy you on tons of emails so choose which battles are worth fighting and which aren’t.
Hipchat: Avoid this also if you are on creator schedule, I am now a days trying to stick to checking it only if I am waiting for build to finish or have some time before meetings.
Motivation: Set some tough projects for you and work on them. Even if you are making slow progress you should finish it by making some progress every week.
Coding: I like coding so now I have reserved calendar time of 3 hours a day to it. This is by far the best way for avoiding burnout for me.
Buddy: Have a buddy who you can share some things, may be you will just vent out and he will also but it would de stress you.
Vacations: I try to take 1 big vacation(2weeks) every year and try to go to national parks for camping where there is no connectivity so you are really unplugged.
HQ visit: this recharges me as I again get some more brainstorming on things outside my purview.
Hobby: I recently picked up photography and I do read and practice that.
Humans: I feel more burnout when I need to make something live and there are many humans in the process chain that makes things slow, things that should take 1 week takes months and then you are burned out as your baby isn’t live. I try to automate and avoid humans in the process as much as possible. Its unavoidable but if you are lazy and think hard then its doable in many cases.

Finishing what you started

2016-09-15T17:58:00.003-05:00

A quality of a good Startup engineer is "Finishing what you started". I value finishers more than people who talk good theory and start many things but don’t finish most of them. Most of the time people get excited and start many things concurrently but leave them in limbo or they finish 95% of the thing. Over 7 years at my current Startup I have observed that in many projects doing first 95% may take you X units of time but its the remaining 5% that demands 10x cognitive effort and time from you. Its this last 5% where people either burnout or give up. Its the finishers who keep the compass pointing to North star, never give up and do what ever it takes to Finish( in indian mythology saam, daam, dand, bhed).

Recently One of my respected colleague "Sachin" left for a sabbatical. He was one of the Finishers whom I would hire any day on my team or would choose as a buddy to watch your back. "Sachin" has a zeal like a Rhino or Triceratops to Finish what he started and for this I value him the most. Best of luck to him and I hope he gets bored with sabbatical and joins back :).

What a rocky start to labor day weekend

2016-09-03T17:34:00.002-05:00

Woke up by earthquake at 7:00 AM in morning and then couldn't get to sleep. I took a bath, made my tea and started checking emails and saw that after last night deployment three storage node out of 100s of nodes were running into Full GC. What was special about the 3 nodes was that each one was in a different Data centre but it was named same app02. This got me curious I asked the node to be taken out of rotation and take a heap dump. Yesterday night a new release has happened and I had upgraded spymemcached library version as new relic now natively supports instrumentation on it so it was a suspect. And the hunch was a bullseye, the heap dump clearly showed it taking 1.3G and full GCs were taking 6 sec but not claiming anything.

I have a quartz job in each jvm that takes a thread dump every 5 minutes and saves last 300 of them, checking few of them quickly showed a common thread among all 3 data centres. It seems there was a long running job that was trying to replicate pending storage objects as per replication policy and it had 50M or so backlog to catchup. It seems newrelic was holding on to all that data for the thread and not just collecting the aggregate stats, the job was running for hours causing memory to increase with each replicated file.

Now I had two options:-

Rollback the library
Suppress newrelic instrumentation

I thought of going option#2 as it wont require any new build, just run puppet and restart all nodes. Boy I was wrong , newrelic has negligible documentation for the yml file. I filed a support ticket as they have no phone support even for emergencies even though we pay thousands of dollars a month. After IPO their support team quality has also reduced, the support told me hey go and disable the class, i am like its natively supported by you so give me some hints how to do it. She told me that a OOM ticket is filed 2 versions ago so I can wait for it to be fixed, wtf. I waited 20 min for her to tell me how to disable it and no reply, It seems even she didnt knew how to do it so she was buying time letting me hanging dry on a cloth line. I started mining the support forum and google and I tried 5-10 settings with our devops engineer and newrelic agent is stupid that it would take any wrong setting too and show it in the UI. I ended up with a settings nightmare

Finally I tried one setting which was weird but worked, I had updated library to spymemcached-2.12.1 but the setting to turn it off in newrelic was

class_transformer:

com.newrelic.instrumentation.spymemcached-2.12.0:
enabled: false

notice the setting is called 2.12.0 even though my library is 2.12.1, wtf. It was late in night for the devops engineer and I was almost 5 hours on the desk, we just patched those 3 special nodes and called it a day. The other nodes would be patched by puppet before Monday using automation.

Today I also complete 7 years at the company :).

Hanging on to old things

2016-09-01T17:30:00.001-05:00

I have seen my Kid grow from 0 to 6 year and one startup growing from 20 to 150 employees and another growing from 5 to 300+ employees. A common trend I seen is hanging on to old things. Not sure if its due to sentimental value attached to things or inability to make hard choices. My Kid still has his toys that he used to play when he was 1 year old. He wouldn’t let us throw them or give it to someone, I am recently working with him to let go of old things. Problem with old things is that they are a drag, they just sit around the house and are a clutter. Its the same with old books, I have many of them and one day I need to go and clean them once I read a book, I rarely read them again so why not donate them to library.

A common observation with Startups I have worked is feature bloat due to holding on to old features. Features that are not maintained but are in use by a handful of customers causes drag. Recently I wanted to refactor folder listing code to not load additional versions data as the new UI doesn’t require it. But as I started chasing reference graph I found 5-10 places in old versions of UI using it with some complex intertwined data structures and what seemed like 1 hour worth of work seemed like 1 week of work. I gave up refactoring it due to other important things on plate. Ultimately biggest loser in this is the new UI.

Sometimes we have to make tough decisions and let go of older things else it creates a baggage not only to us but others too.

Little things and frustrations

2016-06-21T20:06:00.001-05:00

Small shitty things frustrates me, like someone came up with an idea to put epic on each Jira task. Well its suited for big Tasks that has lot of subtasks but 50% + of my cases don’t require an Epic.

I am lazy and dont like to waste my time. I am creating an issue and filled all things and now this process requires me an Epic, total BS. Yes this is total BS so either the process requires fixing where epic is first required field so I wont invest filling so many fields or this needs to be removed/made optional.

Data motivates you to do more

2015-07-15T17:16:00.003-05:00

I always believed in "Trusting data" over human gut but lately I have been observing a simple fact that being data driven has a side effect, "it motivates you and keeps you on track". Some recent examples are:

Fitbit: Last year I started the afternoon walk because by 2:00PM after doing calls and replying to a lot of emails the brain would be fried and I wont have energy left to code or think. Doing these 30 min walk daily recharges the brain. I had an Omron pedometer sitting around for almost an year and I seldom took it with me on walks. The problem with it was that it used to store last 30 days data of my steps and other things but it didn’t had a good way to graphically see the data. When it comes to data "less is more" but also one more important aspect is that you need to present your data in graphs so it doesn’t take a huge amount of cognitive effort to make sense of it. Recently my employer gave a fitbit to everyone who participated in summer challenge and it isn’t a better pedometer than my old Omron but immediately I saw that it can sync data stored on pedometer via bluetooth to my fitbit app and after a week I see this. Immediately in 1 sec I can see that I am lagging this week and need to catch up.

Its another thing that I need to remember to carry this dongle with me, I always carry my cell with me on walk and I saw that Iphone6 has a pedometer built into it so that would eliminate the need for this when I upgrade to Iphone6.

Large scale migration to Elasticsearch: We recently migrated billions of files to Elasticsearch and the migration took months but data kept us on toes and telling us if the compass is pointing towards north or not. We built various dashboard to monitor migration rates and as migration was running day and night I would start my day with checking how many more files we migrated and whether we need to add more servers, jvms, memory or CPU to meet the goal. Here is a graph in one data center.

Data kept us on track and we were able to spot many issues before they were able to create a disaster.

Exception Analysis: We do exceptions and 5xx status analysis on incoming requests daily and in 2 week sprint we strive to fix as many as we can, but after the release either new issues pops up or customers use the flow in a different way causing some components to buckle under pressure but one thing that has kept us on toes is data. By looking at the one screen report we can tell how did this data center do yesterday and which areas require immediate attention vs areas that require attention in 1-2 days. This leads to lesser no of customer escalations as we are able to spot many issues before them. Little little things adds up and having this report daily strives us to optimize more and we can spend quality time on doing things we love which is writing code for scalable systems.

In short data points you that there is a problem and once a bug has been implanted in your brain that a problem exists you would try to fix it so you can get back to normal routine.

Have I started hating mysql and falling in love with distributed databases

2015-05-06T16:54:00.001-05:00

It seems Mysql is rock solid if you want:

Transactions
ACID support

So I would still recommend mysql for any thing that is mission critical data and it should be the primary datastore for your transactions. But what about derived data, immutable data or analytical data?

In past I have built large scale cluster of mysql server storing metadata about billions of files and folders used by tens of thousands of customers daily and its scaling fine and working good, its still growing at a healthy rate and holding up. But this requires a lot of baby sitting if you have 100s of nodes and you need to do

replication
add more nodes
rebalancing data
monitoring entire cluster
Sharding
Backup/restore

You have to write a lot of tooling and lot of monitoring/babysitting to scale the cluster. Plain stock Mysql will scale up to a limit but vertically scaling has its own issues. So +1 for Mysql but not everything should be stuffed there.

Recently me and my team built full text search/indexing on same dataset using elasticsearch and so far it hasn’t disappointed me. With just 1 engineer and 1 devops guy we are able to build a cluster per datacenter to store same data. The thing I liked most about Elasticsearch was half way through migration we started facing performance issues and we just added more nodes and the cluster rebalanced itself. Also Elasticsearch has tools like kopf/HQ where I can monitor all nodes in one place. For e.g. this is one of the smallest cluster that we just started migrating and as it grows if we see high load averages then we can add more data or client nodes.

I don’t need an lot of DBAs to manage the cluster as elasticsearch has built in support for

replication
adding more nodes
rebalancing data
monitoring entire cluster
Sharding

I had earlier built an event store to store events on top of mysql but I was storing only few months of events into it. Now I have to build a store that can store events for 7 years and I dont want to use Mysql for it as I dont want to baby sit it. Our events data is way huge than the metadata. Because every change to a file generates an event and over 7 years this could be a huge no of records. I dont want to manage an army of mysql servers so researching for some database that has good querying support and can store long lived data with eventual consistency and rock solid durability. You could store them using kafka or scribe but they lack good querying support. I am still not sure if Elasticsearch is a good store for this because I don’t need search, I need querying based on some paths and time based querying. Also most querying will be on recent data and past data would be queried rarely. Logstash uses Elasticsearch in same fashion so I need to find something else and then compare with Elasticsearch.

Also I see a trend emerging that in today's world DBAs need to get out of cocoon and know more than just relational databases. There are a lot of new tools like OpenTSDB, HBase, Mongo, Cassandra, ElasticSearch, BigQuery that are now getting used to store BigData so they need to catch up and catch up fast. Google recently released BigTable http://venturebeat.com/2015/05/06/google-introduces-cloud-bigtable-managed-nosql-database-to-process-data-at-scale/ need to check it out. I had checked BigQuery but writes are cheap, its the reads that are costly when using things like BigQuery.

Rate limiting APIs and Java services when operating at Scale to solve Thundering herd problem

2015-05-06T16:00:00.004-05:00

When you are operating at scale and handling peak traffic of 1K+ request per sec on a jvm then no matter what you do you would get hit by a Thundering herd problem. There would be operations that happens once in a while but take more than 10 sec and if there are too many of them happening then you could choke backend services or worse cause a downtime. So you need to Rate limit these long running operations that only X can run at a time, this way you are leaving room for running lots of short lived transactions.

When you have millions of users then not all users are doing these long running operations and not all traffic is coming from online users. We are a cloud storage company and we give sync client to users so 80%+ traffic at a given time is coming from these clients that are trying to sync changes between cloud and local system behind the scenes. Our application is written using REST apis and these clients are using the same REST apis that our web ui is using. Also some customers may have 10K users and some may have 10. So it may happen that a big customer may always starve the small customers. So you need two things:

Fair distribution of use of REST apis
Separate priority for online vs bot traffic.
Rate limit or protect a tier if a Thundering herd occurs

Most of these sync clients can exponential backoff, they handle 503 response and retry after some time with exponential delay with an upper threshold.

For fair distribution of use among customers I was using a pool of thread pools with hashing based on customerId and a salt, the technique is described here http://neopatel.blogspot.com/2013/06/java-fair-share-threadpool.html.

for Rate limiting I was using ThreadPools. When requests would come I would create a callable for the action to be executed and submit it to threadpool and wait for the result. The Thread pool had a fixed processing capacity and a fixed queue length. After the queue is full we would send 503s.

But thread pools have many disadvantages:

You already have a http thread and now that is idle as you delegated to a threadpool to do the job.
New relic somehow goes nuts when you delegate to a threadpool and doesnt record any trace info, AppDynamics is smart and it recognizes it but I am not a big fan of AppDynamics.
Logging context gets messed up and you now have to propagate it to the new thread.
Exception handling is messed up as it would get wrapped in ExecutorService exceptions
Under high load we ran into an issue where in tomcat if you wrote to response from 2 threads and sync clients abort a connection then it hangs the thread causing all threads to be gobbled up in a course of 6-12 hours.

I was looking for a RateLimiter that would give me a gate with a finite opening and a finite queue length, kinda like restaurant where you have a finite no of tables and a finite queue before they start accepting more guests. I didnt found anything so I cooked up a one in kitchen.

The code is going live this weekend, we did perf test and the results looks promising. To be conservative I had to add a switch so that in case of issues you can revert to old way with the flip of a flag in config at runtime.

public class RateLimiter {
    @Getter(AccessLevel.PACKAGE)
    private DiagnosticSemaphore allQueue;
    @Getter(AccessLevel.PACKAGE)
    private DiagnosticSemaphore processingQueue;
    private volatile boolean shutdown;

    public RateLimiter(int numProcessing, int numWaiting) {
        this.allQueue = new DiagnosticSemaphore(numWaiting + numProcessing, true);
        this.processingQueue = new DiagnosticSemaphore(numProcessing, true);
    }

    public T executeWithRateLimit(Callable callable) throws Exception {
        handleShutdown();
        boolean allQueueAdded = false;
        boolean processingQueueAdded = false;
        try {
            allQueueAdded = allQueue.tryAcquire();
            if (!allQueueAdded) {
                throw new RejectedExecutionException();
            }
            try {
                processingQueue.acquire();
                processingQueueAdded = true;
                handleShutdown();
                return callable.call();
            } catch (InterruptedException e) {
                throw new ApplicationRuntimeException(e);
            } finally {
                if (processingQueueAdded) {
                    processingQueue.release();
                }
            }
        } finally {
            if (allQueueAdded) {
                allQueue.release();
            }
        }
    }

    private void handleShutdown() {
        if (shutdown) {
            throw new RejectedExecutionException("Not accepting requests as shutting down");
        }
    }

    public void shutdownNow() {
        shutdown = true;
        for (Thread t : allQueue.getQueuedThreads()) {
            t.interrupt();
        }
    }

    static final class DiagnosticSemaphore extends Semaphore {
        private static final long serialVersionUID = 1L;

        public DiagnosticSemaphore(int permits, boolean fair) {
            super(permits, fair);
        }

        @Override
        public Collection getQueuedThreads() {
            return super.getQueuedThreads();
        }
    }

Cognitive overhead of email

2015-04-10T16:57:00.003-05:00

Looks like these days bitching about email is my favourite topic. What can I do no matter what I do I just cant keep up with email flow. The new kind of emails just keep increasing, I was doing a good job 2 years ago and close to inbox zero daily and was writing a lot of code but two things have increased the email flow to me recently:

Internal reports about production montioring from Newrelic, haproxy, exception analysis by team.
Review requests. This is a recent one. Most review requests are sent to Java review group and somehow I atleast need to spend 1-2 min on every email to first decide whether this is for me or not. Even if there are 30 review request per day it consumes 30 min of my time just to make sense of it. Daily when I see 100 emails to be answered I feel disheartened that I cant write any code today.

Both above items are necessary evil, #1 has increased production stability so my weekends are less busy. #2 I am hoping eventually will increase code quality and lead to more ownership and processes that would lead to a self correcting system requiring less involvement from leads.

But I need to do a better job or come up with some other way to do this. Its almost 5:00 and I had two 1 hour meetings, 1 Elastic search node down that consumed 1.5 hours and 2 team calls 30 min each. And I cleared some 60 emails. The only code I wrote is removing some junk code as it was causing some exception when a penetration tester was testing that flow.

Feeling frustrated as need to do something about this. This Slack killing email sounds interesting.

New relic and statuspage.io integration

2015-02-28T13:39:00.000-06:00

Monitoring tools exposes a lot of data and we use Nagios, cacti, graphite,newrelic, mixpanel, flurry, boundary and many more tools. But one of the ask for Support and marketing teams is how can they internally know if something is wrong. We cant expect them to wade through so many systems and so many applications to make sense of what is operational and what is not. For e.g. we use a lot of services to serve the cloud filer server solution and this is the first page of status of our services in new relic and it spans 2 pages.

Support team and management relies on Operations team to notify them if an issue is on going. To make this easy I did a Proof of concept integration application responsible for serving main website with Statuspage.io. The idea is simple

Create public metrics in statuspage.io that are human readable.
Query new relic and various systems for application status.
Map new relic green/red/yellow status and other system status to statuspage.io status.
If there is a status change then update statuspage.io metric status.
Statuspage.io lets user subscribe to status change via SMS or email so support can signup for that.
Once this is fully baked then we can make this public and let customers do the same.

Now this is a proof of concept so I wrote a python script only with new relic integration with one application. I setup a cron job that runs every 2 min to do this integration. But we can enhance this script to derive status from various tools like new relic/nagios/boundary and map it transitively to human readable status. We can then even load this page up in various Tv screens in all offices to improve visibility.

The script I used to do the integration was for proof of concept and I cooked up in 2-3 hours including api research so use with caution.

import os
import sys
import requests
import logging
from logging.handlers import RotatingFileHandler

spiPageId = "PutYourPageIdHere"
cloudComponents = ["Metadata APIs","Web Interface","Webdav","Sharing"]
nrToSpiStatusMap = {"green":"operational", "yellow":"degraded_performance", "red":"major_outage"}

def getNewRelicApplications(apiKey):
    interestedApps = ["cloud","mobile"]
    status = {}
    url ="https://api.newrelic.com/v2/applications.json"
    headers = {'X-Api-Key': apiKey}
    logging.info(url)
    r = requests.get(url, headers=headers)
    json=r.json()
    for application in json["applications"]:
        if application["name"] in interestedApps :
            status[application["name"]] = application["health_status"]
    return status

def getSpiComponents(spiApiKey):
    url ="https://api.statuspage.io/v1/pages/%s/components.json" % spiPageId
    headers = {'Authorization:': "OAuth %s" % spiApiKey}
    logging.info(url)
    r = requests.get(url, headers=headers)
    json=r.json()
    return json

def updateSpiComponent(spiApiKey, componentId, status):
    url ="https://api.statuspage.io/v1/pages/%s/components/%s.json" % (spiPageId, componentId)
    headers = {'Authorization:': "OAuth %s" % spiApiKey}
    data = {"component[status]":status}
    logging.info(url)
    r = requests.patch(url, data=data, headers=headers)
    json=r.json()
    return json

def deriveSpiComponentToUpdate(spiApiKey, newRelicAppStatus):
    spiComponents = getSpiComponents(spiApiKey)
    nrCloudStatus = newRelicAppStatus["cloud"]
    spiCloudStatus = nrToSpiStatusMap[nrCloudStatus]
    componentsToUpdate = {}
    for component in spiComponents:
        if component["name"] in cloudComponents :
            if component["status"] != spiCloudStatus :
                componentsToUpdate[component["id"]] = spiCloudStatus
    return componentsToUpdate

def updateSpiComponents(spiApiKey, componentsToUpdate):
    for componentId in componentsToUpdate :
        status = componentsToUpdate[componentId]
        updateSpiComponent(spiApiKey, componentId, status)

def init():
    logger = logging.getLogger('')
    logger.setLevel(logging.DEBUG)

    log_format = 'Process-%(process)d %(asctime)s %(levelname)-8s %(message)s'
    handler = RotatingFileHandler("statuspage.log", maxBytes= 20 *1024 * 1024 , backupCount=10)
    handler.setFormatter(logging.Formatter(log_format))
    logger.addHandler(handler)

# Main
if __name__ == "__main__":
    newRelicApiKey = sys.argv[1]
    spiApiKey = sys.argv[2]
    init()
    newRelicAppStatus = getNewRelicApplications(newRelicApiKey)
    componentsToUpdate = deriveSpiComponentToUpdate(spiApiKey, newRelicAppStatus)
    updateSpiComponents(spiApiKey, componentsToUpdate)

Move fast break things but with monitoring

2015-02-28T13:15:00.005-06:00

We run a complex system with multiple services and every 2 or 3 week we update the Java applications. I want to do it every week as most applications are stateless and can be patched anytime but the application serving the main website is using sticky session. We are working to make it failover sessions, once we do that, we can do mid week deployment and that will allow us to go faster than 3 weeks. This week I pushed a huge infrastructure change related to user Id generation. I had asked ops team to check the status of new relic after the midnight deployment and it looked like this so everyone was happy.

I woke up and checked new relic mobile app and things looked ok to me. After finishing my morning routines I ran my daily exception report and one thing that caught the eye was 90K exceptions in last 12 hours in one of the files I had changed. To gauge the impact I went in new relic and it showed me an error rate of 0.07 in one of the app

I then checked new relic and I see this blip that caused 90k errors, when the blip was there the status must have been red but then it was quickly green and Ops team didnt caught it

The issue was due to a wrong method invocation in one of the class used by this one specific application and it took just 15 min to fix after reproducing with a testcase. So why didnt QA/UAT/automated tests caught it, well the issue was like a heisen bug and would occur only when the object is missing in cache. 99% of the calls would go to cache and only 0.07% were going to dao layer that had the bug. I quickly made a patch and ops deployed it and I can see the issue is now gone.

Had there been no monitoring it would have been difficult to catch these kinds of bugs. New relic and internal monitoring tools makes life easy as it exposes anomalies.

Email slavery

2015-02-11T12:04:00.003-06:00

It seems I have become an EmailSlave. The first half of the day is spent in just answering to emails. There are so many emails where I am copied but I need not be. There are many emails where its a 1-2 page email and somewhere down someone says @KP please answer this. So it seems daily my work schedule is:

Signin to newrelic and check anomalies for 15 min.
Check emails related production exception report and yes there are a ton of these report daily. Need a better tool here as this model is not scalable. I need to reduce the incoming data at me to only see relevant data like what newrelic does. May be I need to create a webapp out of these emails.
Check emails for next few minutes before team calls
Do team calls
Then again back to checking emails until a I have taken a best shot at answering everyone waiting for my reply.
Attend team meetings on Tue/Thu

Being an architect and coder at heart I don't feel satisfied at end of the day if there is nothing tangible getting done at the end. Yes I can say hey I replied to 100 emails in a day and did 2 calls but that seems bullshit. If you read this article http://tomtunguz.com/burnout/ it quotes
""
I suspect burnout is much more pronounced for information workers - people who deal in bits each day - because unlike a mason or an architect, the product of much of our work isn’t visible. Even the tools we use, virtual to do lists and email, hide the work we’ve completed: tasks checked off and emails sent.
""

I had same problems when I was doing daily calls with the team but I have reduced it a lot by only talking to Team on Monday, Wednesday, Friday and instead of talking to 5 people now I talk to only 2. I need to find something similar for email.

It seems I am not the only one chasing this dreaded #inboxzero, I reached it once on December 15th but am again back to lots dreaded slavery.

As per this venturebeat article http://venturebeat.com/2015/02/09/sendgrid-launches-iphone-app-so-marketers-can-track-their-email-performance-stats-on-the-move/ "Email’s demise may have been greatly exaggerated — certainly if SendGrid’s efforts are anything to go by.
Last year, we reported that SendGrid had sent out almost as many emails as McDonald’s had sold burgers. A year on from that, SendGrid reports that it has sent more than 300 billion emails since launch, equating to an average of 435 million emails per day, or 15 billion emails per month.
"
Action Items:

I will myself try to think every time I hit reply All to see whether I need all these guys to be on the email or not.
Try writing short emails, twitter really make it harder for you to write as it allows only 140 chars but you eliminate a lot of garbage when you are constrained to only 140 chars. check http://www.igzebedze.com/2014/12/business-haiku-efficient-emails/?wpst=1

Lint in mouse :)

2015-01-13T20:19:00.000-06:00

My wired mouse in home setup is as old as I started working from home for my employer. Suddenly this week the scroll wheel would scroll up fine but it would not scroll more than one page, you had to scroll up half page before it would resume scroll down. At first I thought its my Firefox upgrade but soon I noticed same issue in eclipse and terminal. In evening I use the laptop without wired mouse and it would work fine. That led me to conclusion that issue is with mouse.

In USA people have a replace mentality, if something is not working and cheap, you can replace it. I was this close to ordering a new mouse on amazon when I thought let me shake it to see if something is stuck, but nothing came out. I then saw a small screw in bottom, I unscrew it and I hit the jackpot, I was like wth "Lint". My friend had similar issues with Lint in Iphone5 when his iphone wont charge and we used a breadtie to get out the lint and it was charging fine.

I was like yay problem solved and now mouse is working like new again.

Can you remain a fullstack developer?

2014-12-20T13:12:00.000-06:00

http://astartupguy.blogspot.com/2014/12/can-you-remain-fullstack-developer.html

Webhooks and Integrating Aseembla svn commits with Pivotal tracker stories

2014-11-23T13:37:00.000-06:00

I am intrigued by this webhook concept and it seems a very nice way for B2B communication. Webhooks are powerful and it eliminates Poll for integrating with third parties. All you need is to have some REST api registered that will be called when an event occurs. A good e.g. of webhook for a cloud storage provider can be "automatically print this document on registered printers when a file is dropped in this folder". Lets assume all the customer needs to do is register a webhook "http://xyz.foo.com/printer/xxd344/print?token=authTokenXXX and the cloud storage provider can then call this url and POST the body of document in input.

Recently I had a chance to play with webhooks when I was trying to move jenkins to EC2 for my friend and as part of it I moved his svn to assembla.com. I saw webhooks and I thought I can integrate commits into svn hosted by assembla.com to pivotaltracker.com tickets. It took just 1 hour to do it but it was fun, apparently there are post commit hooks that you can add in svn but this was much easier to do because assembla.com has a webhook concept https://helpdesk.assembla.com/customer/portal/articles/1607029-configuring-the-webhook-tool so all I need to do is find a pivotal api that would allow to post commit messages to it.

It seems Pivotal tracker already has this https://www.pivotaltracker.com/help/api/rest/v5#source_commits_post . The challenge was that assembla webhook doesnt allow you to pass any custom header where I can pass the X-TrackerToken header but after some api documentation trolling I found that you can pass the token in query string like

https://www.pivotaltracker.com/services/v5/source_commits?token=sdfsdfdsfdsfdsfdsf and I was done. After Configuring the integration the commit messages show up in pivotal story.

Data driven performance issue and NewRelic

2014-11-20T22:13:00.003-06:00

NewRelic really shines at discovering these data driven performance issues. Earlier we would find them late or these would be buried but now they seem so obvious if the engineer is paying attention. I was casually trolling new relic and sorted all apps by avg time per api and one of our core application in one DC was taking twice the avg time for each call than all other DCs. I immediately compared that DC with other DCs and I saw was a graph like below in DC1

and I saw this in DC2

Clearly DC1 is spending abnormal amount of time in database. So I went to database view and saw this in DC1

and I saw this in DC2

Clearly something is weird in DC1 even though its same codebase. 309K queries per minute seems abnormal. Within 5 min I found out its a n query problem. Aparently some customer has 4000 users and he has created 3000 groups and the group_member table has 40K rows for this customer. Normally all of our customers would create 10-50 groups and there is a code that iterates over each group and calls get members.

For normal customers if he makes 100 calls per minute to this api it would cause 100*10 calls or 1K calls per minute but in this dc it causes 100*3000 or 300K queries. As we are near to weekend release, for now I replaced n query with a bulk query and then we would optimize this code in next release or work with customer to see if his data modelling has some flaws and it can be achieved some different way.

AWS and rise of devops

2014-11-16T16:25:00.001-06:00

I used to always wonder how Snapchat, Pinterest and Instagram were able to scale to millions of users with just 10-15 engineers. I am a Java Architect but when it comes to networking, operations and other stuff I am a Noob beyond basic skills. Recently our ops team did some subnet changes and some IP changes and added 10G network between some services, All this is grey area to me and I was like you really need to hire Operations for this so how come these other startups did without so many people. One of my friend was after me for months to help him move his jenkins servers from Ukraine to EC2 as Ukraine is in turmoil. I have no ops expertise so this was tricky but here is how I got it done over 2 weekends as Dallas is freezing due to cold front and I dont have driver license due to immigration fiasco by USCIS. So this friend really got benefit due to it as I had nothing else to do on weekend.

I took a vanilla CentOS AMI and launched an instance in EC2. But when launching it asked me whether I want to launch in ec-classic or EC2-VPC. I was curious so read that EC-VPC will allow you to isolate your instances and allows better security groups by blocking traffic to internal servers from outside using security groups.
Creating a VPC was piece of cake as I followed the wizard and read the docs. I really wanted to use http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Scenario2.html but went with http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Scenario1.html as the earlier one was requiring an extra NAT instance.
Finally I launched an EC2 instance with CentOs and installed jenkins using sudo yum install jenkins
One thing I wanted was a banner when I ssh to the instance so I googled and found that all you need to do is go to http://patorjk.com/software/taag/#p=display&h=0&f=Big&t=Jenkins%0A and generate a text banner and put it in /etc/motd and you are done. now when you login to instance it prints "Jenkins"
Now how do I move jenkins, his old instance was up for 2 years and I read moving jenkins from one box to other requires just copying home from one box to other http://stackoverflow.com/questions/8724939/how-to-move-jenkins-from-one-pc-to-another But when I did du -hsm I got 40G. I was like no way I want to move all this shit.
Finally i installed jenkins thinbackup plugin https://wiki.jenkins-ci.org/display/JENKINS/thinBackup on both servers. On the old one I took a thinkbackup and then zipped it and overwrote the JENKINS_HOME directory /var/lib/jenkins .
Then I ran "chown +R jenkins /var/lib/jenkins"
I asked him to switch DNS to new server.
I took a dump of svn and import into a new https://www.assembla.com account and due to this I didnt needed to setup svn server separately.
We restarted the jenkins using "service jenkins restart" and the new jenkins was up with all configs as old server, I changed all svn repo paths to point to assembla credentials/urls and we were done.

Now comes the hard part. After all this was done, he was using one box already in EC2 for running selenium tests and that agent was down. No matter what I do it wont connect. I checked jenkins config page and it was using port 15001. I edited Ec2-security group and allowed 15001 port and it wont connect.

Then I thought may be I need to run this box in same VPC, me being a noob this was a bad idea that derailed me. In EC2 you cant migrate an instance from ec-classic to EC2-VPC. The only way is to create an AMI from old instance and launch a new instance, I did that and it took 4 hours including reading docs to create an AMI, but even then it wont connect. I deleted that AMI/snapshot and new instance and fired up old instance.

Finally I switched to debugging using raw telnet and saw that from the jenkins instance I can telnet to localhost 15001 but I cant from his laptop or from selenium box. Finally figured out that the CentOS AMI I picked had its own firewall and had only 80 and 443 port open.

We added
sudo iptables -I INPUT -p tcp --dport 15001 -j ACCEPT

and finally all was done.

In short EC2 had made bare bones operations people job in jeopardy, they need to move to devops. AWS is innovating like crazy and today he sent me some links on AWS Lambda http://aws.amazon.com/lambda/ and AWA Aurora https://aws.amazon.com/blogs/aws/highly-scalable-mysql-compat-rds-db-engine/ I was like hmm if startups have all this then do they need to hire Mysql dbas until the site has reached a large momentum.

Finally after doing all this I understood how Pinterest, Snapchat, Instagram were able to keep up with a large infrastructure with least no of employees, even less than my employer when they were at similar scale. For storage you use S3, for database you use aurora, for load balancing you use ELB and you automate build/deploy via jenkins/puppet and now a days docker. I remember 10-14 years ago when I started working for startups, they have to hire an army of people to get prototype out of the door and that means VCs need to write a Series A check. It seems today people at YC are doing it with 3-4 people with only Seed funding. So the seed funding has become the old Series A. Who needs an army of people to install/manage 100s of severs when automation tools combined with Power of AWS, GCS can do the job for you.

Off-course all this comes at a cost, when the site becomes really big, AWS bills are higher, recently at my employer one of our GCS performance testing env had a bill of $10K for 1 month which went in poof as you don’t own the hardware, you lease it so its like Renting vs owning the house. But we were able to finish perf testing quickly as bringing up new env was faster than stuck in hardware pipeline. Also Adrian crockfort once said if you are leasing SSD then its even better as the wear out of SSD is not your problem (http://observationdeck.io9.com/not-my-circus-not-my-monkeys-457554833). But startups have an opportunity cost, if they can get the product out with less developers in less time then later when they become big they can hire specialist Also each developer now a day in bay area has a fully loaded cost of >150-200K so for startups strapped for cost, AWS and GCS can be a boon and they can solve the high bill problem when they really reach that stage. Isnt it great if you get an AWS bill of $100K because that means you must be making 10-30 times the $$$$.

A well intentioned public api can bring down a server

2014-11-10T15:55:00.000-06:00

Apis are powerful creatures and people can use them to do tons of weird things. We had exposed a public api to create a link but our UI had a capability to select multiple files and generate links for them in bulk in call, so our public api mimicked this behavior.

Today a server was running hot with full GC, I took a heap dump and restarted it. Upon analysing the heap dump in Eclipse Memory Analyzer I found that Sys log appender was choked and it had a queue of 10K messages with each being 2MB. I copied the value and found the class name in log message.

Aparently whenever a link was created a log message was written that would iterate over each file and log a line for each file. There was a bug in the log message that it would log entire message instead of that file.

for (target in linkRequest.getTargets()) {
logger.info("queuing preview generation for {}", event) ;
}

QA/developers cant detect this and most people in code review focuses less on logger messages.

But there was a customer who today did some automation and created a link for each file in the folder in one call, he had 10K files in that folder. What this means is that the 2MB payload for link generation request was logged in for loop 10K times causing full GC.

The fix was simple to just change logger message in for loop but I will talk to API designer to put some upper limit on no of links created in 1 call.

for (target in linkRequest.getTargets()) {
logger.info("queuing preview generation for {}", target) ;
}

Paralysis by Analysis

2014-10-31T23:31:00.004-05:00

A lot of times we run into Paralysis by Analysis. We use Lucene for indexing cloud storage data and its a very old home grown system, we are soon going to retire it and replace it with Elastic Search. Recently we ran into Paralysis by Analysis situation where we were waiting for Elastic Search project to go live to implement Folder Search and we were debating over it for months while customers were cribbing for it. The move to Elastic Search requires migration of billions of files and PetaBytes of contents to be migrated which can take months. Finally I got approval that instead of waiting for ElasticSearch lets do a simple canonical prefix based search out of database directly. It took 2 days to implement server api and 1 week for UI,plumbing work and QA. As we use Mysql Sharding, the solution scaled fine, all I needed to ensure was proper indexes were there. Two days I got email from Support that customers are in love with Folder search :).

Offcourse this is not perfect solution because we cant do fuzzy searches, word tokenization and all the power that comes with Elastic Search but may be 90% of the customers dont even need them and eventually after Elastic Search we would reach there. This is why hackathons are more important because people would come up with quick hacks that weren’t earlier done due to Paralysis by Analysis but improve the product. Not each hack would make it to production or some would make it but not scale or would get low usage from customers, so they need to be culled. But sometimes all you need is "to try". Some hacks If I have to participate in a hackathon would be :

Activity streams on Folders
Folder comments
Workflows on top of documents
User and Groups Audit report
Better custom SMTP integration
Status monitor like https://status.newrelic.com/
Show creation time and posted time of file on UI
Server based trigger for uploading sync client logs (this is one of my pet peeve).
Parallel execution of unit tests
and many more

Thinking of moving the blog to Quora or starting a new public blog

2014-09-04T23:30:00.002-05:00

Reading a book on inbound marketing http://www.amazon.com/Inbound-Marketing-Attract-Delight-Customers/dp/1118896653 by Hubspot founders and it clarified some of my concepts. I intend to finish the book, its just that in the age of Quora, Twitter and Techcrunch you can get easily distracted so I am reading at turtle's pace. Even Netflix and cable TV has lost its charm for me. One reason I had ordered a hard copy instead of kindle so I don’t get distracted and can read it with focus, but now a days I feel like Quora is much more interesting than reading books. But books have their own importance, Quora and blogs are random reading with no pattern and you get some pointers whereas books are more structured reading so on any day I would prefer book over blogs if I have to read a concept in detail and call me old school but I still would prefer paper copy over ebook.

Anyway coming back to the topic I still get close to 50K hits per year on this blog, so thinking how can I apply concepts of book to the blog.

On Quora in past 6 months my content had 31.6K views

In terms of hits it doesn’t makes much difference to switch but earlier I used to think that you write blogs and search engine would scrape it and you would get find(which happened for a rabbitmq post I wrote which get linked to rabbitmq site and is the most visited post on this blog). Apparently I was dead wrong the book says you need to not only need to write content but you need to market it also on social sites like facebook, twitter, linkedin and retweet it and what not so people would link it. I don’t feel comfortable sharing all the posts from this blog as is due to various reasons so I am thinking of moving to Quora for posts that I feel comfortable sharing and I can share them on twitter/linked and leave the private and more technical posts here (the idea is not to market myself but just compare stats and see if same concepts in book can be applied and I can see them in action. I had applied Google analytics first on my blog and then in the my employer startup the same way. Alternatively another idea is to start a new blog on blogger for public posts and leave semi private posts here.

Still debating and will probably reach some conclusion soon.

Using email to manage work queue can be deceptive

2014-08-30T01:01:00.000-05:00

I loathe on emails that have been replied more than 5-6 times. I am sometimes copied on emails where there are 20 replies. Today I have an email about post commit reviews which has 20 replies, Grrr. I procrastinated whole day to read it and looks like I am not goint to read it today.

Problem with 20+replies email is that by the time you end up reading all 20 of them you have lost context of what is the final picture.

I saw this infographic from 2012 http://mashable.com/2012/11/27/email-stats-infographic/ that 28% of time is spent on reading emails. I think for me its may be 50% or more as I work from home and most communication is over emails except very few which happens over skype or phone calls.

Would love to explore tools to save time. I found 2 new tools in gmail so far, "Canned Response" and "Mark as Read" button, lets see if it helps.

For e.g. I would be copied on email "23.7.3 CL- 120157 on all Appnodes and EOS nodes in SJC, AVL and AM2", now I don’t want to delete it as I may need it in case there is an issue in prod. But I also want to move it out of my Unread queue as I am using email to manage my work queue. I am hoping now I don’t need to open email and mark it as read, a one click would do it.

Also thinking of adding more filters to just sweep some emails I care less to trash as other members are taking care of dealing with them.

But this got me into thinking should I use email to manage my work queue, I used to use a task list before but problem was that some items would never get done. Then I switched to gmail and all pending work would be in emails marked as Unread, I would have read it but if any action is required I would mark it as Unread. I am almost at a tipping point because it takes me almost from 9-4 to talk to my team and deplete the email queue. That leaves sometimes 1 hour or sometimes 2 hours to write any kind of code. Sometimes it feels like I got rid of all emails that came to me in the day but I got nothing productive done. So was I just pretending to work?

Also gmail is deceptive, when I come in morning it shows me I have 76 unread emails but daily there are many emails that have between 3-5 replies and some are suckers with 10+ replies. So really its almost 150+emails to deal with and this is after sending lot of emails to trash directly.

Need to find some other way to manage the work queue, I had tried using JIRA but that also hasn’t fared well.