Programming fun at startup

This year's plants in my vegetable garden

2013-04-27T10:15:00.000-05:00

hybrid tea

garden beans seedling

tuver/pigeon pea seedling

okra sedling

cucumber seedling

mint

beet root

basil

carrot/cilantro

toro 7.0hp lawn mower would start and dies within 20 sec

2013-04-24T22:13:00.002-05:00

This season after the first mowing on my second mowing my Toro 7.0 hp lawnmower would start but dies quickly. I first thought its a air filter issue so I cleaned it but then it again showed same symptoms. Then I thought its a spark plug issue so I went to lowes and got a new spark plug and replaced it but again it wont start. So I started googling.

Turned it was due to the gas I was using, I had some 4 months left over gas that I had used in first mowing and then I got new gas and on second mowing. So either it was due to old gas or the new gas was bad. Anyways googling tells me that its a carb issue and I need to open the carb bolt and carb cup and clean it. In case you dont know where it is see the image. This bolt has a hole and if air filter is bad and grass clippings goes in this or dirt goes in here then that hole will be clogged. Or if the gas has ethanol then I read that this hole can get clogged

So you need to pinch the fuel line or empty your fuel tank and then open the bolt to release the bowl, then you can clean the bowl and use the bread tie knot wire to clean the hole or use a needle to clean it.

I plugged it back and problem solved.

I got some fuel stabilizer and added to the remaining new gas but after fourth mowing I again ran into same issue so It must be because of the new gas.

2000 unit tests mark

2013-04-23T15:41:00.000-05:00

This is a great progress. Six to eight months ago tests were there but not running on jenkins and after I added to jenkins the team has stood up and added more than 1500 unit tests. Thats a good sign because now I am not called on wekeend by Ops team to fix a regression bug.

Still the code coverage is only 35% but we are making progress to improve it, when we started it was near 25% so this is a good progress.

Misconceptions of working from home

2013-04-22T17:38:00.003-05:00

People think "oh you work from home", 50% of think you are unemployed, wth.
Many people think you slack
Your family think you can take an hour off early just because you work from home.
Family come with all sorts of requests and enters in your work area many times a day.
Kids dont understand the concept of working from home and if they are at home instead of daycare then they can enter X no of times and want to work on your laptop.

Still I enjoy working from home because I get long stretches of time to think and implement cool things for the startup.

Mysql deadlock during delete query and index

2013-04-16T20:56:00.001-05:00

I had written a hierachical lock manager using mysql and once a thread finishes it was unlocking by deleting the locks. The 20 thread perf test was fine but daily some 100-200 of deadlock exceptions would come randomly on some nodes. Now each thread is working in isolation so it doesnt makes sense to get deadlock, the query was like

delete from hie_path_locks where customer_id=? and thread_id= ?

Finally after some hit and try and troubleshooting I found that we had an index on customer_id only and when 10 thread on same workgroup would try to read locks to detect conflicts they would do

select * from hie_path_locks where customer_id=? and thread_id= ?

apparently mysql default isolation of REPEATABLE_READ would take locks on even rows that are read, check http://www.mysqlperformanceblog.com/2012/08/28/differences-between-read-committed-and-repeatable-read-transaction-isolation-levels/

Adding an index on customer_id,thread_id instead of just customer_id solved the random deadlocks.

Spring 3.2 quartz 2.1 Jobs added with no trigger must be durable.

2013-03-27T22:22:00.001-05:00

I am trying to enable HA on nodes and in that process I found that in a two test node setup a job that has a frequency of 10 sec was running into deadlock. So I tried upgrading from Quartz 1.8 to 2.1 by following the migration guide but I ran into an exception that says "Jobs added with no trigger must be durable.".

After looking into spring and Quartz code I figured out that now Quartz is more strict and earlier the scheduler.addJob had a replace parameter which if passed to true would skip the durable check, in latest quartz this is fixed but spring hasnt caught up to this. So what do you do, well I jsut inherited the factory and set durability to true and use that

public class DurableJobDetailFactoryBean extends JobDetailFactoryBean {
    public DurableJobDetailFactoryBean() {
        setDurability(true);
    }
}

and used this instead of JobDetailFactoryBean in the spring bean definition

    <bean id="restoreJob" class="com.xxx.infrastructure.quartz.DurableJobDetailFactoryBean">

graphite dynamic counters trending

2013-03-23T00:35:00.001-05:00

I generate this report daily using cron as to top exceptions across all datacentres and top queries across all datacentre and top urls across all datacentres and send them via email.

Problem is that after every release the no goes up and down as due to some bug a new exception will popup or some exception will resurrect. How do I trend and correlate these dynamic counters.

Solution came from my colleague in just an informal chat and he recommended I should md5-hash the url and create a graphite counter for it and in the email just make the count a hyperlink like shown below.

Now I can trend the query as clicking on this shows me a graph as shown below. My next plan is to inline the graph for top 10 urls in the email itself so I don't even need to click them.

Being Analytics driven vs firefight driven

2013-03-23T00:26:00.003-05:00

We have doubled our incoming traffic every 6 months and past 3 years have always been in firefighting mode where some customer reports an issue or a node goes down and we try to analyze the root cause and fix them.

Lately I am trying to move away from working in a firefight driven mode to Analytics driven mode. What I meant is being proactive to monitor and understand the system by gathering various metrics and fixing issues before cusotmer notices them. For e.g. to put our nodes in HA mode I had to store a sessionId to userId mapping in database, the only real reason to do this was for our Flash file uploader because it makes a request but doesnt pass the sessionId in cookie but it passes as request parameter. This causes the request to go to a completely different node. So to handle this we wrote a session listener that would save the sessionId to userId mapping in db. The code went live and suddenly after some days the db went down. What happened was that the developer forgot to remove the row in session purge and the table had grown to 140M. I didnt expected this table to be more than 1-2M at a time but due to code bug this issue occurred. So first thing I did was to add a monitor on all tables in our transient db and it alerts me if any table has >10M rows. Immediately it caught another code bug where an audit table has grown to >70M for one customer so we caught it and fixed it in hotpatch it without causing another downtime.

Another incident happened two weeks back when I travelled to Bay area and for two days a db went down exactly between 11:30 and 11:45 and restarted on its own. What we found that again that session table was 20M even after adding the session destroy delete fix and a backend purge job from all nodes were trying to delete 18M records out of this 20M table. What happened was that some code flows(webdav and FTP upload) that are not supposed to be session dependent were setting user in session. Users uploaded 20M files using FTP/webdav and that caused this table to ballon. The only reason I was able to find the issue was because I log stats on each node as to what methods happened every 5 mins. So I grepped queries in those between 11:30 and 11:45 on all nodes.

Immediately I realized that I need something more global, we have 40 mysql servers and I needed some report as to how many queries are happening on each node and in overall at DC level. If I had this report then I woudl have caught the session thing before it became an issue. Scribe comes to rescue, now I write those 5 min stats to scribe that gets aggregated in central scribe across all datacentres. I roll the log nightly and wrote a python cron to generate report that dumps queries and no of times they happened and their avg time.

Yesterday it went live in UAT and immediately I found 2-3 issues where top queries were not what we expected (caching bug :)). Lesson learnt is be analytics driven rather than hunch or firefight driven.

Haproxy and tomcat JSESSIONID

2013-03-23T00:00:00.002-05:00

One of the biggest problems I have been trying to solve at our startup is to put our tomcat nodes in HA mode. Right now if a customer comes, he lands on to a node and remains there forever. This has two major issues:

1) We have to overprovision each node with ability to handle worse case capacity.

2) If two or three high profile customers lands on to same node then we need to move them manually.

3) We need to cut over new nodes and we already have over 100+ nodes.

Its a pain managing these nodes and I waste lot of my time in chasing node specific issues. I loath when I know I have to chase this env issue.

I really hate human intervention as if it were up to me I would just automate thing and just enjoy the fruits of automation and spend quality time on major issues rather than mundane task,call me lazy but thats a good quality.

So Finally now I am at a stage where I can put nodes behing HAProxy in QA env. today we were testing the HA config and first problem I immediately saw is that we have to use sticky sessions due to some design issue that will take long time to solve. Now we were doing sticky session by JSESSIONID like

appsession JSESSIONID len 32 timeout 12h request-learn

Immediately I realized that Two tomcats can generate same JSESSIONID so there is a potential for security breach. Thank god tomcat has a way to add a node identifier to JSESSIONID to solve this issue :). You can go to conf/server.xml and add jvmRoute to Engine like

This way your JSESSIONID would be generated like BBF8B5EF74EAAECE0278DC92A9F1353D.192.155.4.5_8080

As a side effect now we would know by looking at cookie as to which node is serving the request and this will help in trouble shooting node specific issues.

I hope to cut down the 100+nodes to 40 nodes after this HA. I will keep 40 because we have pod/farms in each DC so we need to overprovision each pod else this could have been reduced to 10 or 15 nodes. The pods are there to avoid DC meltdown.

%E2%80%90 and links issue

2013-03-07T13:38:00.004-06:00

Ran into an issue where a customer creates public link to a file and pastes into word and it works fine but when he converts it to PDF it no longer works thought the link in browser url bar looks exact same. I reproduced it and saw that "h-s" in link url was getting converted to h%E2%80%90s. After some Googling it turned out to be an adobe bug http://forums.adobe.com/message/2807241 related to hyphen character.

Final nail in BDB coffin-Part2

2013-03-01T21:03:00.002-06:00

Missing indexes can be really pain. We were migrating data from bdb to mysql and the migration on few nodes were going on for 3-4 days. As I was involved in firefight, I didnt got a chance to look at it. But on one node only 10% of the workgroups were migrated and while chasing a customer reported issue I found on index was missing. we created that index and restarted migration and wow it finished in 5 hours.

Finally we ended up with 400M+ more rows in our mysql infrastructure and now bdb is finally out of the product. Hurray!!

Spring an Quartz JobDataMap serialization exception

2013-02-14T15:52:00.001-06:00

We dont run our nodes in HA yet and once a customer registers he is assigned a node and he lives there. Problem is that if the node dies we incur a downtime for that customer and also we need to overallocate hardware to prepare for worse case scenario. for the past 6 months I have been working to making the code stateless so that we can do HA and reduce our node count by 4 times.

So we used to run quartz using in memory scheduler but for HA I need to run quartz in a cluster. We chose org.quartz.impl.jdbcjobstore.JobStoreTX for this.

Problem was that as soon as I tried it I ran into issues because I was injecting spring beans into our quartz job using JobDataMap and JobStoreTX was trying to serialize the jobData into a table and our spring beans are not serializable. There were two options:

1) Load the entire applicationContext in each job and read the bean from there.

2) Use the schedulerContextAsMap.

After evaluating options I found scheduler context as the best option .The way to do this would be

    <bean id="haQuartzScheduler" class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
        <property name="configLocation" value="classpath:quartz.properties"/>
        <property name="triggers">
            <list>
                <ref bean="CommitFileJobTrigger" />
            </list>
        </property>
        <property name="schedulerContextAsMap">
            <map>
                <entry key="commitFileJobHelper" value-ref="commitFileJobHelper" />
                <entry key="numThreads" value="2" />
            </map>
        </property>
    </bean>

In the quartz job I wrote a function

    protected T getSchedulerContextBean(JobExecutionContext context, String beanName) throws JobExecutionException {
        SchedulerContext schedulerContext;
        try {
            schedulerContext = context.getScheduler().getContext();
        } catch (SchedulerException e) {
            throw new JobExecutionException(e);
        }
        T bean = (T)schedulerContext.get(beanName);
        return bean;
    }

and then all jobs can use this function like

        CommitFileJobHelper commitFileJobHelper = getSchedulerContextBean(context, "commitFileJobHelper");

Final nail in BDB coffin

2013-02-14T15:43:00.002-06:00

This weekend we would finally put the Final nail in BDB coffin. We were using BDB in webdav, Cloud file and backup product. Over the course of last 6 months my team was able to remove BDB from webdav and Cloud file and the mysql is scaling fine. We have now billions of rows in mysql and last weekend we had a pilot migration of few backup product nodes. This weekend we would strive to migrate all the backup nodes. I am hoping we would increase the no of rows in mysql by 30%.

Mysql and sharding rocks!!

Being ruthless

2013-02-09T23:33:00.002-06:00

Lot of time our system deals with abuse. For e.g. some customer will move the same file between 2 folders the whole day and normally it doesn't cause issues but in some extreme cases it would generates hundreds of thousands of records. Also there are some customers who have bots written that will make millions of call in a day. Or sometimes some customer will put malware on FTP and use our servers as a way to spread malware, this causes antvirus to flag our site as spammers causing field issues. One of the strategy we use to deal with abuse is to throttle the user for a while but sometimes it hurts good users also. In some cases the abuse is so much that it can bring down the system or hurt other genuine users. Like in case of malware we just block the customer as there is not time to reach the customer and solve the issue, some user might have accidentally share the file but we have to be ruthless.

Feels great to see that your product is used

2013-01-30T11:03:00.001-06:00

I work remotely for a company in california from texas and was working on a customer ticket and saw that the customer account in my city only.

Feels good to see that the product you work on is used globally and even at a stone's throwaway from you.

openldap adding an index

2013-01-15T17:14:00.001-06:00

We use ldap to store customer metadata like users, customers and other stuff. Right now each node has a ldap and we are trying to consolidate ldaps across different nodes. So I loaded one ldap with data from 20 nodes and some of the queries were taking 4000 msec. So I saw that indexes were missing .

I went to slapd.conf and added

index customerNumber,customerDomain,email eq
index customerName eq,sub

I also updated the cache size to 4G in DB_CONFIG file

set_cachesize 4 0 1

I restarted ldap and suddenly things were flying but something was fishy as the new numbers for 100 threads was faster then previous 1 thread time for all operations. So I picked one query and ran it manually and found that a simple query like below was coming empty

ldapsearch -x -H ldap://localhost:389 -b "dc=xxx,dc=com" "(&(objectclass=objuser)(email=kpatel@comcast1.net))"|grep email|more

I removed the index and restarted ldap and same query started returning results. Upon googling I found that if you add a new index you have to reindex your data, wth. so I
1) shutdown ldap
2) ran

/usr/sbin/slapindex -f ~/openldap/etc/slapd.conf -b "dc=xxx,dc=com"
3) started ldap
and now the query was returning results.

And LDAP rocks even 500 threads its rocking fast.

It seems people work less in december in US

2013-01-14T20:32:00.001-06:00

Traffic to my blog was down to 60% in december and I was worried what had changed but I am glad to see that its back to original level in 2nd week of January. It seems people work less in December.

Fun to see bugs

2013-01-14T20:28:00.004-06:00

Received this spam email today morning and first thing i saw that instead of image it had a path of "c:\users\Manoj". :)

LDAP wildcard search

2013-01-02T15:29:00.002-06:00

I was able to make a LDAP query with wild card on a field username like

(&(username=*kpatel))

but I wasnt able to make it on a field like

(&(customername=*kpatel))

me and my teammate searched and finally found that username was inheriting from a super type uid and thats why it was working on it but customername was not.

Luckily solution to the problem was to add a SUBSTR clause to enable wildcard match on a field in ldap schema.

SUBSTR caseIgnoreSubstringsMatch

Making life easy at mysql command line

2012-12-21T21:07:00.001-06:00

I was annoyed with these same problems especially paging query results at mysql command line or grepping within results or recording queries for future purpose. For all those debugging productions issues related to mysql here is an excellent blog from perconna to make your life easy

http://www.mysqlperformanceblog.com/2012/12/21/be-productive-with-the-mysql-command-line/

CGlib enhancer and finalize method

2012-12-19T16:57:00.001-06:00

Yesterday was a bad day because one of the node after updating to tomcat7 ran into full garbage collection. I took a heap dump and finally found that one of the enhanced class had 300K references hanging around in finalizer thread. I was enhancing SimpelJDBCTemplate to find out time taken by each query and log it. The reason this happened because CGLib also enhanced the finalize method and in the method interceptor I was delegating the call to the delegate object which was not existing at that time. Anyways the solution was to skip the enhancing of finalize by adding a callback filter and nooping on it.

As you can see in below code the filter returns 0 if finalize method is called and that means use the callback at 0th position in the callbacks which is a NoOp callback and for all others it uses 1st position which is my real code.

The simple code fix was

    public static SimpleJdbcTemplate createPerfInterceptedTemplate(Class callerClass, DataSource dataSource) {
        final SimpleJdbcTemplate simpleJdbcTemplate = new SimpleJdbcTemplate(dataSource);
        final String className = callerClass.getSimpleName();
        Enhancer enhancer = new Enhancer();
        enhancer.setSuperclass(SimpleJdbcTemplate.class);
        enhancer.setCallbackFilter(FINALIZE_FILTER);
        enhancer.setCallbacks(new Callback[]{NoOp.INSTANCE, new MethodInterceptor() {
            @Override
            public Object intercept(Object obj, Method method, Object[] args, MethodProxy proxy) throws Throwable {
                ..........do your real delegate code here..........
            }
        }
}

private static final CallbackFilter FINALIZE_FILTER = new CallbackFilter() {
        public int accept(Method method) {
            if (method.getName().equals("finalize") &&
                method.getParameterTypes().length == 0 &&
                method.getReturnType() == Void.TYPE) {
                return 0;
            }
            return 1;
        }
    };

Tomcat7 session fixation and session listener

2012-12-19T16:51:00.003-06:00

Ran into an interesting issue. We use flash uploader in our web ui for browsers that dont support html5 and flash has a problem that each request it makes to server has a new session because it wont send any cookies back to server. The only way to circumvent around this is to send original sessionId as a post parameter and on server cache all the sessions in tomcat memory and then join this new session to the original session using sessionid coming in post.

Anyways long story short we updated to tomcat7 and suddenly one of our feature that allows us to impersonate a user broke. Finally nailed it to a security fix in tomcat7 that will renew sessionId on basic authentication but the issue is that to do flash based file upload we relied on HttpSessionListener.sessionCreated to cache all sessions by sessionId. And when tomcat7 was renewing sessionId it was not calling the sessionCreated event for the new session. There were two ways to solve it:

1) Disable session fixation security fix by adding the below valve to context.xml . I did not chose to do this as it would make tomcat less secure.

<Valve className="org.apache.catalina.authenticator.BasicAuthenticator" changeSessionIdOnAuthentication="false"/>

2) Extract the code in HttpSessionListener.sessionCreated to a common method and call it manually during impersonation. I chose this way for now as its more secure. In future when tomcat fixes this bug I will remove this code.

Biggest relief from NOSQL to SQL migration

2012-12-13T22:31:00.000-06:00

This year my biggest accomplishment was to move our old NOSQL system from BDB/Cassandra to Mysql and so far its holding billions of rows and working fine. Though this move has given me and my buddy peace and good sleep and I can now focus on other fires. But the biggest relief comes from being able to delegate some tasks to junior team and also being able to quickly script adhoc requirements quickly.

For e.g. today I got an adhoc requirement to find list of customers with > 1000 versions of a single file. Had it been BDB I would have to write a program and then run it on each app node to find the answer and it would have taken days to get this info. But with mysql all I had to do was to write a script that will execute a federated query and get me the output so all I need to do is run something like

nohup python sweep_all_shards.py "select pid, max(cnt) from (select customerid,file_id,count(version_id) cnt from \${SCHEMA_NAME}.version_\${TBL_SUFFIX} group by customer_id,file_id having count(version_id) >1000)a group by customerid" &

and within 6 hours the script had sweepeed all the shards and got me the result. So I had to just do some grep/awk/sed to get the answer.

Few weeks back I got some adhoc requirement to find max size of file uploaded by each customer and same I got it in less than one day whereas earlier I would have spent days getting this info.

Also one more advantage I see is that I have created a read only user on each mysql slave and given access to production support people and as everyone knows relational database very few requirements come to me so I get more free time to work on more interesting things.

Also last night mysql server crashed twice while creating index on a 85M record table and boy the DBAs knew how to recover it whereas when it was BDB it was an Achilles hill to recover data out of it and this time Ops team did it without involving everyone and they were all confident and relaxed and BDB/Cassandra was always panic mode.

Got first mysql table with 84M records

2012-12-13T22:17:00.000-06:00

wow this is the first time I had scaled a database with 84M records in one table. Though I didn't expected it to grow this big but a weird customer behaviour where he had 17K versions of a file and he moved it 5K times. 17K version * 5K move=85M events generated in one shard in one table.

Within a month these will get purged as we retain last one month event, so table size will go down but it feels good that the system is behaving nice after adding the missing index on this 85M table, today its again sleeping like a baby.

A copy paste mistake can bring down a server

2012-12-13T16:53:00.000-06:00

we recently migrated our eventstore on Mysql and I did a small boo boo. I had two tables event and event_details and I had created two indexes on it

CREATE INDEX events_${TBL_SUFFIX}_i2 ON events_${TBL_SUFFIX} (event_detail_event_guid, pid);

CREATE INDEX event_details_${TBL_SUFFIX}_i1 ON events_${TBL_SUFFIX} (event_detail_event_guid, pid);

As you can notice the boo boo in second index Instead of creating it on event_details table I created the same index on event table :(.

Yesterday night a shard event_detail table balloned to 85M records and there were 20 threads doing a full table scan on this table.

so I fixed the copy paste mistake and generated ddls for each shard but the mysql server kept going Out of memory everytime it tried creating index on this 85M record table. Ultimately the only way to get it done was to start mysql on diff port so no one will connect to it and then give innodb more memory and create index, reset innodb settings back and then restart the server.

All in all it took 6 hours in night to fix the after effects of this copy paste mistake.