The Microsoft Bizspark challenge is one of the events my company participated in. We made it into a regional final but didn’t progress beyond that ( the why part of it probably warrants another blog post ). We presented our product and watched many other startups take their turn telling everyone what they had built. I’d like to talk about one of the companies that presented their product on MS Bizspark. To keep things anonymous, lets call the company that presented their product – X.

Company X has a simple solution. They take a process that is performed manually today and automate it through their web portal. All that the portal does is upload / download data in some form. Something that took 1-2 weeks can now be done in 2-3 days. They are cash positive. Whenever I talk about a solution like this with a fellow hacker, some of them go – ‘Really !? I could have done that in under a week !’. They’re missing the point.

Read other HN posts like the Spanish Bible app that made 70k in revenue and the 7 minute workout app that clocked near 100k downloads and you know that its not about how much time you spent working on a problem or how complicated the problem was. It is how you execute and pull other strings that differentiate you from the crowd.

So Dear fellow hacker – Just focus on making something that solves a problem and executes on the idea with good design and simplicity. That is a good first step. The envy behind ‘I could have written that in X days’ usually gets you nowhere.

Discuss on HN

Book recommendations based on my entire browsing history

I’ve already read one of these books and wrote a review for it, but not on Amazon. There is a good probability that I’ll buy some of the books recommended to me.

Spot recommendations based on what you browsed over the last X minutes

I made the mistake of clicking on a link that led to a page containing a 5 kilo nutella jar for sale. Now my landing page is full of nutella products. No, I’m not buying  nutella anytime soon.

Hit or miss – Recommending product to me based on inferred preferences. 

I’ve personally found item based recommendations to be more effective (at least for me) to get me to buy something. User profile based recommendations by Amazon have never worked on me and there is a good reason why. We’ll get to that in a minute. The video games and watches were recommended to me based on my profile ( Amazon probably thought – likes programming and is male – lets show video games and watches ).

It’s probably been a decade since I played a PC game. I rarely buy watches.

If all else fail, show the user something hip that drive sales for your store

What is this recommendation page missing ? – Personalized recommendations based on what your friends / colleagues are doing. You see, amazon does not know enough about me to stop up-selling PC video games and watches. Unlike Google / Facebook, the engine does not have enough social context to make user recommendations. The reason why the acquisition of GoodReads is a major win for Amazon is the additional social context it gets to make personalized recommendations (at least for books). With the integration of GoodReads on Kindle, the combo is pretty powerful. This adds a new dimension to recommendations.

Looking at the bigger picture, products like Wanelo / Fancy encourage crowd curated niche marketplaces that are pretty social. If you can successfully build an engaged social audience for a popular e-commerce category (fashion / books / jewellery ), it may well interest Amazon.  The last time I checked, MyHabit (Amazon’s fashion site) does not use anything other than your gender and location to personalize content. These are good opportunities to fill that gap.

Discuss on hacker news.

Like this post ? Follow me on twitter.

Follow @deepakbsub

After much prodding he finally told me what the idea was and concluded his sentence with ‘.. yes and this is my idea btw’. That obviously meant he was going to try to kill me if I implement it without his knowledge  His proposition was that I write the code that implements the idea and give it to him. He wanted to keep his idea a secret.

My friend is not alone. He does not have a background in programming but has one in management. This idea sprouted in his head a while back and he kept it locked away in the hopes that he can pay a programmer to get it implemented some day. There is just one problem. His idea alone is worth zilch / Nada / Nil. The minute you have an idea, 10 other people probably had the same one. Building products for companies before and also nurturing them on my own from scratch have taught me that if you have an idea, it’s closest representation in the physical world would probably look like this.

That is an uncut diamond. In its raw form it has great potential. A diamond cutter with skill can fashion a diamond that can sell for thousands of dollars. He can also botch it up into a worthless rock. Ideas are like that. They need refinement / cutting / polishing before they can turn into something useful. Executing and refining the idea is infinitely more difficult than coming up with one. You never make money just because you have this brilliant idea that no one else thought of. You make money because you executed your idea better than someone else. Case in point – Google and Facebook. There were search engines before Google. There were social networks before Facebook (Heck Google created Orkut at the same time).

As for my friend’s idea…

• It was a location based app which was already implemented (Not that competition is a bad thing).
• His ideas had many holes in them which we discovered while discussing it. The holes would have remained hidden had he kept from talking about it.

 PS: Yes I’m aware that uncut diamonds still have value but that is the closest analogy I could come up with

Like this post ? Follow me on twitter.

Follow @deepakbsub

Test your app on the phone and not the emulator

• Time – The emulator is slow. It takes up to 2-5 minutes to load the app on the first run and it can take 40+ seconds to load it from a snapshot.
• Resources – The emulator is bulky and can slow your computer down. Not what you want when in any situation.
• Hardware – Features such as recording audio need access to hardware on the phone. While there are options to enable certain hardware access on the emulator,  it never works consistently. Some of our code would work seamlessly on the phone but fail on the emulator.

There are solutions out there that attempt to provide better / faster emulation. Out of personal experience I’ve found that nothing beats the real thing.

Parsing data structures ? Watch out

Memory

It is common to come across this code snippet while parsing certain data structures (XML / JSON etc)

BufferedReader reader = new BufferedReader(new InputStreamReader(stream));
StringBuilder sb = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null) 
{
    sb.append(line + "\n");
}

While this would work in most cases, there are phones out there whose memory limit cannot handle large chunks of strings. Even when data structures are reasonably short, the application can force close. Use a JsonReader. It parses the json in stream and occupies less memory.

GSON

Packages

Have you packaged gson with your build ? Change the package name of the gson jar to ensure it parses structures correctly at runtime. Some HTC phones have old versions of gson pre-packaged in them. Parsing json on these phones will fail.

Proguard

Gson and proguard do not work well with each other unless you follow the recommended rules set out in the google-gson project.

Exception handling

Handle exceptions aggressively. No one likes to see an ‘Application force closed’ message. Use frameworks like ACRA to report error back to you. Fix them quickly. Whatever you do, do not use a google spreadsheet for bugs reported by ACRA. The spreadsheet will eventually stop you from exporting data in any format. ACRA has also been contacted by Google to stop using spreadsheets as the default method to report bugs. Use another provider instead or write your own.

Debugging

Service

Services cannot be debugged by default. You must explicitly tell the device that you wish to connect to the debugger by writing a line of code. I lost an hour on this one alone !

Memory leaks

Attaching a debugger to your program can make it seem like there are memory leaks. Quoting the android documentation on this

The debugger and garbage collector are currently loosely integrated. The VM guarantees that any object the debugger is aware of is not garbage collected until after the debugger disconnects. This can result in a buildup of objects over time while the debugger is connected. For example, if the debugger sees a running thread, the associated Thread object is not garbage collected even after the thread terminates.

So if you ever wondered why your program crashes eventually when you try to debug it but runs fine when it is run in stand-alone mode, this could be the reason.

Async task and the UI thread

Async tasks should be instantiated from a UI thread. Quoting the Android documentation.

The AsyncTask class must be loaded on the UI thread. This is done automatically as of JELLY_BEAN.

Failure to do this will result in weird runtime errors. Even if you instantiate the first Async task incorrectly, it can affect the others.

Prepare for release

Dry run

If this is the first time you are releasing to the android store, release a simple app before releasing the one you were working seriously on. The dry run will give you a fair idea of what the process is and what to expect.

Android already offers guidelines on what to expect when you upload your app. There is also a separate guideline on graphical assets associated with your project. Take a look at the preparing for release guide before you take the plunge.

Shameless plug

You can find these tips implemented on my voice messaging app – PhonOn. Check it out on the play store.

Like this post ? Follow me on twitter

Follow @deepakbsub

Deployment strategy

There are a couple of problems to field when deploying Riak.

1. The EC2 instances that are provisioned by default change the following on restart.

• Private IP address
• Public IP address
• Private DNS
• Public DNS

2. EBS instances provide stable durable storage while Ephemeral storage provides for better predictable performance at the cost of losing data on restarts.

3. Performance.

IP Address change

Riak does not like change in IP addresses. You have a couple of solutions.

1. Listen on 0.0.0.0 to accept any connection.

2. Configure a host on /etc/hosts with a valid FQDN. For example on a 3 node cluster you would have beam listening in on riak1.something.com ; riak2.something.com ; riak3.something.com. When IP addresses change you must change the IPs that these FQDNs point to.

3. Same as #2, except you can use a private DNS server or Route 53 to manage the mapping.

4. Assign a public static EIP to your node.

5. Deploy to Amazon VPC where IP addresses do not change.

What to pick ?

#1 does not change the fact that beam instances need to find other Riak nodes in the cluster. You are still stuck trying to get Riak to find other nodes correctly.

#2 is a pain in terms of maintenance. The time taken to boot back a cluster is proportional to the number of nodes in the cluster even if you have automated scripts to change IPs.

#3 Same as #2. An over-engineered solution.

#4 You can assign at most 5 EIPs per account and that will cost you. It is also not a good idea to expose Riak to the internet with a static IP.

#5 I’d recommend this. Your IP address never changes on a private VPC which makes configuration and cluster management all the much easier. The nodes cannot reach the internet without a NAT instance. The con is that it takes some work to understand a VPC environment and set it up. Trust me though, it is worth the effort.

Storage

There is a post on the riak-users list that does justice to discussing points about storage and which one you should use. There is no single answer to this really. Your storage requirements vary widely based on the backend you choose to run Riak on. Bitcask stores all keys in memory while LevelDB stores x % of the keys in memory.

One thumb rule to guide you – if you can sleep at night knowing that you have reliable backups and can survive disk failures then you can go with ephemeral storage. EBS provides varying levels of performance based on load. However the chances that you will lose data are much lower. Remember that Riak nodes are meant to recover from failures anyway. Choose what storage option seems best for you.

Performance

I would recommend that you execute 2 performance tests.

1. Use Basho bench.

2. Use a custom load test case of your own.

Why ?

Primarily because the needs of your load will vary from what basho bench can produce. Don’t get me wrong. I think Basho Bench is awesome. But there are cases where it will not do. Consider a M/R query that is executed at 50 Ops/s. By default the number of VMs allocated to handle M/Rs cannot handle this load. This failure will not show up on basho bench since we are loading the DB with GET / PUT / DELETE and not using complex features like M/R ; linking ; link-walking ; full-text query ; index query.

Whether you use these additional features on your implementation is up to you. Just keep in mind that load test results vary depending on how the database is used.

Load and its characteristics

Your performance curve will vary drastically depending on the load you generate. If your requests are write heavy and the write_buffer configured is insufficient, you will witness horrible performance from good machines. The number of vnodes per instance also determine how much runway you have. Choose these carefully.

Best practices

Basho has a good deal of advice on its site about tuning performance. Take a look at the riak operations webinar to gain some additional insights on deployment. Feel free to drop an email on the riak users list too should you need more information. Someone always replies with good advice.

To start with create an image for your speech bubble.

Image – Original dimensions

This speech bubble is so small that any text that is placed inside it will cause the image to expand and become distorted. Lets try setting this image as the background for a cell on a list view.

Stretched Image

The result is not pretty. To get around the problem, the image must define boundaries that will tell the Android framework which parts of the image can be stretched. You can use the draw9patch tool to set these boundaries or use a custom tool like the better-9-patch tool (which I find easier to use).

So how do you use a 9 patch tool to tell android which areas of the image can be stretched ?  You define that by tweaking the boundaries on the tool until you get a defined area within the image that is confined by a rectangle.

Define stretchable area

When you save the image you get a new image with marked boundaries. The name of the image should end with “.9″ (say bubble.9.png). This tells android that it is a 9 patch drawable.

Image with boundaries

And that is it ! Pop this image into your ‘drawable’ folder and you can start using it. Here is a project that uses this image. If you’d like to look at some source-code, this project is available on github. You can also install the app on your phone by visiting the play store, where the app has been deployed.

1. You must download and install a bundle.

2. Bug fixes require you to download a new version.

3. Getting users to upgrade to version X is cumbersome.

4. ‘Upgrade’ code is necessary between versions ( Like the onUpgrade() call to SQLiteOpenHelper that handles database changes)

and so on…

You can use HTML 5 and create a web version of your app but you would miss out on the features exposed by native code (widgets etc). Features that require controlling hardware or accessing them can get tricky. HTML is not the ideal markup you would want to use to communicate a view to a rich touch device like a phone / tablet anyway.

We are stuck with this fate for the foreseeable future. I’m not sure what would make a good solution either. You could create a markup language written in a well formed document that can be sent from a server to a rich client device, and the device would interpret the markup and draw a screen using the native UI. It would not matter whether the UI should be rendered on Android / iPhone / BlackBerry. A “spinner” element for example can be rendered equally well on all devices. But I seriously doubt such markups would come into existence. You’d need several vendors to implement the UI consistently and adhere to standards. We have enough trouble getting HTML and CSS to work consistently across browsers today

A Microsoft Home CD

MS used to sell these to encourage folks to learn about the ocean. This is just one product under the Microsoft Home family. This was before Google , wikipedia became popular. Heck this was even before the internet became popular (the year – 1993). They cost 100$ a pop

 A floppy drive and some floppies

Oh the excitement of splitting a 5 MB file into several 1.44 MB disks only to later be told that there was a checksum error on disk 2. No one born after 2000 can relate to that pain. Even installation disks were distributed this way. – ‘Please insert disk 3 to continue installing blah’

A 512 MB hard disk (placed near a pencil for scale)

This thing weighs in at ~7 kilos. Who needs gym equipment when you have hard disks from the 1990s ? When I showed this to my 10 year old nephew, he said ‘Whats that ?’ . I told him it was a hard disk and it can hold 512 MB of memory. He refused to believe me.

I think I’ll ask Seagate to replace it someday. The seal is still unbroken so I guess that means its still covered under a warranty :p

A shareware CD

During the early days of the internet, download speeds sucked (33 kbps modem – You know, the ones that make a weird sound when they connect to the internet. Yes, no one from this generation will hear that either). CDs like these were a popular way to distribute a shareware version of your product. This one came with a car magazine and ‘Need for Speed’ was one of the bundled games. Wallpapers and screensavers were the rage back then.

What does this have to do with your grandchildren ?

Just like we are nostalgic about these devices / distribution methods, our grandchildren will experience something similar. Here are 5 thing I expect them to say.

1. Hey remember when you had to drive your own car ?  (This will be a sad day)
2. Here’s a picture of my granddad working at his office. Oh that giant box near him ? That was called a PC.
3. This is an old TV my grandmother used to have. You actually had to wait for a program to air at a particular time to watch it ! How did people ever tolerate that ?
4. That’s a picture of my dog frisky. No he is not trying to eat paper. Folks used to predominantly read news from paper back in the day.
5. Email ? Whats that ?

I sincerely hope someone takes #5 by the horns and succeeds. It is a behemoth which if you solve successfully, can change the way tons of people communicate everyday. I think it is a matter of ‘when’ and not ‘if’.

Countless hours of mine have been spent gawking at the graphs section of my github repos.  Its a great place to visualize how your project is growing / shrinking with time. While there is support for visualizing data on commits made on github, there is no way to visualize issues that were created on the code that was commited. Fortunately the github issues API provides a RESTFul way to extract data related to the repo’s issues.

I wrote a short script that accepts the following parameters for input and churns out some CSV data to the console.

-r <The name of your github repo>

-u <Your user name in github>

-p <Optional parameter – Your github password. This is needed if your repo is private>

The CSV data is printed on the console under two groups

• Issue timeline – A sorted data set of the issue fields created_date , closed_date. This can help you visualize the time span between any issue creation and closure. The larger the area between the two curves, the larger the inactivity on issues.

 Issues – Created Vs Closed time lapse

• Issues Open / Closed at any time – A sorted data set that shows the number of open / closed issues at any point in time. Larger space between the lines indicate more open issues at that point in time.

 Issues – Created Vs Closed count

The script is available on github. Feel free to execute it on a repo of your liking. Some things to note

1. The lines on the second graph will always converge since only closed issues are taken into consideration.
2. The graphs themselves can be misinterpreted because open issues do not form part of the data set. You can always add open issues into the list (on the python program) by changing the ‘status’ query param on the github call.

Before we get to the details…

• If you are interested in the data alone, that is fine. Just skim the sections that involve programming / analysis.
• If you know python but do not know pandas – no worries. The code is self explanatory in most places.
• If you know python and pandas, that is awesome. You can play around with the code and analyse the data in greater depth. The full program is available on github under the MIT license.

H1B – Generic foreign workers

H1B1 – Workers from Chile and Singapore

 E3 – Workers from Australia

Data of interest

The following columns on the CSV are the ones I found most interesting.

 Munging

Before the data is sliced and diced, it must be munged (a fancy way of saying cleaned).

• The wage rate column is empty in some cases – Set these to 0.
• The rate of wage is non-uniform (Salary per Hour vs Year vs Week etc) – Normalize this to a year.
• Employer name contains whitespaces / periods , which make grouping difficult – Clean the employer string.
• Some of the wage data appears to be incorrect. Like the preschool spanish teacher that makes 2 mil $ a year – Set any wage above 1 million to 0.

Slice and dice

The python program begins by reading the CSV data (all 48 MB of it) into a dataframe. The dataframe is a SQL-table like structure that is composed of series (columns) and can be sliced and diced in various ways through applied functions.

main_frame_chunks = pd.read_csv('/usr/local/LCAFY2012_Q2.csv')

Apply a function to this frame to clean the data. Each row on the frame is processed using this function.

main_frame_chunks = main_frame_chunks.apply(normalize_sal, axis=1)

def normalize_sal(x):
#    Normalize salary to a year
    if x.ix[RATE_FROM] is None or "" or math.isnan(x.ix[RATE_FROM]) or x.ix[RATE_FROM] > 1000000:
        x.ix[RATE_FROM] = 0
    if x.ix[WAGE_RATE_UNIT] == WAGE_RATE_UNIT_HOUR:
        x.ix[RATE_FROM] *=  (8 * 240)
    elif x.ix[WAGE_RATE_UNIT] == WAGE_RATE_UNIT_MONTH:
        x.ix[RATE_FROM] *= 12
    elif x.ix[WAGE_RATE_UNIT] == WAGE_RATE_UNIT_BI_WEEK:
        x.ix[RATE_FROM] *= 52 / 2
    elif x.ix[WAGE_RATE_UNIT] == WAGE_RATE_UNIT_WEEK:
        x.ix[RATE_FROM] *= 52
#    Clean employer name
    if isinstance(x[EMPLOYER_NAME],str):
        x[EMPLOYER_NAME] = x[EMPLOYER_NAME].replace('.','').strip()
    return x

Obtaining the top N counts of a particular column requires only one line of code

pretty_print('Most sought after VISAs')
    print(main_frame_chunks[VISA_CLASS].value_counts())

# Output
# ********************************************************
# Most sought after VISAs
# ********************************************************
# H-1B                          165099
# E-3 Australian                  3248
# H-1B1 Chile                      186
# H-1B1 Singapore                  184
# Select Visa Classification         1

Grouping data and then applying a function over them (say sum()) requires about 4-5 lines of code

pretty_print('Employers with most number of VISAS in any status')
    name_group = main_frame_chunks.groupby([main_frame_chunks [EMPLOYER_NAME], main_frame_chunks[STATUS]])
    name_copy_group = name_group[STATUS].count().copy()
    name_copy_group.sort()
    print(name_copy_group[-top:])

You can visualize the data using pandas (since it has the ability to plot graphs)  , but I chose to export the data to excel to make pie charts out of it. Here is some of the visualized data followed by the program’s full textual output

Most sought after VISAS

Popular job titles

Jobs with the highest pay across all companies

Companies with the most certified VISAs

Full text output

Finished reading data
Processing 168718 rows
Finished normalizing data
********************************************************
Most sought after VISAs
********************************************************
H-1B                          165099
E-3 Australian                  3248
H-1B1 Chile                      186
H-1B1 Singapore                  184
Select Visa Classification         1
********************************************************
Top ten job titles
********************************************************
PROGRAMMER ANALYST          13388
SOFTWARE ENGINEER            6069
COMPUTER PROGRAMMER          4035
SYSTEMS ANALYST              3651
BUSINESS ANALYST             1874
COMPUTER SYSTEMS ANALYST     1851
SENIOR CONSULTANT            1723
PHYSICAL THERAPIST           1676
SOFTWARE DEVELOPER           1495
SENIOR SOFTWARE ENGINEER     1382
********************************************************
Employers with most number of VISAS in any status
********************************************************
LCA_CASE_EMPLOYER_NAME             STATUS   
LARSEN & TOUBRO INFOTECH LIMITED   CERTIFIED     952
ERNST & YOUNG US LLP               CERTIFIED    1142
PATNI AMERICAS INC                 CERTIFIED    1161
ACCENTURE LLP                      CERTIFIED    1255
IBM INDIA PRIVATE LIMITED          CERTIFIED    1464
WIPRO LIMITED                      CERTIFIED    1940
MICROSOFT CORPORATION              CERTIFIED    2085
TATA CONSULTANCY SERVICES LIMITED  CERTIFIED    2134
DELOITTE CONSULTING LLP            CERTIFIED    2194
INFOSYS LIMITED                    CERTIFIED    5276
********************************************************
148021 VISAs are certified and 20697 are not
********************************************************
********************************************************
Employers with the highest salary budget
********************************************************
LCA_CASE_EMPLOYER_NAME
MICROSOFT CORPORATION                       2.239855e+08
BANYAN ENERGY, INC                          2.304000e+08
VIRGO INC                                   2.332100e+08
INTEL CORPORATION                           2.359001e+08
IGATE GLOBAL SOLUTIONS, AN IGATE COMPANY    2.730697e+08
AMAZON CORPORATE LLC                        4.185291e+08
NEWMONT INTERNATIONAL SERVICES LIMITED      4.237428e+08
GOOGLE INC                                  4.445123e+08
INFOSYS LIMITED                             6.228491e+08
QUALCOMM INCORPORATED                       7.204821e+08
Name: LCA_CASE_WAGE_RATE_FROM
********************************************************
The city offering the highest dough (summed over all positions)
********************************************************
LCA_CASE_EMPLOYER_CITY  LCA_CASE_EMPLOYER_STATE
IRVING                  TX                         4.118216e+08
GREENWOOD VILLAGE       CO                         4.368157e+08
SEATTLE                 WA                         4.586871e+08
BELLEVUE                WA                         4.989642e+08
MOUNTAIN VIEW           CA                         6.737251e+08
HOUSTON                 TX                         6.851985e+08
CHICAGO                 IL                         7.032560e+08
PLANO                   TX                         7.072252e+08
SAN DIEGO               CA                         7.694944e+08
NEW YORK                NY                         1.280386e+09
Name: LCA_CASE_WAGE_RATE_FROM
********************************************************
Jobs with the most dough (summed across offers from all employers)
********************************************************
LCA_CASE_JOB_TITLE
RESEARCH SCIENTIST II                                2.258712e+08
CONSULTANT                                           2.291776e+08
CHIEF EXECUTIVE OFFICER                              2.445069e+08
DATABASE ADMINISTRATOR                               2.625037e+08
COMPUTER SYSTEMS ANALYST                             3.641315e+08
REGIONAL DIRECTOR HEALTH SAFETY & LOSS PREVENTION    4.224000e+08
COMPUTER PROGRAMMER                                  4.587185e+08
SYSTEMS ANALYST                                      6.021489e+08
SOFTWARE ENGINEER                                    1.809624e+09
PROGRAMMER ANALYST                                   2.060111e+09
Name: LCA_CASE_WAGE_RATE_FROM
Done

 More on pandas

Pandas is a great library to perform quick analysis on data. The ease with which you can prune and analyze the data is impressive. Knowledge of the library and its internal structure is of course necessary to make the best use of it. If you’d like to know more about pandas, the following book is a great read.