Manoj Awasthi

Programmability in nginx config using map

2024-06-08T00:00:00+00:00

NGINX is one of the most beautiful piece of tech that I explored briefly in 2011-12 and heavily 2016 onwards while scaling an ecommerce platform at work in Indonesia. It is so simple and so powerful that it is hard to not fall in love with it. Primarily, it is an HTTP reverse proxy server and can be used to serve static and dynamic content behind web services with excellent inbuilt support for load balancing, caching, logging, rate limiting, and even request mirroring.

For a more elaborate understanding of the feature set it comes with, head to NGINX documentation.

Most of the features in NGINX are controlled through easily configurable NGINX configuration that usually sits at /etc/nginx/nginx.conf and extended by included files in the configuration (including /etc/nginx/<sites>/<site.example.com>). This ability to modify the conf file followed by a simple config test and reload as follows is what makes it so easy to operate:

$ nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
..
$ nginx -s reload

Don’t get disheartened.

This post is not a general manual for NGINX. It is about something I discovered today while experimenting on something for a friend. This person wants to run a simple web tool and serve it on different end points for different employees.

http://example.com/it/review-1
http://example.com/it/review-2
http://example.com/it/review-3

All the three sites are similar in nature. They have some logic to fetch some data and present on the web page in a form, which when submitted post review gets submitted to backend and redirected back to the original endpoint for fetching more data for review.

Initially they set up three endpoints in the config as following:

	location /it/review-1/ {
		proxy_pass http://localhost:5000/?param=1;
    	}
	location /it/review-2/ {
		proxy_pass http://localhost:5000/?param=2;
    	}
	location /it/review-3/ {
		proxy_pass http://localhost:5000/?param=3;
    	}

Now they want to expand it to 10 endpoints. So, one way was to simply proceed with duplicating the endpoints in NGINX config. So, he asked me if there is a more elegant solution to this. And as I researched I found about map in NGINX which I’d never used.

`map` in NGINX

map coupled with excellent support for regex in NGINX made it possible to avoid all duplication and enable a very elegant solution.

# Define this in http block but outside the server block
# Define a map to extract the 'param' parameter based on the URL path
map $request_uri $param {
	default 0; # Set default value to 0
	~^/it/review-(\d+)/ $1;
}

This assigns param variable to a default value of 0 and otherwise parse it from the request_uri. So a request to /it/review-2 will assign param a value of 2.

Next step is to use it for the location block:

        location ~ ^/it/review-(\d+)/ {
            # Extract the param parameter
            set $param_value $param;
	    
            # Use the odd parameter in the proxy_pass directive
            proxy_pass http://127.0.0.1:5000/?param=$param_value;
	}

But then I realised that this will lead to a situation where even unintended endpoints like beyond ten (10) will work e.g. /it/review-12. For handling that I needed to check for the value and return 501 (Not Implemented) for the requests of that nature.

So, the question was HOW!?

And again the superhero to the rescue is our map. Following code helped do the validation and set a variable for every request if the param value is greater than 10.

# Define a map to check if param_value is >= 10
map $param $param_ge_10 {
	default 0;
	~^[1-9]$ 0; # Param value is less than 10
	~^1[0-9]$ 1; # Param value is greater than or equal to 10
}

Finally, we can use this for additional logic in the location block:

        location ~ ^/it/review-(\d+)/ {
            # Extract the param parameter
            set $param_value $param;
	
            # Return 501 Not Implemented if param_value is >= 10
            if ($param_ge_10) {
                return 501;
            }
    
            # Use the odd parameter in the proxy_pass directive
            proxy_pass http://127.0.0.1:5000/?param=$param_value;
	}

And that was it.

Quite a learning, followed by the excitement to share it and hence this post!!

Probability of death by different causes

2022-08-04T00:00:00+00:00

The discussion about death is usually considered morbid and is avoided in most cultures that I’ve grown in despite the fact that it remains the most certain and inescapable event for all mortals.

While walking today, I thought I would go and read about the odds of dying from different causes. Looks like, lot of studies have happened in different parts of the world.

Following is a summary of some of the probabilities of deaths in United States per 2020:

Cause of Death	Probability	Odds (x in y)
Heart Disease	16%	1 in 6
Cancer	14.2%	1 in 7
COVID-19	8.3%	1 in 12
All preventable causes of death	4.7%	1 in 21
Chronic lower respiratory disease	3.6%	1 in 28
Opioid overdose	1.5%	1 in 67
Suicide	1%	1 in 93
Motor Vehicle Crash	0.99%	1 in 101
Fall	0.98%	1 in 102
Gun Assault	0.45%	1 in 221
Pedestrian Incident	0.18%	1 in 541
Motorcyclist	0.12%	1 in 799
Drowning	0.09%	1 in 1024
Fire or Smoke	0.07%	1 in 1450
Choking on food	0.04%	1 in 2745
Bicyclist	0.03%	1 in 3396
Sunstroke	0.015%	1 in 6368
Accidental gun discharge	0.012%	1 in 7998
Electrocution, radiation, extreme temperatures, and pressure	0.007%	1 in 14705
Sharp objects	0.004%	1 in 26744
Cataclysmic storm	0.003%	1 in 35074
Hot surfaces and substances	0.002%	1 in 50341
Hornet, wasp, and bee stings	0.0017%	1 in 57825
Dog attack	0.0014%	1 in 69016
Lightning	-	Too few deaths in 2020 to calculate odds
Railway passenger	-	Too few deaths in 2020 to calculate odds
Passenger on an airplane	-	Too few deaths in 2020 to calculate odds

Source: InjuryFacts

Just looking at different cause heads, I feel that these odds will look so different for developing nations.I would really want to find and prepare similar data for India (2020+) and possibly, Indonesia (2020+).

If you have come across any published data source related to this then please do help share on Twitter or Mail me.

Nuggets of wisdom from the book 'The Richest Man In Babylon'

2022-02-14T00:00:00+00:00

I finished reading this popular book The Richest Man In Babylon by George S. Clason and can testimony the amount of wisdom in the book around financial success.

Following are bunch of nuggets of wisdom from the book.

“Our acts can be no wiser than our thoughts. Our thinking can be no wiser than our understanding.”

“No one lends his entire fortune, not even to his best friend.”

” A fat purse quickly empties if there be no golden stream to refill it.”

“I wish an income that will keep flowing into my purse whether I sit on the wall or travel to far lands.”

“The reason why we have never found any measure of wealth? We never sought it.”

“Wealth is power. With wealth many things are possible”

“And when youth comes to age for advice he receives the wisdom of years”

“Opportunity is a haughty goddess who wastes no time with those who are unprepared”

“A part of all I earn is mine to keep”

“Then learn to make your treasure work for you. Make it your slave. Make its children and its children’s children work for you.”

“Invest thy treasure with great caution that it be not lost.”

“Provide also that thy family may not want should the Gods call thee to their realms. For such protection it is always possible to make provision with small payments at regular intervals”

“Counsel with wise men.”

“A small return and a safe one is far more desirable than risk”

“Enjoy life while you are here. Do not overstrain or try to save too much.”

“Life is good and life is rich with things worthwhile and things to enjoy”

“Just as the weeds grow in a field wherever the farmer leaves space for their roots, even so freely desires grown in men whenever there is a possibility for them to be gratified”

“No man’s family can fully enjoy life unless they do have a plot of ground wherein children can play in the clean earth and where the wife may raise not only blossoms but good rich herbs to feed her family.”

“To a man’s heart it brings gladness to eat the figs from his own trees and the grapes of his own vines.”

“Preceding accomplishments must be desire. The desire must be strong and definite.”

“More interest in my work, more concentration upon my task , more persistence in my effort and, behold few men could carve more tablets in a day than I.”

7 Curses for a lean purse

Start thy purse to fattening. Keep at least one-tenth of your earnings for yourself.
Control thy expenditures. Have a budget. Budget thy expenses.
Make thy gold multiply.
Guard thy treasures from loss. The first sound principle of investment is security for thy principal. Consult with wise men.
Own thy own home.
Provide in advance for the needs of thy growing age and the protection of thy family. Insure a future income (retirement and in case of fatalities).
Increase they ability to earn.

Such things as following, a man must do if he respects himself:

He must pay his debts with all the promptness within his power, not purchasing that for which he is unable to pay.
He must take care of his family that they may think and speak well of him.
He must make a will of record that, in case Gods call him, proper and honorable division of his property be accomplished.
He must have compassion and upon those who are smitten by misfortune and aid them within reasonable limits. He must do deeds of thoughtfulness to those dear to him.
He must cultivate himself, study and become wiser to become more skilful and respect him more.

“Opportunity stands for no man. Today it is here. Soon it is gone. Therefore, delay not!”

“Procrastinator is the one who accepts not the opportunity when she comes. He waits. He says I have much business right now. Bye and bye I talk to you. Opportunity, she will not wait for such slow fellow. She thinks if a man desires to be lucky he will step quick. Any man not step quick when opportunity comes, he big procrastinator.

“The spirit of procrastination is within all men. We desire riches; yet, how often when opportunity doth appear before us, that spirit of procrastination from within doth urge various delays in out acceptance”

“Recognize and conquer procrastination”

“No man can arrive at a full measure of success until he hath completely crushed the spirit of procrastination”

“To attract good luck to oneself, it is necessary to take advantage of opportunities”

“Men of action please the goddess of opportunity the best”

Five laws of Gold

Gold cometh gladly and in increasing quantity to any man who will put by not less than one-tenth of his earnings to create an estate for his future and that of his family.
Gold laboreth diligently and contentedly for the wise owner who finds for it profitable employment, multiplying even.
Gold clingeth to the protection of the cautious owner who invests it under the advise of men wise in its handling.
Gold slippeth away from the man who invests it in businesses or purposes with which he is not familiar.
Gold flees the man who would force it to impossible earnings or who followeth the alluring advice of tricksters and schemers or who trusts it to his own inexperience and romantic desires in investments.

“There is no chain of disasters that will not come to an end”

“Wealth that comes quickly goeth the same way”

“Wealth the stayeth to give enjoyment and satisfaction to its owner comes gradually, because it is a child born of knowledge and persistent purpose”

“What indiscretions hast thou done that thou shouldst seek the lender of gold? Hast been unlucky at the gaming table? Or hath some plump dame entangled thee?”

“Gold bringeth unto its possessor responsibility and a changed position with his fellow men. It bringeth fear lest he lose it or it be tricked away from him. It bringeth a feeling of power and ability to do good. Likewise, it bringeth opportunities whereby his very good intentions may bring him into difficulties”

“To borrowing and lending, there is more than the passing of gold from the hands of one to the hands of another”

“If you desire to help thy friend, do so in a way that will not bring thy friend’s burdens upon thyself.”.

“Help wisely!”

Where the gold can be lent:

The safest loans, my token box tells me, are to those whose possessions are of more value than the one they desire. They own lands, or jewels, or camels, or other thing which could be sold to repay the loan. Some of the tokens given to me are jewels of more value than the loan. Others are promises that if the loan be not repaid as agreed they will deliver to me certain property settlement. On loans like those I am assured that my gold will be returned with the rental thereon.
In another class are those who have the capacity to earn. They are such as you, who labor or server and are paid. They have income and if they are honest and suffer no misfortune, I know that they also can repay the gold I loan them and the rental to which I am entitled.
Others are those who have neither the property nor assured earning capacity. Life is hard and there will always be some who cannot adjust themselves to it. Alas for the loans I make them, even though they be no larger than a pence, my token box may censure me in the years to come unless they be guaranteed by good friends of the borrowers who know him honourable.

“If they borrow for purposes that bring money back to them, I find it so [borrowers repaying their loans]. But if they borrow because of their indiscretions, I warn thee to be cautious if thou wouldst ever have thy gold back in hand again.”

“Your purpose is wise and your ambitions are honorable. I shall be glad to lend you the fifety pieces of gold if you can give me security that they will be returned… I treasure much each piece of gold. Were the robbers to take it from you as you journeyed, or take the rugs from you as you returned then you would have no means of repaying me and my gold would be gone.”

“Gold, you see, Rodan, is the merchandise of the lender of money. It is easy to lend. If it is lent unwisely then it is difficult to get back. The wise lender wishes not the risk of the undertaking but the guarantee of safe repayment.”

“Better a little caution than a great regret”

“We cannot afford to be without adequate protection”

“Ill fortune pursues every man who thinks more of borrowing than repaying”

“Thy debts are thy enemies. They ran thee out of Babylon. You left them alone and they grew too strong for thee.”

“Where the determination is, the way can be found”

Plan for debt free life

First, one-tenths of all I earn shall be set aside as my own to keep.
Second, seven-tenths of all I earn shall be used to provide a home, clothes to wear, and food to eat with a bit extra to spend, that our lives be not lacking in pleasure and enjoyment.
Third, two-tenths of all I have earned shall be divided honorably and fairly among those who have trusted me and to whom I am indebted.

and the one which is my most favorite is in the last chapter, about ‘Work’:

“Some men hate it. They make it their enemy. Better to treat it like a friend, make thyself like it. Don’t mind because it is hard. If thou thinkest about what a good house thou build, then who cares if the beams are heavy and it is far from the well to carry the water for the plaster. Promise me, boy, if thou get a master, work for him as hard as thou canst. If he does not appreciate all thou do, never mind. Remember, work well-done, does good to the man who does it. It makes him a better man.”

A recommended read.

Solve Problems, Not Symptoms - An Interview with Naveen Jain

2021-05-24T00:00:00+00:00

Loved this talk.

A quick primer into isolation levels in databases

2020-08-22T00:00:00+00:00

Transaction Processing is one big source of complexity in correctly designing concurrent applications working with database systems.

A transaction is defined as one indivisible logical entity comprising usually of more than one operations which are meant to be considered a single step that either is success (a ‘commit’) or a failure (an ‘abort’). This unit of work should act as if the transaction (the group of operations) is atomic.

This is quite important to keep the state of database in a consistent state at all the time and being adherant with ACID semantics.

I will not go into the examples of transactions in databases assuming the familiarity of the reader to them. There are zillions of articles talking about concurrent processing of bank accounts and why it is so important to ensure atomic behavior.

In this article, we talk in the context of concurrent processing on the databases systems.

What is an isolation level?

An isolation level specifies ‘how’ and ‘when’ parts of the transaction can and should become visible to other concurrently executing transactions.

Isolation levels describe the degree to which transactions are isolated from other concurrently executing transactions, and what kind of anomalies can be expected in each.

Possible Read and Write anomalies specified by SQL standards

Read anomalies

A dirty read is a situation in which a transaction can read uncommitted changes from other transactions.
A nonrepeatable read is a situation in which a transaction queries the same row twice and gets different results. For example, a transaction T1 reads a row, then transaction T2 modifies the row and commits this change. Now, if T1 requests the row again then the result will be different from previous read.
A phantom read is a situation a transaction queries the same set of rows twice and receives different results. It is similar to nonrepeatable reads but only for range queries.

Write anomalies

A lost update happens when transaction T1 and T2 both attempt to update the value of V. T1 and T2 read the value of V. T1 updates V and commits and T2 updates V after that and commits as well. Since the transactions are not aware about each other so the result of T1’s update will be overwritten by T2’s update.
A dirty write is a situation in which one of the transactions takes an uncommitted value (i.e. a dirty read), modifies it and commits it.
A write skew is a situation when each individual transaction respects the required variants, but their combination does not satisfy these invariants.

Isolation Levels

The lowest isolation level (or the weakest) is Read Uncommitted which allows dirty-read.

The next isolation level is Read committed that allows transactions only to read committed changes. Althought this isolation level does not allow dirty reads but still suffers from nonrepeatable reads and phantom reads.

The next isolation level is Repeatable Read that does not allow dirty read and also takes care of nonrepeatable reads but suffers still from phantom reads.

The strongest isolation level is ofcourse Serializable where all transactions are executed serially. This has significant negative impact on performance and hence not a practical choice.

Summary

So, for most cases Repeatable Read would be the go-to isolation level to operate in. You can read more about Performing concurrency-safe DB updates in postgres using ‘repeatable read’ isolation level.

Happy and safe coding!

Time Complexity of Algorithms

2020-06-03T00:00:00+00:00

Assuming 100M ops per second on the machine.

Socratic questioning, a way towards first-principles thinking

2020-05-09T00:00:00+00:00

Recently, started reading The Great Mental Models: General Thinking Concepts by Shane Parrish and came across Socratic Questioning in the chapter on first principles thinking.

Anything that is not a law of nature is just a shared belief. Money is a shared belief. So is a border. So are bitcoin. So is love. The list goes on.

Socratic Questioning is a method that can help us reflect on our thinking process. This can help us from relying on our gut and limit strong emotional responses. We can cut through the fog of dogma and shared beliefs to identify unchanging principles.

Methods are many. Principles are few.

Socratic Questioning generally follows the following process:

CLARIFYING YOUR THINKING AND EXPLAINING THE ORIGINS OF YOUR IDEAS
- Why do I think this?
- What exactly do I think?
CHALLENGING ASSUMPTIONS
- How do I know if this is true?
- What if I thought the opposite?
LOOKING FOR EVIDENCE
- How can I back this up?
- What are the sources?
CONSIDERING ALTERNATE PERSPECTIVES
- What might others think?
- How do I know I am correct?
EXAMINING CONSEQUENCES AND IMPLICATIONS
- What if I am wrong?
- What are the consequences if I am wrong?
QUESTIONING THE ORIGINAL QUESTIONS
- Why did I think that?
- Was I correct?
- What conclusions can I draw from the reasoning process?

This process of reflection or an alternative of this called FIVE WHYS is useful in understanding deeply the root cause of what we observe or think. There is no doubt that both of these methods slow us down in the short term but reap huge rewards in terms of limiting emotional responses, better thought process and better decision making in long term.

Science is much more than a body of knowledge. It is a way of thinking. – Carl Sagan

How fast do you read?

2019-04-19T00:00:00+00:00

As someone who loves reading books as a hobby, I have always wondered if I’m able to read at a good enough speed or not. I’ve friends who claim to finish a 200 page book in 4 hours. In terms of words, assuming each line would be ~ 10 words and each page would be ~ 35-40 lines, it translates to ~350-400 words per minute.

With all the interruptions and distractions, I have never been able to sit for 4-5 hours in one go and finish a book (IIRC). This is equivalent of being able to finish a book every weekend (one of the days over the weekend rather) and 4-5 hours is something you can always trade with your family, spouse amidst all commitments of time to her and children. So, no doubt I envied the friend claiming this.

Today, came across following stats while browsing:

The average adult reading speed is between 200 and 300 words per minute (same reading speed you want to achieve by 6th grade).

For success in college you should be able to read 350 to 450 words per minute if you want to have any extra time.

Speeds above 575 words per minute are typical for successful high level executives and people who enjoy reading.

Reading in itself comprises of two things -

Speed (words per minutes)
Comprehension (how much do you understand and retain in short term, from page to page)

There are lot of good resources on measuring your current reading speed alongwith comprehension. This is one such site (no affiliation. feel free to google and find more options). My current reading speed is less than the average with a high comprehension.

Note that Comprehension is more important than the Speed in reading. So, always value the former.

But it kind of proves that the claim from the friend about 200 pages in 4 hours was not a puffery claim! :-)

Happy Reading!

Doing it alone

2019-01-04T00:00:00+00:00

Originally posted on 8 Sep’ 2010 here

“If you’re that rare engineer who’s an inventor and also an artist, I’m going to give you some advice that might be hard to take. That advice is: Work alone.”

Steve Wozniak

As far as I remember I was never given such advice by anyone, be it my parents, my teachers, professors or mentors during my professional career. It’s hard to understand and harder to believe some such thing when no one talked about it in such a large span (I am past quarter of my life span). But when I, for last few weeks, started to devote some time to this idea of working alone I could clearly think the cases where it was a pure win. These are the cases of “diffused responsibility”, “too much dependence” and “plain pretence”.

I remember working with a group in my last company where each of us in the group felt that the given system of diagnosing the issues was substandard and needed an overhaul. “We” were asked to do that task of making the diagnosis tool better but since it was “We” and apart from this work, everyone had there own set of tasks in their plate - it never happened. It happened when one of us took the responsibility of doing so (after being pointed to sadly). Everyone knew someone has to do it but did not take the task because they assumed that if it is important then someone will take care of it otherwise it’s not that important. I should also point out that everyone was competent enough to do the task. This was the case of “Diffused Responsibility”.

In my current work, for quite some time I was having a task of automating a stuff which was causing pain to us - a very very repetitive stuff which should have been automated long back (I believe this, being part of an engineering company, other companies - I am quite sure - look up to). I promised that I’ll do the task but then came a phase of planning - how to do it? (On this question I imagine the face and style of a a very senior hardcore geek I worked with in a previous company yelling at me - “stupid, how do you write scripts?”). Well, trust me I was asked to plan it out, I was asked to meet so that we use the same automation system which a larger activity uses. So there comes the whole gimmick of breaking the tasks in such a way that there are multiple tasks available with multiple people and yes, there must be a dependency so that there is one more guy involved who will look at getting it done. Hmm. Too much work at hand. The tasks planned moved but very slowly. Thanks to a guy in my team who “broke” the rule and deployed the simple script. That was the power of doing it alone. All the friction was gone from the same set of guys talking about planning and “N” factors to be taken in account. This was the case of “too much dependence”. If you get into such case, do the same - break the rule.

I also remember another incident that happened to me recently. I was working on a killer feature (a killer feature is not a feature that kills but a feature which is most talked about in your product) of the software product I work on as part of my day job. It was a team of three and we decided on few things. Then all of a sudden scope of the work to be done due to some reasons was decreased. We still thought it would be good to decide on few investigations just for Intellectual Property reasons. Here the third case of “Plain Pretence” came in. I did not do, I am pretty sure, because I thought someday I would sit with other guys and work on it. That did not happen. I am also sure that had I started working on these investigations alone, someone in my team would have joined for sure - these are guys who are undoubtedly competent and have keen interest in software development. That initiation was the key.

I’ve written only about the work life experiences but I think this applies to many non work experiences. For me most of the things that I started without worrying about whether someone will join in or not were marked done at some point of time - either successfully or unsuccessfully. They were not lying in on my table under the was_not_able_to_even_try folder. So I have started believing in starting alone. That should not be interpreted as “don’t do in a group” but as - if it is possible to do it alone then don’t wait for the group to begin - just begin and the group may join.

Larger dreams in life like starting a company, I’ve been told should not be done alone. They are such goals which require some one to be there to take care of each other in adverse situations and more reasons. Imagining this, I agree but since I am yet to taste actually that sweet feeling of starting an enterprise I won’t be able to comment on this but I feel this small piece of gyan will be helpful in lot of smaller activities in day to day life. A random post on a random thought.

How I learned hard way the pitfalls of a dynamically typed languages?

2018-08-07T00:00:00+00:00

Dynamically typed (weak types) languages (like Perl, Python, Ruby etc) are great for developer productivity or so they say. I agree to large extent but then for a large scale system with growing components and services this may become a nightmare to maintain. This is based on some of the learnings being on both sides.

In this post, want to discuss one interesting issue that I got reminded of while going through some archives and boils down to one of the perils of using a dynamically type language.

For the context, we are a marketplace with millions of sellers and buyers on the platform. Sellers can open a shop on the platform and start selling while buyers can discover the products and shops they want to get access to. Marketplace also provides many payment methods and escrow so that your payment is safe.

Marketplace platform was originally written in Perl and though we modularised it from Monolith that it was to a huge number of micro services while taking the opportunity to write them in a modern statically typed language (golang), we still had some parts of it written in Perl. We have been moving these parts to golang fast.

So one fine day last year, suddenly we started receiving an ever increasing number of merchant complaints that they were getting an error while adding products (hence failed uploads of products) which resolved to the following message:

"category not complete"

Yes, not that useful but this was not supposed to occur so much, a highly unlikely message and hence never got much attention from the product managers.

At the peak of the error, the frequency of reported issues was ~ a couple of thousands per hour.

After assembling multiple teams involved and tracking the root cause, we found that the bug itself was pretty interesting.

There was a sql query to check if a category is leaf or not - try finding out categories whose parent is the given category and see the number of results returned - if zero then ok other wise throw error. It looked something like this in the perl code:

                     $str_sql = qq~
                         select
                             d_id
                         from ws_department
                         where parent = <:n=$val_dep:>
                         and status = <:n=1:>
                     ~;

$val_dep i.e. category ID was fetched in a previous code block using another SELECT query and was expected to be a single integer (equivalent of LIMIT 1). But somehow $val_dep, instead of being the category_id (i.e. int) ended up being the array (multiple values) whose address would look like 0x245bf20. then this function

$svq->sql($str_sql);

will convert the query into

                     $str_sql = qq~
                         select
                             d_id
                         from ws_department
                         where parent = 24520 
                         and status = <:n=1:>
                     ~;

So, note how Perl showed its intelligence in converting the 0x245bf20 to 24520 (silently). Grr.

This caused the intermittently occurring bug. Sometimes that memory address would resolve to a valid category ID in the marketplace and many times it won’t. When it doesn’t it leads to the error that leaf category ID is not present which in turn led to the “bad” error message “category not complete”.

Side effect of dynamically typed language.

Statically typed languages (C, C++, Java, Scala, Go etc) ensure type safety and would help avoid such pitfalls.

So be mindful about using the language carefully. If it is dynamically typed then you can expect no type safety. So ensure proper validation (err towards more of it) to write rock solid code (if there is anything like that).

:-)

Undoing git reset hard (remove local commits)

2018-04-01T00:00:00+00:00

Although this post is being published on 01 April celebrated as April Fool’s Day and known for pranks all around, you can trust this post.

This was a long weekend and I sat down to code something that I planned for quite sometime but didn’t get time to. After finishing with the coding and testing I prepared the git repo and pushed the code in there.

$ git add . 
$ git commit -m "Initial commit." 
... 
$ git push origin master 
...

This was taking long time and I realized, inadvertently, I committed some binary embedded databases (big size for git). That is when I thought of quickly removing the commit and re-commit again:

$ git reset --hard HEAD~1

As a side effect, all my local files were gone too. Yes, all work that went into over the weekend. :-(

For sometime I was shaken but then a quick search over google led to this QnA on stackoverflow. This helped! :-)

Following taken verbatim from the above QnA post summarizes what to do:

$ git init
Initialized empty Git repository in .git/

$ echo "testing reset" > file1
$ git add file1
$ git commit -m 'added file1'
Created initial commit 1a75c1d: added file1
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 file1

$ echo "added new file" > file2
$ git add file2
$ git commit -m 'added file2'
Created commit f6e5064: added file2
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 file2

$ git reset --hard HEAD^
HEAD is now at 1a75c1d... added file1

$ cat file2
cat: file2: No such file or directory

$ git reflog
1a75c1d... HEAD@{0}: reset --hard HEAD^: updating HEAD
f6e5064... HEAD@{1}: commit: added file2

$ git reset --hard f6e5064
HEAD is now at f6e5064... added file2

$ cat file2
added new file

Particularly:

$ git reflog
1a75c1d... HEAD@{0}: reset --hard HEAD^: updating HEAD
f6e5064... HEAD@{1}: commit: added file2

$ git reset --hard f6e5064
HEAD is now at f6e5064... added file2

Life saver! :-)

Problems using Hindi variables in golang

2017-11-12T00:00:00+00:00

Go is a beautiful programming language. It is not so due to its many features but rather due to lack of them.

It has managed to remain simple, performant, backward compatible (it’s a promise by creators, think python 2.7 vs 3.0 tussle) and has some unique features (compared to popular modern programming languages) like the support for concurrent programming (channels, goroutines).

Apart from the main features of the language, there are some cute little things which go provides. One of them is that your variable names can be composed of any unicode characters. What does that mean? This means, following program containing simplified chinese characters is a valid program which compiles and runs fine (you can try):

import (
    "fmt"
)

func main() {
    量 := 5000    // Amount
    速率 := 10    // Rate
    持续时间 := 6 // Duration

    总 := (量 * 速率 * 持续时间) / 100 // Total

    fmt.Println(总)
}

Now, that is some fun. I thought why not try my mothertongue, Hindi (Devanagri script), so coded following:

func main() {
    मूलधन := 5000
    दर := 10
    समय := 6 // वर्षमें

    fmt.Println((मूलधन * दर * समय)/100)
}

On compiling:

Manojs-MacBook-Pro:test2 manojawasthi$ go build
# _/Users/manojawasthi/gocode/src/github.com/awmanoj/test2
./main.go:8: invalid identifier character U+0942 'ू'
./main.go:12: invalid identifier character U+0942 'ू'

Heartbroken!

On further search on the error and the issue, I found that I wasn’t alone (ofcourse).

This was also discussed in golang-nuts and Nathan Kerr added a response:

The Go team knows this a problem (see the issues listed by Rob Pike). The difficultly in fixing it is that the ecosystem surrounding Go relies on the current definition for identifiers. There is no way to add the characters you need without breaking other tools like syntax highlighters. It is a socioeconomic problem than a technical one. The next time it can be fixed in Go is for Go 2.0 and is already on the list of issues to be considered. If your main goal is to write programs in your language, you might consider using one from https://en.wikipedia.org/wiki/Non-English-based_programming_languages

There are also some good comments by Bakul on this thread. He discusses briefly the intricacies involved and why this would be rather low priority item even for Go 2.0 to support Hindi (while it is in plan). I concur.

For the curious, these are some of the issues related to this (latter added by Rob Pike):

https://github.com/golang/go/issues/5167 https://github.com/golang/go/issues/20706

So, while “not holding my breath”, will still be looking out for Go2.0 to support hindi names (Devanagri script).

Note: Without the problematic characters, things worked well.

How to get approximate row count in postgres tables?

2017-08-31T00:00:00+00:00

If there is a table with tens of millions of rows in a postgres table then following query will be very slow (non interactive):

SELECT COUNT(*) FROM SAMPLE

Following can help us get an “approximate” count of rows in postgres:

SAMPLEDB=> SELECT reltuples::BIGINT AS estimate FROM pg_class WHERE relname = 'SAMPLE'; 
 estimate 
----------
 54296044
(1 row)

Ref: https://wiki.postgresql.org/wiki/Count_estimate

So how slow can golang regular expressions be?

2017-08-23T00:00:00+00:00

Sometime back, I wrote on slow regular expressions in golang. Recently, I thought about experimenting with regular expressions and use Golang benchmarking to estimate their slowness.

Disclaimers:

It’s not a generalisation (please read the previous post to understand the context of the claim).
This post is NOT a comparison between golang regex performance and PCRE (or other language) performance.
This post is meant only for a possible alternative in some of the cases where speed is important and less comprehensive checks still acceptable.

Experiment

For the experiment, at hand was a problem to identify whether the connection string to redis followed a certain pattern.

// Check if connectionString is host:port format and 
// not the tcp://user:pass@host:port/0?q=1 format.
//  Dial(&DialConfig{"tcp", "127.0.0.1:6379", 0, "", 10*time.Second, 10})
//  DialURL("tcp://auth:password@127.0.0.1:6379/0?timeout=10s&maxidle=1")

Problem is solvable using following regular expression:

var s2 = "127.0.0.1:6379”
func IsRedisConnectionStringType2Fn1() {
   regexp.MatchString("^(\\d+\\.\\d+\\.\\d+\\.\\d+):(\\d+)$", s2)
}

To compare if there will be a gain by replacing this with “less intricate”, “less powerful” but possibly “enough” heuristics based check, wrote this function:

func IsRedisConnectionStringType2Fn2(connectionString string) bool {
   // if starts with tcp:// then not a type2 URL
   if strings.Index(connectionString, "tcp://") == 0 {
       return false
   }

   // if more than two components separated by ':' then not a type2 URL
   splits := strings.Split(connectionString, ":")
   if len(splits) != 2 {
       return false
   }

   return true
}

We use golang benchmarking support (such a beautiful language!) to benchmark both versions:

import "testing"

func BenchmarkIsRedisConnectionStringType2Fn1(b *testing.B) {
    for n := 0; n < b.N; n++ {
        IsRedisConnectionStringType1()
    }
}

func BenchmarkIsRedisConnectionStringType2Fn2(b *testing.B) {
    for n := 0; n < b.N; n++ {
        IsRedisConnectionStringType2(s2)
    }
}

The results:

parallels@ubuntu:~/gocode/src/github.com/awmanoj/test2/test$ go test -bench=.
BenchmarkIsRedisConnectionStringType2Fn1-2    100000	    15139 ns/op
BenchmarkIsRedisConnectionStringType2Fn2-2   5000000	      378 ns/op
PASS
ok  	github.com/tokopedia/test2/test	3.968s

Here, strings based heuristics checks is 40x faster than the regex version. Is it equivalent (apple to apple)? not at all.

Should we fret over this?

Not necessarily — in fact most probably, need not.

If your code is not hit that often it doesn’t make sense to fret over something which is still imperceptible (ns per operation here) in terms of runtime. But if it is a code which is expected to be hit very frequently (e.g. for every hit to the server) — you should evaluate if there could be a less stringent and complex alternative.

Is it equivalent? Not at all. In all probability — this heuristics based check will not be as powerful and comprehensive as the regular expression check so it is not always possible to replace the regex with such heuristics check. But when possible it can be a good replacement.

Advice

Though there is good advice hidden in this post which can be useful at times, always remember this nugget:

We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. – Donald Knuth

Using nginx logs to identify SSL certificate details

2017-06-13T00:00:00+00:00

Adding client side SSL certificate is one of the best ways to restrict access on public internet. The way it works is that you create server certificates using a certifying authority (which can be you, yourself) and then create some client certificates. Then it is a matter of putting some settings in nginx (or other server you may be using) to enable it. Post that, you share the client certificate with users who are authorized to access the resource with revocation possible from server side. Your resource can be accessed by only people who have the client certificate installed. You can read this and this for details of how to generate and use certificates for nginx.

At my workplace, we have a distributed CRUD system for key values which is protected using client side SSL certificates in similar manner. Initially, we created a common client certificate which was shared to all who needed access. But in this setup, there was a loophole. Since the certificate is shared so whether user A makes the change or user B makes the change, there is no way of identifying. So, we came up with per-user certificates which will have the user id and user email identification. These certificates will be delivered to user’s email on request for access.

Next step (and the subject of this blog post) was to identify through logs who does what actions. On Nginx, it is as simple as updating the log_format in /etc/nginx/nginx.conf file:

  log_format  main '$remote_addr - $ssl_client_s_dn - $remote_user [$time_local] "$request" '
                '$status $body_bytes_sent "$http_referer" '
                '"$http_user_agent" "$upstream_response_time"';
  access_log /var/log/nginx/access.log main;
  error_log /var/log/nginx/error.log;

You notice that I added $ssl_client_s_dn in the log format, This variable returns the “subject DN” string of the client certificate for an established SSL connection according to RFC 2253 (1.11.6);

Following logs are examples of what we see in our logs:

10.0.2.31 - /C=ID/ST=DKI-Jakarta/L=Jakarta/O=PT XXXXXXX/OU=Engineering/CN=XXXX/emailAddress=manoj.awasthi@XXXXXXX.com - - [13/Jun/2017:18:06:38 +0700] "DELETE /services/kv/random/key HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36" "0.001"

The value corresponding to the variable $ssl_client_s_dn is:

/C=ID/ST=DKI-Jakarta/L=Jakarta/O=PT XXXXXXX/OU=Engineering/CN=XXXX/emailAddress=manoj.awasthi@XXXXXXX.com

This uniquely identifies the user who does a certain change e.g. DELETE a key. A simple, low cost change control system. :-)

You can read more about other variables related to SSL certificates which are supported in nginx.

Why do (some) engineers write bad code?

2017-06-07T00:00:00+00:00

I’ve been reading System Performance Tuning, 2nd Edition recently. Though the content is a bit dated as a reference (was written while Linux kernel 2.4 was coming up) but this is still a fantastic book on refreshing the workings of the components of a computer and knowing the tweaks for solaris and linux.

This post though, is focussed on a topic that is touched towards the end of the book. Why do we end up writing bad code?

Bad code could mean many things - bad design or architecture decision, badly written code that is harder to maintain (duplication, tightly coupled components, less modular etc.), code that is bad at performance (time or space) and so on. The book talks about following reasons for badly written code (most of it is verbatim with few modifications here and there):

Writing bad code is much faster than writing good code. Good software engineering practices often trade an increase in initial development time for a decrease in the time required to maintain the software by eliminating bugs in the design phase or for a decrease in application runtime by integrating performance analysis into the development process. When faced with a deadline and irate management, it’s often easier to just get it done rather than do it well.

TIP: Learn to say No to “can we do this quickly” in favor of better thought of and more solid code or design. You will thank yourself for it more often than not.

Implementing good algorithms is technically challenging. This coupled with management’s desire for a product yesterday and the time to implement such an algorithm, often the good algorithms lose out.

TIP: Invest in learning about common algorithms & data structures and weigh upon choices when you’re faced with a problem. Almost always the first solution that strikes one’s mind is not the best.

Good software engineers are difficult to find. Software engineering is more than writing good source code. It involves architecture development, soliciting support from other development organisations, formal code review, ongoing documentation, version control etc. Concepts like pair programming are just beginning to move from the academic world into the mainstream and aren’t yet heavily practised.
Legacy is hard to maintain. It’s quite rare that a group of engineers sit down to develop a piece of software from scratch. More likely, they’re adding onto an existing framework, or developing the next version of the current product. The mistakes have already been made, and it’s not time effective (or cost effective) to throw everything and start over. Solution can be refactoring which makes the existing code more amicable to catching bugs, bottlenecks and improvements.

Also, experience wise I find that, bad code gets copied more than good code (remember ‘it is implemented the same way in that file / module / project etc.’? at the workplace). While re-writes are hard, it’s always possible to refactor the code as and when possible to make it more friendlier to improvements.

TIP: The basic idea here is whenever you write a line of code, it should be refactored. If you’re rewriting pieces of legacy code, those should be refactored as well.

Good night!

Writing good error messages (for developers)

2017-03-12T00:00:00+00:00

Writing good software is an Art, albeit the one which can definitely be learnt. As a developer, most of your time would go in maintaining an application - enhancing it with new features, finding bugs, debugging them to isolate the cause and fixing the issues. This post is mostly associated with the debugging of issues.

Writing good error messages should not be an Art. (this is also the goal of this post!)

Following are some general comments and suggestions on writing “good” error messages:

(this is by no means a comprehensive document, neither authoritative - just based on my experience)

Errors should explain the problem concisely and well

Good error messages provide the proper error message that explain the reason well. Easiest way to do that is to include the error code (e.g. errno in C or err.Error() in golang) in the error message.

Example:

Instead of:

05/Mar/2017:12:08:54 +0700 err database error.

Use:

05/Mar/2017:12:08:54 +0700 err error opening database postgres://a:b@host/maindb

Errors should provide immediate context

Good error messages provide details of immediate context. In all likelihood, only one error scenario will hit at a time. So each log (however unimportant it may seem) should contain the context e.g. immediate variables related to error condition, exact error code, anything else that may be important etc.

TIP: while writing code, imagine if this error is hit then what information would you be immediately seeking without attaching debugger. That information should be logged.

Example:

Instead of:

05/Mar/2017:12:08:54 +0700 err parsing integer

Use:

05/Mar/2017:12:08:54 +0700 err parsing integer: "248BAD"

Errors must be actionable

Don’t log an error if it is not actionable. As an example, if you read a record from a database and you get an error that “the record is not present” - if you can proceed without it - for example, you insert the record then do not log this as an error. If you want to count from your logs number of inserts add an “info” log on inserts or may be a “warn” log on error.

Judge: if you think you don’t need to act on an error in error log - either make it an info or warn or don’t log it.

Errors must be logged in single line

Don’t use multiline errors. Error logs should contain one (note, one!) error per line. This is to be able to fully use UNIX commands over streaming (pipe).

Errors should be logged only at top level

Good errors logs are logged at the highest level and hence there is usually only ONE error per occurrence. E.g.

func DoStuff() {
	… 
	if err := DoStuffHelper(); err != nil {
		// log the error since function itself not returning error
		log.Println(“err”, err)  
		return
	}
	...
} 

func DoStuffHelper() error {
	… 
	if err := Stuff(); err != nil {
		// do not log the error since function returning an error. 
		// responsibility of logging error is of the caller
		return err  
	}
	...
}

Errors should roll up in function calls

If there are functions calling other functions and failure happens deep below then it is OK to add some additional error message explanation at each function level (don’t log though - see above - just return more detailed error at each level). It is advisable that in such rollup we retain original error code:

func DoA() {
	if err := DoB(); err != nil {
		log.Println("err", err)
		return
	}
}

func DoB() error {
	if err := DoC(); err != nil {
		return errors.New("failed executing C(), " + err.Error())
	}
}

func DoC() error {
	if err := DoD(); err != nil {
		return errors.New("failed executing D(), " + err.Error())
	}
}

...

Conclusion

Your error log is your friend in bad times - often one of the darkest hours of your work life (problem in production, panic, downtime etc) so an investment in thinking through and writing error logs well will reap significant rewards in saved time and faster resolutions.

If you have more suggestions do tweet to me.

421 Misdirected request error

2017-02-05T00:00:00+00:00

Yesterday, I faced an interesting issue that I wasn’t aware of earlier. Let me tell the context first.

We have one server (i.e. 1 IP) which serves multiple domains (websites). Yesterday, we decided to put it behind client side SSL certificates. Everything was hunky dory till we started hearing that some users (thankfully, this was internal website so only colleagues!) complained about 421 Misdirected Request when they try accessing those websites.

A couple of searches over the internet helped understand the problem a bit. This post is a summary.

So, basically when you enable SSL (e.g. set ssl_verify_client to optional, on or optional_no_ca in nginx) and you serve more than one domains from the same server (i.e. 1 IP address) and one of the domains are using http2 then HTTP/2 tries to reuse connections to different servers.

Let’s first understand what protocol does and how servers and clients are expected to behave with respect to this. Following are snippets from RFC 7540 (HTTP/2):

Protocol suggests:

An origin server might offer a certificate with multiple “subjectAltName” attributes or names with wildcards, one of which is valid for the authority in the URI. For example, a certificate with a “subjectAltName” of “*.example.com” might permit the use of the same connection for requests to URIs starting with “https://a.example.com/” and “https://b.example.com/”. … In some deployments, reusing a connection for multiple origins can result in requests being directed to the wrong origin server. … This means that it is possible for clients to send confidential information to servers that might not be the intended target for the request, even though the server is otherwise authoritative.

So servers are suggested to implement:

A server that does not wish clients to reuse connections can indicate that it is not authoritative for a request by sending a 421 (Misdirected Request) status code in response to the request (see Section 9.1.2).

.. and browsers (clients) are suggested to implement:

Clients receiving a 421 (Misdirected Request) response from a server MAY retry the request – whether the request method is idempotent or not – over a different connection.

In our case, we serve two domains a.foobar.com (over http) and b.foobar.com (over http2) from 1 IP address. We added SSL to this. So when a user:

opens browser and accesses a.foobar.com, he gets served the web content successfully.
from the same browser, accesses b.foobar.com. he gets a 421 Misdirected Request error in chrome (specifically, Google chrome 56.0.2924.87).

What happened?

Seems like, in second case - HTTP2 tries to use the same connection as the previous - owing to a single IP Address. In case of TLS, it additionally depends on whether certificate is valid for the host of the domain being accessed. As mentioned in RFC above the behavior is left to servers. If server finds this TLS request on a different domain than the one for which the connection was used previously and it thinks it is not authoritative then it can indicate so by responding with 421 misdirected request error. In response to this error code, the client (i.e. the browser) should create a new connection and retry the request.

So, while some browsers like Safari, Mozilla firefox behave in the expected manner causing no trouble, Google Chrome (particular version is the only one I tested with) doesn’t abide with this.

Solution:

For now, I disabled HTTP/2 for the site since it was internal website and it is ok to miss out on the performance benefits of http2. Everything works fine.
One post suggests it happens due to difference in SSL configurations on the two domains - this is what I am yet to investigate. I will keep the post updated.

Useful links:

From books to e-books

2017-01-29T00:00:00+00:00

I don’t remember distinctly when I got in love to reading books but this goes back to early age. It was self induced somehow. With time, the kind of books I read changed drastically which is expected. As a book lover, there is one constant challenge I faced - becoming more of a book collector than reader. There are increasing number of books that I have on book shelf which are unread. This causes not only an internal distress but also becomes one of the many reasons of complaints my wife has. She always imagines how this space, devoted to almost three books shelves, could be better utilized by other stuff (I don’t understand how would that be “better” though!). As you guess, she is not into reading that much.

Well, this post is about a transformation that I see. I purchased a kindle (the older small one, without touch) I think in year 2012-13. Even before that, being a programmer I spend more of my time on a laptop or desktop and would read e-books in the form of PDF or EPUB on the system. Kindle was received by family (especially, wife) with joy since I would purchase less of physical books (paperback, hardcover) and atleast going forward, there won’t be more space needed by those little things.

But after a small period of romanticism with kindle (purchasing few books, flaunting how easy it is on eyes as compared to reading a PDF on laptop, talking technicals like glare etc.), I felt something missing. The joy of reading was not really the same holding an electronic device and pressing buttons to go forth and back as with a book - unnatural - isn’t it? So, the kindle got placed on the same bookshelf meant for books and started gathering dust while I switched to books again. I also realized that this feeling was not unique to me - many friends would share similar thoughts on social media. So atleast the guilt of spending money and not using something was mitigated - there were others in the same situation like me and they too took similar decision.

Last year, I relocated to Jakarta. This meant I would not be able to get access to huge “unread” collection of books back home. I’d to resort to ebooks. By this time (early 2016), there were much better kindle apps with lot of features on Android and Mac. For this one year I read many books on kindle on Mac and I feel that I am already making that transition. Following are a couple of points which I like about kindle books as compared to physical counterparts:

	Digital book (e.g. on Kindle Reader)	Physical book
Fulfillment	delivered instantly after purchase	can take anytime from 2-3 days to weeks sometimes
Storage space	stored on device. can store thousands (and more) of books per the device memory	needs physical space
Carry while travel or during relocation	only space and weight is that of device	each book takes space and adds to weight
Highlights	highlight electronically, also know how many others have highlighted, see all highlights together etc.	highlight using marker or pencil, harder to go through highlights later
Translate meaning	check meanings of certain words and phrase online immediately (just select the word)*	hmm, keep a dictionary alongside (and switch context)
Distractions	possible to distract to internet, social media since you’re on the device	focus is better in this case
Feel		smell of pages of a new book, cozy feeling of turning pages of a physical book

So other than the last two points, I think there are many benefits of switching to a digital book on a good reader.

*This becomes particularly important since I come from India and English is not my first language. Additionally, I’m particularly weaker on Vocabulary. Handy access to exact detailed meaning of a word makes me understand the stuff more clearly and also helps building a better vocabulary.

Keep-Alive in http requests in golang

2016-12-16T00:00:00+00:00

The three-way handshake in a TCP connection setup is heavy wrt performance and re-using the already created connections is a considerable optimization. Keep alive is a method to allow the same tcp connection for HTTP conversation instead of opening a new one with each new request. Most servers and clients allow for configurations and options to utilise this optimisation.

I had a strange encounter with the way this is handled in go.

I’ve a go application - an nsq (a distributed message queue) consumer - which reads some data off nsq and send that data over a HTTP POST to a remote server over SSL. This involves a TLS handshake which is even more heavy than non-ssl handshake.

We found that nsq queue depth for the topics was high and increasing, additionally we kept seeing errors like in nsqd logs:

[nsqd] 2016/12/16 17:48:34.563958 ERROR: [19.68.8.7:55239] - E_FIN_FAILED FIN 0b3caf88 failed ID not in flight - ID not in flight
[nsqd] 2016/12/16 17:48:34.567675 ERROR: [19.68.8.7:55239] - E_FIN_FAILED FIN 0b3ca9f4 failed ID not in flight - ID not in flight
[nsqd] 2016/12/16 17:48:34.571008 ERROR: [19.68.8.7:55239] - E_FIN_FAILED FIN 0b3c955c failed ID not in flight - ID not in flight
[nsqd] 2016/12/16 17:48:34.573917 ERROR: [19.68.8.7:55239] - E_FIN_FAILED FIN 0b3c86f8 failed ID not in flight - ID not in flight

Based on the limited number of documents we got over the internet it seemed like this was due to a slow consumer (hints like need for heartbeat from consumer pointed in this direction). Since consumer was very simple - just doing some http posts so it was the only suspect.

We did a strace on the running process to see the read,connect system calls. It indeed was doing some certificate interchange (read) for each connect or POST:

[pid 18043] read(90, "g\201\f\1\2\0020|\6\10+\6\1\5\5\7\1\1\4p0n0$\6\10+\6\1\5\5\0070\1\206\30http://ocsp.digicert.com0F\6\10+\6\1\5\5\0070\2\206:http://cacerts.digicert.com/DigiCertSHA2SecureServerCA.crt0\f\6\3U\35\23\1\1\377\4\0020\0000\r\6\t*\206H\206\367\r\1\1\v\5\0\3\202\1\1\0\222\327\361\374\211T\330\363\305~8Z\270\210\313<Z\336o\274?\366\34\252\206Q\332\0044\257'\3751\205\273sa\200\251\325\224\267\\*\221\1/Ws\246\351bl=\330q?\200\256f\21\257\331\246\242w\313\202\245\361\241a?\2\345<a\35\313n\276\220\217k\377C\335}\235\2000+%?… ", 3166) = 1948

From the code, we noticed that:

a http client was being created for each incoming message in the handler. this was overkill clearly.
the http client created was using default transport. default http client does have a keep-alive setting.

Since keep-alive is already there so we thought #1 is the problem which is causing new connections to be created for every request which in turn leads to the TLS handshake (and eventually to slow consumer). We fixed this by creating a global client:

var client *http.Client

func init() {
   tr := &http.Transport{
       MaxIdleConnsPerHost: 1024,
       TLSHandshakeTimeout: 0 * time.Second,
   }
   client = &http.Client{Transport: tr}
}

and using this to do the Post:

   resp, err := client.Post("https://api.some-web.com/v2/events", "application/json", bytes.NewBuffer(eventJson))
   if err != nil {
       log.Println("err", err)
       return err
   }

   defer resp.Body.Close()

But we observed that problem persisted still (slowness as evident from errors in nsqd logs). We confirmed by doing strace again:

$ sudo strace -s 2000 -f -p 18120 -e 'read,connect' 
[pid 18120] read(90, "g\201\f\1\2\0020|\6\10+\6\1\5\5\7\1\1\4p0n0$\6\10+\6\1\5\5\0070\1\206\30http://ocsp.digicert.com0F\6\10+\6\1\5\5\0070\2\206:http://cacerts.digicert.com/DigiCertSHA2SecureServerCA.crt0\f\6\3U\35\23\1\1\377\4\0020\0000\r\6\t*\206H\206\367\r\1\1\v\5\0\3\202\1\1\0\222\327\361\374\211T\330\363\305~8Z\270\210\313<Z\336o\274?\366\34\252\206Q\332\0044\257'\3751\205\273sa\200\251\325\224\267\\*\221\1/Ws\246\351bl=\330q?\200\256f\21\257\331\246\242w\313\202\245\361\241a?\2\345<a\35\313n\276\220\217k\377C\335}\235\2000+%?… ", 3166) = 1948

This appeared for each connection request. This meant that golang was somehow not honouring the keep-alive. This is when this thread on stack overflow helped us -

You should ensure that you read until the response is complete before calling Close().

res, _ := client.Do(req)
io.Copy(ioutil.Discard, res.Body)
res.Body.Close()

To ensure http.Client connection reuse be sure to do two things:

Read until Response is complete (i.e. ioutil.ReadAll(rep.Body))
Call Body.Close()

So we followed the suggestion:

   resp, err := client.Post("https://api.some-web.com/v2/events", "application/json", bytes.NewBuffer(eventJson))
   if err != nil {
       log.Println("err", err)
       return defaultErrStatus, err
   }

   io.Copy(ioutil.Discard, resp.Body)   // <= NOTE 

   defer resp.Body.Close()

While, as a I see the golang implementation, it does include a subtle comment in the code but I don’t understand yet why this idiosyncrasy exists. But would be something to be aware of and keep on the back of mind.

 // Post issues a POST to the specified URL.
 //
 // Caller should close resp.Body when done reading from it.
 //
 // If the provided body is an io.Closer, it is closed after the
 // request.
 //

Take Away

Use a global client as much as possible to avoid new connections.
Even when you don’t need the response back, read it fully before closing the response body - to take benefit of persistent connections (keep-alive).

Simulation of chaos game in go

2016-12-12T00:00:00+00:00

Fractals are interesting. As happens with most things interesting in life I got introduced to fractals serendipitously while researching on another mathematical construct ‘golden ratio’ which i needed to know about for implementation in an image editing feature I worked on as part of my day job few years back.

Fractals are self repeating patterns (either exact or approximately) and are found in nature aplenty (mostly the latter kind). One of the most intriguing ones are Mandelbrot sets which have a mathematical connotation i.e. you can write a mathematical formula to describe mandelbrot sets - the fact that it involves complex numbers and the fact that this is found in nature is what makes them particularly interesting as fractals. This is when you start wondering if God is mathematician or if we discover mathematics rather than invent it. More on that later

After reading through some of the articles on fractals online, I started reading a book on fractals recently. The first fractal it talks about is sierpinski triangle. So one sunday I spent implementing a simulation in go.

You can check the simulation in the form of GIF at this page: https://awmanoj.github.io/public/sierpinski.html

The code for the service generating the GIF is available at: https://github.com/awmanoj/chaos

If you have questions you can reach out to me.

Performing concurrency-safe DB updates in postgres 9.4+ using 'repeatable read' isolation level

2016-11-30T00:00:00+00:00

Let’s accept it for the nth time again, writing concurrent programs correctly is hard.

In this post, I discuss about a specific problem while concurrently writing to databases that I encountered recently, how to fix it and I am using that encounter to discuss a general idea about concurrent updates in databases.

Concurrent - What’s that?

It’s sometime good to understand the antithesis to understand something. The opposite of concurrent would be sequential. A sequential program is a program with a single thread of execution so the order of events (instructions) executed in the program are predictable and are as per the program logic. A concurrent program is a program with more than one threads of execution and so the order of events (instructions) executed in the program are not really predictable — order of events within one thread remains predictable but whether an event x in thread A will happen before or after an event y in thread B or they will be simultaneous is generally unknown.

So a good “informal” definition (quoting from “The Go Programming Language”):

When we cannot confidently say that one event happens before the other then the events x and y are concurrent.

Concurrency-safe programs

A function is concurrency-safe if it continues to work correctly even when called concurrently (that is from two or more threads of execution without needing any additional synchronisation code).

The analogy can be extended to “programs” when programs are accessing a shared resource for writing e.g. databases. If two or more than two programs are executing concurrently (on one or more than one servers) and are accessing a shared resource for writing then too the definition holds true that programs are concurrency-safe if they keep performing their updates correctly. This is the topic of this post.

Example program

Let’s create a toy program for demonstration which does some updates to a local postgres instance. I will be using golang for implementing this but idea is language agnostic.

We will use a test table in a test database (both table and database are aptly named ‘test’). Following is the schema of the table:

test=# \d+ test
                         Table "public.test"
 Column |  Type   | Modifiers | Storage | Stats target | Description 
--------+---------+-----------+---------+--------------+-------------
 id     | integer |           | plain   |              | 
 value  | integer |           | plain   |              | 

the example code:

package main

import (
  "database/sql"
  "github.com/jmoiron/sqlx"
  _ "github.com/lib/pq"
  “log"
)

func main() {
  for i := 0; i < 750; i++ {
    err := dofn()
	if err != nil {
		log.Println("err", err)
		continue
	}
  }
}

In this program, we are simply calling the function dofn(), 750 times. In each call, dofn opens a connection to database ‘test’, reads a value from the table ‘test’ and then either writes (INSERT) the value 1 into table if it does not exist or increments the value (UPDATE) in the table.

Following is the body for dofn implementation:

func dofn() error {
	db, err := sqlx.Open("postgres", "host=127.0.0.1 port=5432 user=postgres password=postgres dbname=test sslmode=disable")
	if err != nil {
		return err
	}
	defer db.Close()

	tx, err := db.Begin()
	if err != nil {
		return err
	}

	defer tx.Rollback()

	var isExists bool = true
	var id int
	var value int
	err = tx.QueryRow("SELECT * FROM test WHERE id = 1;").Scan(&id, &value)
	if err != nil {
		if err == sql.ErrNoRows {
			isExists = false
		} else {
			return err
		}
	}

	if !isExists {
		_, err := tx.Exec("INSERT INTO test VALUES(1, 1)")
		if err != nil {
			return err
		}
	} else {
		value = value + 1
		_, err := tx.Exec("UPDATE test SET value = $1 WHERE id = 1", value)
		if err != nil {
			return err
		}
	}

	tx.Commit()
	return nil
}

Lets open the psql console and delete all rows in the table prior to running this example.

test=# DELETE FROM test; 
DELETE 0
test=# SELECT * FROM test; 
 id | value 
----+-------
(0 rows)

Now, we compile and run this example, as expected we will get the value updated to 750. This was the sequential run.

parallels@ubuntu:~/gocode/src/bb/seq$ ./seq 
parallels@ubuntu:~/gocode/src/bb/seq$ sudo -u postgres psql 
psql (9.4.10)
Type "help" for help.

postgres=# \c test
You are now connected to database "test" as user "postgres".
test=# SELECT * FROM test; 
 id | value 
----+-------
  1 |   750
(1 row)

Let make it a bit concurrent..

Now, let’s try to modify the program to do execution with the same dofn but now executing the updates concurrently:

package main

import	(
	"log"
	"database/sql"
	"github.com/jmoiron/sqlx"
	_ "github.com/lib/pq"
)

func main() {
	for i := 0; i < 3; i++ {
		go launchDoFn() // async launch goroutines (similar to threads) 
	}

	select {}
}

func launchDoFn() {
	for i := 0; i < 250; i++ {
		err := dofn()
		if err != nil {
			log.Println("err", err)
			continue
		}
	}	
}

As you see we launch three goroutines - each of which start sequential updates of 250 increments (including one insert). There are basically three concurrent threads you can assume which are updating DB. So let’s compile and run and see the result.

First, we will again clear the table so that it is empty again:

postgres=# \c test
You are now connected to database "test" as user "postgres".
test=# DELETE FROM test; 
DELETE 1
test=# SELECT * FROM test; 
 id | value 
----+-------
(0 rows)

Now run it and check the result:

parallels@ubuntu:~/gocode/src/bb/conc$ ./conc 
2016/12/01 19:06:20 err INSERT pq: duplicate key value violates unique constraint "test_pkey" pq: duplicate key value violates unique constraint "test_pkey"
2016/12/01 19:06:20 err pq: duplicate key value violates unique constraint "test_pkey"
parallels@ubuntu:~/gocode/src/bb/conc$ sudo -u postgres psql 
psql (9.4.10)
Type "help" for help.

postgres=# \c test
You are now connected to database "test" as user "postgres".
test=# SELECT * FROM test; 
 id | value 
----+-------
  1 |   464
(1 row)

Note: your output may be different - since it is unpredictable but in all likelihood it would be incorrect i.e. not equal to 750.

So what happened?

Well, something like this:

Time	Thread A	Thread B
time t0	reads N = 5	reads N = 5
time t1	increments N = N + 1 = 6	increments N = N + 1 = 6
time t2	writes into DB N = 6
time t3		writes into DB N = 6

So instead of the final value = 7 we get the value = 6. This is happening multiple (unpredictable number of) times and hence causing the mismatch between expected and actual results.

Now, we will discuss how to fix this.

A primer on isolation levels

Since this is a common problem, SQL standard defines a way to solve this. Basically there are four levels of transaction isolation that are defined. As mentioned on postgres documentation page:

The most strict (isolation level) is Serializable, which is defined by the standard in a paragraph which says that any concurrent execution of a set of Serializable transactions is guaranteed to produce the same effect as running them one at a time in some order. The other three levels are defined in terms of phenomena, resulting from interaction between concurrent transactions, which must not occur at each level.

dirty read: A transaction reads data written by a concurrent uncommitted transaction.
nonrepeatable read: A transaction re-reads data it has previously read and finds that data has been modified by another transaction (that committed since the initial read).
phantom read: A transaction re-executes a query returning a set of rows that satisfy a search condition and finds that the set of rows satisfying the condition has changed due to another recently-committed transaction.

Isolation Level	Dirty Read	Non repeatable Read	Phantom Read
Read Uncommitted	Possible	Possible	Possible
Read committed	Not possible	Possible	Possible
Repeatable Read	Not Possible	Not Possible	Possible
Serialisable	Not Possible	Not Possible	Not Possible

In PostgreSQL, you can request any of the four standard transaction isolation levels. But internally, there are only three distinct isolation levels, which correspond to the levels Read Committed, Repeatable Read, and Serializable.

Also,

.. phantom reads are not possible in the PostgreSQL implementation of Repeatable Read, so the actual isolation level might be stricter than what you select.

This is permitted by the SQL standard: the four isolation levels only define which phenomena must not happen, they do not define which phenomena must happen.

Quick take away!

When we set the isolation level to serializable then behind the scene all concurrent updates to a database are performed in serial fashion i.e. one after the other. This clearly translates into bad performance.
When we set the isolation level to Repeatable read then behind the scene, database makes sure that of all transactions which are currently executing on a record read in “past” and trying to update the record (pay attention here: say A, B are executing, both read record #3 in table (i.e. did a ‘select’), then try writing into database updated values of record #3) - it is guaranteed that only one of them will succeed and others will fail with “could not serialize access due to concurrent update” error. This is better performant than serializable, fixes the problem of correctness but means that you have to repeat the transactions which fail.
Read committed is the default mode of postgres. It is safe when you are concurrently reading but not safe for concurrent updates.

Fixing using repeatable read isolation level

As mentioned above, in postgres repeatable read is as strict as serialisable and is more performant than serialisable. So, we will use this isolation level to fix the problem we faced while running concurrent updates.

We re-write our code for dofn():

func dofn() error {
	db, err := sqlx.Open("postgres", "host=127.0.0.1 port=5432 user=postgres password=postgres dbname=test sslmode=disable")
	if err != nil {
		log.Println("err", err)
		return err
	}

	defer db.Close()

	var has_committed bool = false

	for ok := true; ok; ok = !has_committed {
		tx, err := db.Begin()
   		if err != nil {
       		log.Println("err", "error begining transaction in postgres", err)
       		return err 
   		}

		defer tx.Rollback()
 		_, err = tx.Exec(`set transaction isolation level repeatable read`)   // <=== SET ISOLATION LEVEL
		if err != nil {
   			return err
		}

		var isExists bool = true
		var id int 
		var value int
		err = tx.QueryRow("select * from test1 where id = 1;").Scan(&id, &value)
		if err != nil {
			if err == sql.ErrNoRows {
				isExists = false
			} else {
				return err
			}
		}

		if !isExists {
			_, err := tx.Exec("INSERT INTO test VALUES(1, 1)")
			if err != nil {
           		 return err
            }
		} else {
			value = value + 1
			_, err := tx.Exec("UPDATE test set value = $1 where id = 1", value)
			if err != nil {
			if strings.Contains(err.Error(), "could not serialize access due to concurrent update") {
                continue
            } else {
            	return err
            }
		}
	}

	tx.Commit()
	has_committed = true
	}
	return nil
}

Let’s again empty the table, compile and run this and see results:

test=# DELETE FROM test; 
DELETE 1
test=# SELECT * FROM test; 
 id | value 
----+-------
(0 rows)

parallels@ubuntu:~/gocode/src/bb/conc$ ./conc
2016/12/01 19:28:06 err UPDATE pq: could not serialize access due to concurrent update pq: could not serialize access due to concurrent update
2016/12/01 19:28:06 err UPDATE pq: could not serialize access due to concurrent update pq: could not serialize access due to concurrent update
...
...
... -- many errors and retries --- 
... 
parallels@ubuntu:~/gocode/src/bb/conc$ sudo -u postgres psql 
psql (9.4.10)
Type "help" for help.

postgres=# \c test
You are now connected to database "test" as user "postgres".
test=# SELECT * FROM test; 
 id | value 
----+-------
  1 |   750
(1 row)

Hence, we see the issue of correctness is resolved and with this change our program is concurrency safe wrt database update.

The post is longish - thanks for reading it through.

Runscope Integration with Mattermost

2016-09-24T00:00:00+00:00

Runscope is an API monitoring and testing tool. It allows you to test your web service endpoints periodically (say every 5 mins) for uptime & errors with a good support for alert and notifications.

We are using Runscope at my workplace to monitor our web services. It has great integration with Slack so any Runscope tests that results in error would post a notification on one of our Slack channels.

As I wrote earlier, we are replacing Slack with an open source alternative - Mattermost.

Now the problem is that Runscope does not provide any ready-to-use integration with Mattermost. So, we need to do more toil. Basically we can use Webhook Notifications. To do this, you run a web service on public IP (or behind ELB) and register it with RunScope for tests you are interested in. This web service will receive a callback from Runscope (and HTTP POST) in a certain format containing all data about the test that ran. Your web service can take appropriate action for the callback - e.g. in our case - if we detect there is an error we send a HTTP POST to Mattermost with data in same format as we were getting on Slack.

I have written some example code in go - runscope-mattermost-hook for the integration service which you may refer or use. If you have feedback please tweet to me.

Ref: https://github.com/awmanoj/runscope-mattermost-hook

Mattermost - an opensource self-hosted Slack Alternative

2016-09-11T00:00:00+00:00

If you’re even remotely in tech you would be knowing about and with high chance using Slack - now ubiquitous messaging system which has replaced (almost) IRCs, Jabber and other IM-like clients especially in corporates.

In my current company, we have been using Slack extensively - free version.

Free version of Slack has some limitations (ofcourse, that’s why it is free!):

There is a limit to searchable items (upto 10,000 of your team’s most recent messages)
Upto 10 apps or service integrations
5 GB of total file storage for the team.

Now, these are some serious limitations. Not all channels are made equal e.g. a channel for discussing emergency response would have a higher bearing on losing messages (entire history of discussions!) because of some random chats elsewhere than a general channel for discussing company gossip. In our case, we were losing history of the messages as soon as we consume the limits (10K messages for example) and this used to come without warning.

Need..

Till the time team is small (say ~ 50-100) you don’t really have to worry about this as much but once the team grows beyond it then requirements arise to curb these limits. The solution is you pay for the service on Slack. This is exactly what we planned.

The pricing of Slack is per user and minimum pricing that is available from Slack is $6 per user per month.

Saviour..

That is when I got to know about Mattermost - an opensource self-hosted Slack alternative which is incredibly Slack-compatible. For more than a month, we have been using this alongwith the Slack to get a sense of “if we are not able to do a certain thing”. Till now, functionality wise Mattermost is working for us well.

Our setup of Mattermost is on AWS:

1 t2.medium EC2 instance (with 1 100G EBS Volume)
1 RDS instance for postgres

It provides following benefits:

No limit to number of messages (well, there is - but it is disk!) so no fear of losing out history.
No limit to number of app of service integrations.
No limit to the storage.

(As pointed, limit is elastic - you can always go and upgrade your disk - also, disk is cheap)

It was simple to switch:

It is easy to setup. There is an awesome guide to setup mattermost on its site.
It is not only slack compatible - the interface is also similar. So no “learning curve” friction.
We were able to move all our existing slack integrations to Mattermost because of awesome compatibility. In most cases (except runscope - which I will write about sometime), you will just need to replace Slack webhook url with the Mattermost webhook url. It’s that simple.
Mattermost comes with both an Android and an iOS App.

Cost comparison..

The approximate per month cost (as per this site) would be $50 (EC2) + $10 (100GB storage) + $30 (2GB RAM RDS postgres instance) = $90. This is approximate so let’s round it to $100.

This is not per user. :-)

The strength of my company is 500+ currently and it’s growing fast. Even if we assume 50% of them being active on slack (which is an underestimate) we can see quickly that cost of buying the Standard Slack would be $6 * 250 = $1500 per month. As compared to this, the cost of setting up and running Mattermost for communication needs is meagre $100 per month.

Downsides..

One problem which we have faced and have heard complaints from teams is around Notifications in Mattermost. Notifications are not as smooth as Slack for Mobile and Desktop both. There is delay and alerting doesn’t work sometime in realtime. In worst cases, what I have observed is that with a delay of few minutes we do get notifications.

But with Mattermost setup, the load on Slack (free version) anyways comes down so you can choose to keep few alerting systems running on free version Slack and move more persistent kind of requirements (like chat, datadog integrations, consul integrations) on Mattermost.

Consul Integration with Opsgenie

2016-09-11T00:00:00+00:00

In past I discussed about using consul as a service discovery and configuration management solution and how to configure for alerts on key value changes.

Opsgenie is a must-have tool for SRE team of any company beyond a certain scale. It enables alerts on some events (services down for example) via email, SMS, mobile push and even phone calls (so this is the enabler for dreaded pager service).

At my company we use opsgenie and SRE team is on a constant roster - so on a rotation basis different engineers are on-call.

There was a request to integrate service health on consul to opsgenie. Quick search over internet revealed that there is a solution called consul-alerts which is also the official recommendation from opsgenie.

After installing it and trying to figure out - to be frank - I couldn’t comprehend it well (couldn’t get it deliver notifications successfully rather). So I thought, let me get to first principles and use the native consul watch command and some bash to do the simple job (isn’t KISS something never to be forgotten?) and came up with this simple(st possible) script which does the job well. This script is invoked from an upstart script (ubuntu 14.04) which takes care of re-running it if it gets killed somehow.

#!/bin/bash 

OLD_SERVICES_CRITICAL=
while [ 1 ]; do 
  rm -f /tmp/consul/services.txt
  consul-template -consul <___CONSUL_SERVICE_IP_HERE____>:8500 -template="/tmp/consul/services.alert.ctmpl:/tmp/consul/services.txt" -once
  SERVICES_CRITICAL=`cat /tmp/consul/services.txt | grep -i critical | awk '{print "["$1" "$2"]"}'`

  if [ ! -z "$OLD_SERVICES_CRITICAL" ] && [ -z "$SERVICES_CRITICAL" ]; 
    OLD_SERVICES_CRITICAL=""
  fi

  if [ ! -z "$SERVICES_CRITICAL" ] && [ $SERVICES_CRITICAL -ne $OLD_SERVICES_CRITICAL ]; then
    curl -XPOST 'https://api.opsgenie.com/v1/json/alert' -d"{
      \"apiKey\": \"______YOUR_OPSGENIE_KEY_HERE______\", 
      \"message\": \"Services down: $SERVICES_CRITICAL.\",
      \"teams\": [\"______YOUR_SRE_TEAM_NAME_HERE_____\"]
    }"
    
    # Add an alert to Slack.
    #... 

    # Add an alert to Mattermost.
    #... 

    OLD_SERVICES_CRITICAL=$SERVICES_CRITICAL
  fi

  ## every 10 seconds
  sleep 10
done

service.alert.ctmpl is simple - it lists down all service names, their IP, their port and current status (passing, warning, critical etc.):

{{ range services }}{{ range service .Name "any" }}
{{.Name}} {{.Address}} {{.Port}} {{.Status}}
{{end}}{{ end }}

It took less than 5 minutes to get this running.

On slow regular expressions in golang

2016-09-08T00:00:00+00:00

Few months back, a colleague mentioned over lunch how slow golang regular expressions are - this was not only compared to awesome performance of Perl regex or PCRE but also other languages. Recently, while looking for threads discussing Go vs Python wrt regex I got hold of following comments (by Russ Cox):

First of all, Ruby and Python are using C implementations of the regexp search, so Go is being beat by C, not by Ruby.

More importantly,

Second, Go is using a different algorithm for regexp matching than the C implementations in those other languages. The algorithm Go uses guarantees to complete in time that is linear in the length of the input. The algorithm that Ruby/Python/etc are using can take time exponential in the length of the input, although on trivial cases it typically runs quite fast. In order to guarantee the linear time bound, Go’s algorithm’s best case speed a little slower than the optimistic Ruby/Python/etc algorithm. On the other hand, there are inputs for which Go will return quickly and Ruby/Python/etc need more time than is left before the heat death of the universe. It’s a decent tradeoff.

Take away 1: Fair enough!

Take away 2: Prefer using strings functions if your job gets done with that. Didn’t authors already tell us Go is an opinionated language!

Using consul watch and git diff to track key value changes

2016-09-03T00:00:00+00:00

In the previous post I introduced how consul is a simple, distributed and fault tolerant system for service discovery and configuration management.

Now, a common need is how do we track if someone has changed a key value or added or deleted a key? Solution is consul watch command. We can use consul watch for registering a handler which gets called whenever there is a key-value change. Its format is:

$ consul watch -http-addr consul.service.local:8500 -type keyprefix -prefix global /usr/local/bin/watch_handler.sh

But there is a problem - everytime there is a change in a key value e.g. key global/redis/cacheA is changed from 192.168.1.1 to 192.168.1.2 then consul watch handler gets invoked with latest values only i.e. 192.168.1.2.

We felt that there is a need to know the change in full context i.e. old_value and new_value. After some searching and not finding a solution we devised a method (rather novel I’d say but I am blinded by bias) by using git - the most awesome change control system - in local mode (no remote set).

Idea is simple.

Following steps are to be done as part of the initial setup:

On the node where you want to setup the alert system designate a directory say /var/consul/data.
Run following command to initialize this directory as a git repo.

$ git init

Run following command to save the latest snapshot of all key values for keys starting with prefix global to kvs.txt.

$ consul watch -http-addr consul.service.local:8500 -type keyprefix -prefix global > kvs.txt

Add and commit to the local git repo

$ git add kvs.txt 
... 
$ git commit -m "Update kvs.txt"
...

Once the above setup is done - run the consul watch in daemon mode (via upstart preferably):

$ consul watch -http-addr consul.service.local:8500 -type keyprefix -prefix global /usr/local/bin/watch_handler.sh

This ensures that when there is a change in any of the key values under the specified prefixed keys then it will trigger the execution of watch_handler.sh. This script does following on each such trigger:

Change directory to /var/consul/data/.
Run following command to get the latest (post change) key values:

$ consul watch -http-addr consul.service.local:8500 -type keyprefix -prefix global > kvs.txt

Execute git diff.

If empty (no change) then don’t do anything else trigger your alert mechanism with the output of git diff command (e.g. mail, slack, mattermost etc.)

Now that the alert is already sent - commit the new change locally.

$ git add kvs.txt 
... 
$ git commit -m "Update kvs.txt"
...

Note: You will need to do a bit more actually than just git diff since the diff command gives values in base64 encoding. To convert that into readable values you will need to use base64 utility and some play with sed, awk etc. Following is some code which can be handy:

OLD=`git diff | sed -e 's/\"//g' | grep Value | awk -F'Value:'  '{print $2}' | awk -F, '{print $1}' | awk '{print $1}' | head -1 | base64 -d | sed -e 's/^/>/g' | sed -e 's/$/\\n/g'` 
NEW=`git diff | sed -e 's/\"//g' | grep Value | awk -F'Value:'  '{print $2}' | awk -F, '{print $1}' | awk '{print $1}' | sed '2q;d' | base64 -d | sed -e 's/^/>/g' | sed -e 's/$/\\n/g'`

This is a screenshot of the change alert in our slack:

Epilogue

In addition to providing the full context of change, this solution is also useful in keeping a local versioned history and is helpful in reverting to older values easily by using git revert or git reset ... commands.

Credit to use of git for this goes to @kernelhacker.

I don’t yet know if there is any other builtin technique in consul-watch to achieve the same thing. If you know please drop a tweet at @awmanoj. TIA.

Service discovery, configuration management with Consul

2016-08-27T00:00:00+00:00

If you have a reasonably growing setup of services then you would already know what kind of hell configuration management becomes. Let me give an example. Suppose there are three services A, B and C which depend on two redis instances 192.168.1.1:6379, 192.168.1.2:6379 and a postgres instance 192.168.1.3:5432 then your configuration for the three services would look something like this:

[Redis]
	cacheA = 192.168.1.1:6379
	cacheB = 192.168.1.2:6379
[Postgres]
	mainDb = 192.168.1.3:5432 
...

Now, add to this the fact that services could be concurrent and running on multiple servers behind some load balancer i.e. A is not one single physical server (or VM) but multiple i.e. A1, A2, A3 and similarly B is say B1, B2 and C is say just C1.

So probably you already see the problem. If we have some trouble with redis cacheA and we have to replace this instance with another one say 192.168.1.4:6379 then we essentially have two options:

Manually ssh to each of the running instances (i.e. A1, A2, A3, B1, B2, C1), make the configuration change and restart or reload the service.
Better - make the change in configuration in the code and redeploy the services on instances using the continuous integration system you’re using (jenkins?).

The second of the above two is less painful but is waste of effort if this switch to alternate redis is temporary. It also takes considerable effort in terms of making change to code on all three repositories (A, B and C), building them via continuous integration system and deploying. It also takes considerable time.

A related problem to configuration management is Service discovery. Let’s take for example following nginx configuration which defines a server that is reverse proxy to few backend instances of services.

upstream backend {
    server backend1.example.com weight=5;
    server backend2.example.com;
    server backend3.example.com;
    server backup1.example.com  backup;
}

server {
    ...
    location /v1/awesome/service {
        proxy_pass backend;
    }

}

Nginx is awesome in distributing load on the site amongst the instances backend1, backend2 and backend3 in weighted round robin fashion. But how do we deal with servers which are down? Suppose backend2.example.com goes down - we would ideally want to remove it from the configuration immediately making it look like:

upstream backend {
    server backend1.example.com weight=5;
    server backend3.example.com;
    server backup1.example.com  backup;
}
...

Also, as soon as the service backend2.example.com is online again, we should add it to the backend:

upstream backend {
    server backend1.example.com weight=5;
    server backend2.example.com;
    server backend3.example.com;
    server backup1.example.com  backup;
}
...

Do we have a solution? Actually, there are many solutions for this problem. One of them is topic of this post.

consul.. !

Consul is a very simple service discovery and configuration management solution. This means it can store configuration data or rather any kind of key-value data. It can also maintain services information i.e. services can register and deregister with consul. Consul provides HTTP and DNS APIs for service discovery. It is distributed, HA and datacenter-aware.

The last property is particularly useful - if you have multiple data centers (multiple availability zones in AWS or may be a DC from AWS and another on from Google or Alibaba) then you can have a consul cluster on each of them which can contain cluster specific configuration (key value stores, service information) and each of those clusters can be made to be aware of the other. As needed configuration can be replicated too across data centers.

Consul is written in go and hence is available as a native binary. It is in the same league of softwares as etcd, zookeeper etc. (in terms of functional usage) but I think it is much simpler for deployment and usage.

Setup

You can play with consul by following this quickstart guide. For production deployment you can use this awesome guide for production setup of consul on Ubuntu (other guides also available there).

As part of consul cluster setup you will also setup consul Web UI which is the interface to list all services registered with consul, how many nodes under each of the services are running successful (‘passing’ or ‘failing’), all key values (complete CRUD for these and hence you can create, modify keys from the UI itself). Following are the screenshots for web UI (services and key values):

Consul-template

Now, that you have the cluster ready and populated with configuration data (key values) that you would want to refer to from apps - how really do you use these. Here comes consul-template - another native binary produced from the go code. From it’s read me:

This project provides a convenient way to populate values from Consul into the file system using the consul-template daemon. The daemon consul-template queries a Consul instance and updates any number of specified templates on the file system.

and ..

As an added bonus, consul-template can optionally run arbitrary commands when the update process completes.

Configuration Management

So - let’s get back to our examples above. To recap, we had three services A, B and C. Each had a configuration referring the two redis instances cacheA and cacheB.

With consul:

o. you replace the fixed configuration file in your code:

file: app.prod.ini

[Redis]
	cacheA = 192.168.1.1:6379
	cacheB = 192.168.1.2:6379
[Postgres]
	mainDb = 192.168.1.3:5432

with a consul template file:

file: app.prod.ini.ctmpl

[Redis]
    cacheA = {{ key "service/redis/cacheA" }}
    cacheB = {{ key "service/redis/cacheB" }}
[Postgres]
    mainDb = {{ key "service/postgresql/mainDb" }} // ignore username, password etc.

o. you install consul-template on each of those instances where the services A, B and C are running.

Note that this is a one-time investment and things will be even easier if you are using some automation tool like ansible or cloudformation (or chef, puppet). If you aren’t using any of these and still setting up every new server by hand - you should try one of these - my current favorite is Ansible.

so, now A1, A2, A3, B1, B2, C1 all have consul-template installed. Create an upstart for consul-template in which you run it as a daemon like following:

$ /usr/local/bin/consul-template -consul "<__ONE_OF_CONSUL_NODES_IP>:8500" -template="/etc/appA/app.prod.ini.ctmpl:/etc/appA/app.prod.ini" >> /var/log/consul.log 2>&1

Your apps will be responsible for upstart scripts in this case and will be installed on deployment.

o. Now, when your redis instance cacheA goes down as we assumed in our hypothetical scenario above and you have to replace it with 192.168.1.4:6379 you simply need to go to consul web UI and update the key service/redis/cacheA to 192.168.1.4:6379 and consul-template which is running as a daemon will update the configuration instantly. Consul-template takes more arguments and you can write better command line by using config instead of -consul and -template as mentioned above. In the config format you can specify how to reload your app after config update happens. You can read more about it on the consul-template readme. Btw, you just saw how automated configuration management now is & you’re done!

Service Discovery

A service is registered with consul cluster by running consul in client mode and keeping some configuration about the service in consul configuration directory. Alternatively, we can also use consul HTTP API for registering a service. You can get a comprehensive documentation on the consul website.

file: /etc/consul.d/redis_cache_a.json

{
  "service": {
    "name": "redis-cache-a",
    "tags": ["master"],
    "address": "127.0.0.1",
    "port": 6379,
    "enableTagOverride": false,
    "checks": [
      {
        "script": "/usr/local/bin/check_redis.py",
        "interval": "10s"
      }
    ]
  }
}

There are few things here to note. We can specify a custom check script which is health checker of the service. If this script fails then consul marks the node as failed or service as down. You can also use following instead of a bash script:

 
{
...
  "checks": [ 
    { 
      "tcp": "127.0.0.1:6379",
      .. 
    } 
  ]
..
}

This basically checks TCP connection on the specified port. There is more comprehensive list of possible checks in docs.

Other thing to note is the name of the service - this is an identifier of the service and hence it should be chosen as service identifier. It is recommended though to keep it readable and easy to remember. If your service A has a codename e.g. some greek god (what else?) like zeus then you can use that here.

Once your service is registered with consul, you can check it on both HTTP and DNS interface. Let’s see how it is to check it on DNS interface using dig unix utility:

$ dig @127.0.0.1 -p 8600 redis-cache-a.service.dc1.consul. ANY

(dc1 is the datacenter name for consul)

Once this is setup and your services are registered you can use consul-template again to generate nginx configuration file based on which services are alive.

file: awesome-service.awesome.net

 
upstream backend {
    server backend1.example.com;
    server backend2.example.com;
    server backend3.example.com;
}

server {
    ...
    location /v1/awesome/service {
        proxy_pass backend;
    }

}

file: awesome-service.awesome.net.ctmpl

 

upstream backend {
    {{range service "web@dc1"}}
    server {{.Address}}:{{.Port}}{{end}}
}

server {
    ...
    location /v1/awesome/service {
        proxy_pass backend;
    }

}

Consul returns, by default, only healthy services. You just saw how it can completely automate your configuration management and service discovery. Following is a sequence diagram of how the config changes take place:

I have just touched the tip of iceberg with respect to the possibilities with consul - an introduction. If you aren’t tied up to another configuration management solution do give it a try.

Using Boltdb as a fast, persistent key value store

2016-08-03T00:00:00+00:00

Originally posted on Tokopedia Tech Blog

Recently, we at Tokopedia rearchitected a golang microservice which maps an Image (as in picture, photo, ..) ID to the fully qualified server name that hosts the image.

A bit of context is always good..

Historically, we stored images and served images from multiple servers for reasons of scale. The mapping of which image is located on which server was in mongodb. And there was a layer of caching on top of it in redis, because mongo didn’t scale very well for the workload.

Gradually, we moved stuff to s3, which means all images uploaded from that point onwards could now be served from a single server, and no such mapping was required. However, old images were still being served in the same way, and they needed the mapping. The difference was, our database was now read only and fixed size.

We still occasionally suffered with memory spikes and service killed by linux oom-killer leading to both latency and downtimes.

Search for alternative..

Requirements for the service:

Persistent storage
Fast retrieval (tens of thousand QPS)
Read only usage

Since our requirement is also “read-heavy” usage on a fixed size database (infact, it is read-only usage) we knew we need a lightweight embedded DB solution. There are multiple solutions that exist for this problem but we zeroed on Boltdb.

Why Boltdb?

written in go and hence fits with rest of the stack well.
compact, fast (it based on LMDB. Both use a B+tree, have ACID semantics with fully serializable transactions, and support lock-free MVCC using a single writer and multiple readers.
LMDB heavily focuses on raw performance while Bolt has focused on simplicity and ease of use.
fits better for a read heavy usage.

How to use?

In pure sense, boltdb is not really a database but simply a memory mapped file but it provides ACID semantics as provided by databases so calling it a DB is not misnomer. It comes as a library so installation is as simple as importing a go package.

You can go through the detailed README and GoDoc to get started.

Opening database is as simple as:

db, err := bolt.Open("my.db", 0666, &bolt.Options{ReadOnly: true})
if err != nil {
    log.Fatal(err)
}

Add a key value (answer = 42):

db.Update(func(tx *bolt.Tx) error {
    b := tx.Bucket([]byte("MyBucket"))
    err := b.Put([]byte("answer"), []byte("42"))
    return err
})

Fetch a value by key:

db.View(func(tx *bolt.Tx) error {
    b := tx.Bucket([]byte("MyBucket"))
    v := b.Get([]byte("answer"))
    fmt.Printf("The answer is: %s\n", v)
    return nil
})

Command line utility - `bolt`

Boltdb comes with a command line utility which can be used to inspect the correctness and statistics of a BoltDB file.

If life was that easy.. !

We imported all mongodb data using mongo-export utility. It was ~4 GB in size. We exported it to boltdb using an export tool we wrote in golang. The output file was a single bolt.db file with size ~ 13 GB. This size increase was expected due to the internal data structure being used by boltdb.

What was not expected (or rather we did not expect it) was that export tool became exponentially slower as the bolt.db file grew larger. We estimated that each export would take days in this manner (in case we need to).

To solve this problem, we decided to use - you guessed it right - sharding.

We broke down the initial mongodb dump into N files (choosing N).
We export each of those files into a boltdb file. We now have N small boltdb files.
When a query arrives for a key K, we hash the key into a bucket between 0 and N-1. This hash value tells us in O(1) time which boltdb to query on.
Query itself is fast due to storage structure and you’re done!

This helped us export the data and generate BoltDB files in less than an hour time.

Following is the output of free -m on one of the 3 servers we use:

and this is the snippet from top on the same server:

Note that the VIRTUAL memory is exactly the total size of boltdb. Also note from above two outputs that it is not limited by physical RAM availability.

Caveats & Limitations

Bolt is good for read intensive workloads. Random writes can be slow especially as the database (file) size grows.
Bolt uses a B+tree internally so there can be a lot of random page access. SSDs provide a significant performance boost over spinning disks.
Bolt can handle databases much larger than the available physical RAM, provided its memory-map fits in the process virtual address space. It may be problematic on 32-bits systems.
The data structures in the Bolt database are memory mapped so the data file will be endian specific. This means that you cannot copy a Bolt file from a little endian machine to a big endian machine and have it work. For most users this is not a concern since most modern CPUs are little endian.

Result

We successfully migrated to using boltdb for our service. The service handles many thousands of QPS, is not limited by physical RAM on the machine and is working well.

BoltDB looks awesome. Do give it a try if this fits your use-case.

Hello World!

2016-06-25T00:00:00+00:00

‘Love’ - in different language scripts.