Here is my learning of how I debugged the CrashLoopBackOff in kubernetes when the pod wasn’t starting.
I wanted to deploy the jenkins docker image in the cluster. As mentioned in the jenkins docker repo I wanted to mount an external drive which is an AWS EBS volume. Here is my deployment yaml.
After starting the deployment the pod never came up and this the output of kubectl get pod
NAME READY STATUS RESTARTS AGE
jenkins-3317895845-x84u3 0/1 CrashLoopBackOff 10 27m
The next step was to issue the kubectl describe pod jenkins-3317895845-x84u3
So from the logs I could make out the container is being pulled correctly but it is failing on StartContainer. Now based on this information the next step was to get the docker logs. But the docker logs aren’t accessible from my box because it is managed by kubernetes. The only way to get the docker logs was to actually ssh into the box.
But which box should I ssh? The describe output has the node information which is Node: ip-172-20-0-29.us-west-2.compute.internal/172.20.0.29 . This is the private ip of the box in aws but with that information you should be able to figure out the public ip to ssh.
Now I know I have to ssh but where is the key for this server. If you have used the kube-up.sh then the keys would be stored in this location ~/.ssh/kube_aws_rsa. ssh -i ~/.ssh/kube_aws_rsa admin@public-ip-oftheabovenode.
After sshing into the box I issued the command sudo docker ps -a | grep naveen because the container could have been stopped and looked for naveen because that was my container name. This gave me container id which was stopped with exit status as 1.
And this was the output of docker logs command
admin@ip-172-20-0-29:~$ sudo docker logs c98a338268a1
touch: cannot touch ‘/var/jenkins_home/copy_reference_file.log’: Permission denied
Can not write to /var/jenkins_home/copy_reference_file.log. Wrong volume permissions?
which identified the /var/jenkins_home which was mounted with aws ebs voulme didn’t have permission to write by the jenkins user https://github.com/kubernetes/kubernetes/issues/2630.
And after doing all of this I realized I could have done kubectl logs jenkins-3317895845-x84u3 which would have given the same output without having to ssh into the box. But knowing this handy because when things go wrong we really need to debug the root cause.
Shortcat app is a software that I use everyday to keep me productive and reduce my dependency on mouse.
But if you have OSX El Captain you get a message to enable the assistive devices,. But it has been changed from the previous versions of OSX.
To enable the assistive devices navigate to System Preferences > Security & Privacy > Privacy > Accessibility and enable for shortcat app.
I have been using docker very often now and because I have been spending most of time in the terminal it made sense to use alias to reduce the typing.
I have curated this from others and have also customized to what I want.
All my aliases are stored in my github repo https://github.com/naveensrinivasan/dotvim/blob/master/zshrc
Like most of the Dev’s I love using my Keyboard more than mouse. I am big vim fan and have been using vim for over 7-8 years. I still consider myself novice.
Here are the ways I manage to use keyboard over mouse
- Shortcat – Great app get rids of use mouse in OSX – This is a Paid App , which IMHO is the best investment for me. There is also a trial version.
- MacVim – I have been using MacVim as my primary text editor.
- cVim – Vim features within chrome. The best part is it allows custom vimrc.
- IntelliJ Vim Plugin – https://github.com/JetBrains/ideavim also allow vimrc
- Here is a plugin https://github.com/athiele/key-promoter-fork for IntelliJ – show hints when using the mouse for something which could be done with the keyboard.
- Spectacle – For window management in OSX with keyboard.
Remapped caps lock – ^ Control Key
Here is my vimrc https://github.com/naveensrinivasan/dotvim/blob/master/vimrc. One of the things I do is , set no-op for arrow keys makes me a better vim user. I have tried vim hard mode https://github.com/wikitopian/hardmode and have failed few times. That is something I want to enable to get rid of bad habits.
Still have to try Neovim. I especially like the neovim engine can be plugged into existing editor which is great.
And I have ordered myself Ultimate hacking Keyboard . The one of the reasons I am excited about this is ,you could use home row keys as mouse which would really replace mouse with this keyboard.
I have been standing at work for the past 2 years. And I have seen significant improvement with respect to my productivity and non-drowsiness with standing desk. The one draw back with that I have seen was standing in a position where I put a lot of stress on one of the legs. And I wanted ti try something different.
That’s when I saw fluidstance. Bought myself one which brings in a new challenge. I have been using it for a week and it is great has made me sore in the places which I didn’t when I was standing without it.
It is also a fun toy!
Here is my solution to adventcode problem 7 using F#.
I am using the adventofcode to improve my functional programming skills. The gist of the problem is to solve the logic gates which is represented as 16 bit integer’s and these depend on other gates.
The puzzle starts of with simple problem
123 -> x
456 -> y
x AND y -> d
x OR y -> e
x LSHIFT 2 -> f
y RSHIFT 2 ->
g NOT x -> h
NOT y -> i
where the if you solve the above the code should work. But the complexity comes in when the order of the gates are not as expected and so the code would have solve the dependent ones before solving the entire solution.
One of the things that I really love about the FP code is union types and pattern matching.
That is the composite type that could hold multiple different data types. In a non-functional static language I would have ended up creating nested types for each one of these.
My solutions to the adventofcode is here https://github.com/naveensrinivasan/adventcode/
After finishing I realized that I could have used the Tree structure which could help in traversal of the tree.
which would have made this problem simpler.
Here is my solution.
I have been contributing to https://github.com/octokit/octokit.net project. It is the API for accessing GitHub. One of the recent questions that came up was to get the list of https://github.com/octokit/octokit.net/issues/968.
The API’s are published in HTML https://developer.github.com/v3/
There are about 63 + categories that have API. Wanted to parse all of these with least manual intervention.
The above goes to the main URL looks for all sub-categories and download’s each of the web pages and extracts “GET“, “DELETE“, “PATCH“, “POST“
This uses the HtmlAgility for parsing.
And the output would look something like this.
The fsharp project has a official docker image https://github.com/fsprojects/docker-fsharp . The one issue with that is it is based on mono 4.0.4 which is buggy and fsharp does not work very well. The latest alpha release of the mono with which fsharp works well is 4.2.0. The 4.2.0 isn’t available in stable channels.
So I created a docker image with the latest mono from their alpha repo and using the latest fsharp from the github.
It is also available in the docker hub. You could get it by
docker pull naveensrinivasan/fsharp
Recently I had to troubleshoot messages that were being sent from an web application hosted on IIS to an external authentication provider. The logs from the application wasn’t something closer to the metal and wasn’t really providing all the details. I really wanted something like fiddler for the webserver. I could have a ran network traces to troubleshoot the issue but the problem was it wasn’t happening consistently. It was sporadic. I knew there would be ETW traces that would have this information. The IIS web logs don’t capture this information.
Here is a example of the SAML authentication process
In the application I was working with, IIS was the relying party and the user was to be authenticated with Identity Provider.
I wanted to troubleshoot the “AuthnRequest” and “Auth Resp” from and to the IIS. This can be applied to any external authentication like credit card authentication.
I fired my favorite tool Perfview and captured all the IIS traces along with other defaults. I wasn’t really interested in the .NET Code.
Here is the command line for Perfview to the IIS Providers
If for some reason that does not work. You could always use the additional providers in Perfview and add these providers which are IIS and HTTP providers.
I let perfview do its job and then stopped the trace when there was an issue.
Here are the ETW events that capture the SAML Request that was sent from IIS to the IDP
Here are the ETW events that capture the SAML Response that was being posted from the IDP to the IIS
With this I was able to troubleshoot message that was being sent and received to the IIS.