by naveensrinivasan on kubernetes

How to debug the CrashLoopBackOff in Kubernetes when pod is not starting

Here is my learning of how I debugged the CrashLoopBackOff in kubernetes when the pod wasn’t starting.

I wanted to deploy the jenkins docker image in the cluster. As mentioned in the jenkins docker  repo I wanted to mount an external drive which is an AWS EBS volume.  Here is my deployment yaml.

After starting the deployment the pod never came up and this the output of  kubectl get pod

NAME                                      READY      STATUS                  RESTARTS  AGE
jenkins-3317895845-x84u3  0/1       CrashLoopBackOff      10                 27m

The next step was to issue the kubectl describe pod jenkins-3317895845-x84u3

So from the logs I could make out the container is being pulled correctly but it is failing on StartContainer.  Now based on this information the next step was to get the docker logs. But the docker logs aren’t accessible from my box because it is managed by kubernetes. The only way to get the docker logs was to actually ssh into the box.

But which box should I ssh? The describe output has the node information which is Node: ip-172-20-0-29.us-west-2.compute.internal/172.20.0.29 . This is the private ip of the box in aws but with that information you should be able to figure out the public ip to ssh.

Now I know I have to ssh but where is the key for this server. If you have used the kube-up.sh then the keys would be stored  in this location ~/.ssh/kube_aws_rsa. ssh -i ~/.ssh/kube_aws_rsa admin@public-ip-oftheabovenode.

After sshing into the box I issued the command sudo docker ps -a | grep naveen because the container could have been stopped and looked for naveen because that was my container name. This gave me container id which was stopped with exit status as 1.

And this was the output of docker logs command

admin@ip-172-20-0-29:~$ sudo docker logs c98a338268a1
touch: cannot touch ‘/var/jenkins_home/copy_reference_file.log’: Permission denied
Can not write to /var/jenkins_home/copy_reference_file.log. Wrong volume permissions?

which identified the /var/jenkins_home  which was mounted with aws ebs voulme  didn’t have permission to write by the jenkins user https://github.com/kubernetes/kubernetes/issues/2630.

And after doing all of this I realized I could have done kubectl logs jenkins-3317895845-x84u3 which would have given the same output without having to ssh into the box. But knowing this handy because when things go wrong we really need to debug the root cause.

 
by naveensrinivasan on OSX

Enable Shortcat app in OSX

Shortcat app is a software that I use everyday to keep me productive and reduce my dependency on mouse.

But if you have OSX El Captain you get a message to enable the assistive devices,. But it has been changed from the previous versions of OSX.

To enable the assistive devices navigate to  System Preferences > Security & Privacy > Privacy > Accessibility and enable for shortcat app.

 
by naveensrinivasan on tips

My Docker aliases

I  have been using docker very often now and because I have been spending most of time in the terminal it made sense to use alias to reduce the typing.

I have curated this from others and have also customized to what I want.

 

All my aliases are stored in my github repo https://github.com/naveensrinivasan/dotvim/blob/master/zshrc

 

 

 
by naveensrinivasan on tools

How I try avoid using mouse

Like most of the Dev’s I love using my Keyboard more than mouse. I am big vim fan and have been using vim for over 7-8 years. I still consider myself novice.

Here are the ways I manage to use keyboard over mouse

  • Shortcat – Great app get rids of use mouse in OSX – This is a Paid App , which IMHO is the best investment for me. There is also a trial version.
  • MacVim – I have been using MacVim as my primary text editor.
  • cVim – Vim features within chrome. The best part is it allows custom vimrc.
  • IntelliJ Vim Plugin – https://github.com/JetBrains/ideavim also allow vimrc
  • Here is a plugin https://github.com/athiele/key-promoter-fork for IntelliJ – show hints when using the mouse for something which could be done with the keyboard.
  • Spectacle – For window management in OSX with keyboard.

Screen Shot 2016-03-27 at 10.24.42 PM

 

 

 

 

Remapped caps lock – ^ Control Key

 

Here is my vimrc  https://github.com/naveensrinivasan/dotvim/blob/master/vimrc. One of the things I do  is , set no-op for arrow keys makes me a better vim user.  I have tried vim hard mode https://github.com/wikitopian/hardmode and have failed few times. That is something I want to enable to get rid of bad habits.

Still have to try Neovim. I especially like the neovim engine can be plugged into existing editor which is great.

And  I have ordered myself Ultimate hacking Keyboard . The one of the reasons I am excited about this is ,you could use home row keys as mouse which would really replace mouse with this keyboard.

 

 

 
by naveensrinivasan on Uncategorized

Taking it up a notch with my standing desk with fluidstance

I have been standing at work for the past 2  years. And I have seen significant improvement with respect to my productivity and non-drowsiness with standing desk. The one draw back with that I have seen was standing in a position where I put a lot of stress on one of the legs. And I wanted ti try something different.

That’s when I saw fluidstance. Bought myself one which brings in a new challenge. I have been using it for a week and it is great  has made me sore in the places which I didn’t when I was standing without it.

IMG_0557

It is also a fun toy!

 
by naveensrinivasan on F#

Solution to Adventcode Day 7 in FSharp

Here is my solution to adventcode problem 7  using F#.

I am using the adventofcode to improve my functional programming skills. The gist of the problem is to solve the logic gates which is represented as 16 bit integer’s  and these depend on other gates.

The puzzle starts of with simple problem

  • 123 -> x
  • 456 -> y
  • x AND y -> d
  • x OR y -> e
  • x LSHIFT 2 -> f
  • y RSHIFT 2 ->
  • g NOT x -> h
  • NOT y -> i

where the if you solve the above the code should work. But the complexity comes in when the order of the gates are not as expected and so the code would have solve the dependent ones before solving the entire solution.

One of the things that I really love about the FP code is union types and pattern matching.

That is the composite  type that could hold multiple different data types. In a non-functional static language I would have ended up creating nested types for each one of these.

My solutions to the adventofcode is here https://github.com/naveensrinivasan/adventcode/

After finishing I realized that I could have used the Tree structure which could help in traversal of the tree.

which would have made this problem simpler.

Here is my solution.

 
by naveensrinivasan on GitHub

Parsing GitHub API

I have been contributing to https://github.com/octokit/octokit.net project. It is the API for accessing GitHub. One of the recent questions that came up was to get the list of https://github.com/octokit/octokit.net/issues/968.

The API’s are published in HTML https://developer.github.com/v3/

There are about 63 + categories that have API. Wanted to parse all of these with least manual intervention.

The above goes to the main URL looks for all sub-categories and download’s each of the web pages and extracts “GET“, “DELETE“, “PATCH“, “POST“

This uses the HtmlAgility for parsing.

And the output would look something like this.

 

 

 
by naveensrinivasan on F#

fsharp docker image

The fsharp project has a official docker image https://github.com/fsprojects/docker-fsharp . The one issue with that is it is based on mono 4.0.4 which is buggy and fsharp does not work very well. The latest alpha release of the mono with which fsharp works well is 4.2.0. The 4.2.0 isn’t available in stable channels.

So I created a docker image with the latest mono from their alpha repo and using the latest fsharp from the github.

https://github.com/naveensrinivasan/fsharp-docker

It is also available in the docker hub. You could get it by

docker pull naveensrinivasan/fsharp

 
by naveensrinivasan on ETW

View the http redirect and response message from an external authentication provider using ETW

Recently I had to troubleshoot messages that were being sent from an web application hosted on IIS to an external authentication provider. The logs from the application wasn’t something closer to the metal and wasn’t really providing all the details. I really wanted something like fiddler for the webserver. I could have a ran network traces to troubleshoot the issue but the problem was it wasn’t happening consistently. It was sporadic. I knew there would be ETW traces that would have this information. The IIS web logs don’t capture this information.

Here is a example of the SAML authentication process

500px-SAML

In the application I was working with, IIS was the relying party and the user was to be authenticated with Identity Provider.

I wanted to troubleshoot the “AuthnRequest” and “Auth Resp” from and to the IIS. This can be applied to any external authentication like credit card authentication.

I fired my favorite tool Perfview and captured all the IIS traces along with other defaults. I wasn’t really interested in the .NET Code.

Here is the command line for Perfview to the IIS Providers

If for some reason that does not work.  You could always use the additional providers in Perfview and add these providers which are IIS and HTTP providers.

I let perfview do its job and then stopped the trace when there was an issue.

Here are the ETW events that capture the SAML Request that was sent from IIS to the IDP

Event Name

  1. IIS_Trace/IISGeneral/GENERAL_RESPONSE_HEADERS
  2. Microsoft-Windows-IIS/EventID(47)
  3. IIS_Trace/IISGeneral/GENERAL_RESPONSE_ENTITY_BUFFER
  4. Microsoft-Windows-IIS/EventID(49)
  5. IIS_Trace/IISGeneral/GENERAL_REQUEST_HEADERS

SamlRequest

Here are the ETW events that capture the SAML Response that was being posted from the IDP to the IIS

  1. IIS_Trace/IISGeneral/GENERAL_REQUEST_ENTITY
  2. Microsoft-Windows-IIS/EventID(51)

samlresponse

With this I was able to troubleshoot message that was being sent and received to the IIS.

 

 
by naveensrinivasan on yoga

Getting my Yoga stats from Yogaglo

I am Yogi and have practiced some sort physical workout for a while now. IMHO physical strength/ movement have always attributed to better clarity in my life! This post shows how I managed to get my yogaglo stats to track and measure my practice.

NaveenSrinivasan-TRX

I am strong believer in habit loops and always have found that has worked a lot for me. One of the good books on this which I recommend to other ishttp://www.amazon.com/The-Power-Habit-What-Business/dp/081298160X and also another good resource is http://getupandcode.com/ which is a audio podcast fitness and technology.

I have been practicing Yoga for a while now and I would like to track my practice. I usually go to studio twice a week to be part of the Sangha and the rest 4-5 days I practice twice a day.

I knew yogaglo had my stats information stored in their site because when I logged into the site it did provide me with history. But I wanted the API to query based on the raw data. I wanted to track how often I worked and what kind of classes have I done. My goal was to work on the strengthening my core and I usually like to track that and API would help with this kind of information.

Thanks to tools like fiddler or http://mitmproxy.org/ I could look at the http traffic that was sent with the headers. The headers are important because it contained the authentication token information. FYI I have set yogaglo to remember my login information which meant I have cookies that it could send across part of the http request.

Here is the code to download the yogaglo stats

[gist https://gist.github.com/naveensrinivasan/2e40f409bf6c386766c6]

You could take the json and dump into excel and get some amazing stats using powerquery.

I am not a excel whiz to do it. I used the json to convert it to C# objects usinghttp://json2csharp.com/ and here is the code it generated.

[gist https://gist.github.com/naveensrinivasan/0cf3cecede742c3587dc]

With that here is a simple query to get total duration by date.

[gist https://gist.github.com/naveensrinivasan/213c2092babc23d7c772]