L E A R N I N G S

AnsibleError: Unable to create local directories(/home/ubuntu/.ansible/tmp)

2019-06-17T22:38:00.003+05:30

Ansible Error:

AnsibleError: Unable to create local directories(/home/ubuntu/.ansible/tmp): [Errno 13] Permission denied: '/home/ubuntu/.ansible/tmp'

When I do ls -al I see the .ansible directory is owned by root.

drwx------ 3 root root 4096 Jun 14 09:09 .ansible

So either run Ansible using sudo
OR
Change permissions of this directory to allow the logged in user to create files.

ERROR! Unexpected Exception, this is probably a bug: 'PlaybookCLI' object has no attribute 'options

2019-06-17T22:32:00.002+05:30

Error when running Ansible Playbook

ERROR! Unexpected Exception, this is probably a bug: 'PlaybookCLI' object has no attribute 'options

Haven't been able to solve it yet. Looks like some Ansible version compatability.

POD showing status Init:ImagePullBackOff

2019-06-14T17:21:00.002+05:30

I'm trying to install Redis using Helm... It goes through all the steps fine but when I try to see the POD's I see the status as Init:ImagePullBackOff.

Not sure what this really is. Have you come across such status for PODS?

ubuntu@ip-10-0-64-140:~/$ kubectl get po --namespace platform-svcs

NAME READY STATUS RESTARTS AGE

session-store-redis-ha-server-0 0/2 Init:ImagePullBackOff 0 17s

ubuntu@ip-10-0-64-140:~/$

Used the following command to get to events:

kubectl describe pod session-store-redis-ha-server-0 --namespace platform-svcs

This pointed me to the problem that I do not have access to the repo having Redis image.

error: the server doesn't have resource type “svc”

2018-08-03T18:56:00.001+05:30

ere is what I did to work it through.... 1. Enabled verbose to make sure config files are read properly.

kubectl get svc --v=10

Modified the file as below:

apiVersion: v1

clusters:

- cluster:

server: XXXXXXXXXXXXX

certificate-authority-data: XXXXXXXXXX

name: my-eks

contexts:

- context:

cluster: my-eks

user: aws

name: aws

current-context: aws

kind: Config

preferences: {}

users:

- name: aws

user:

exec:

apiVersion: client.authentication.k8s.io/v1alpha1

command: aws-iam-authenticator

args:

- "token"

- "-i"

- "my-eks"

# - "-r"

# - "<role-arn>"

env:

- name: AWS_PROFILE

value: "aws"

How to run two container websites on a single port in Docker?

2018-05-20T13:15:00.000+05:30

There is no out of the box solution for this because it is not a valid design using dockers. However you can go about installing & configuring a reverse proxy which sits in front of your docker containers.

See this article for details on configuring ngnix as a reverse proxy for docker containers https://www.thepolyglotdeveloper.com/2017/03/nginx-reverse-proxy-containerized-docker-applications/

How to manage files & space in Docker

2018-05-20T12:53:00.000+05:30

There are three main ways docker stores files:

By default, everything you save to disk inside the container is saved in the aufs layer. This doesn’t create problems if you clean up unused containers and images.
If you mount a file or directory from the host (using docker run -v /host/path:/container/path …) the files are stored in the host filesystem, so it’s easy to track them and there is no problem also.
The third way are docker volumes. Those are special paths that are mapped to a special directory in /var/lib/docker/volumes/ path on the host. A lot of images use volumes to share files between containers (using the volumes-from option) or persist data so you won’t lose them after the process exits (the data-only containers pattern).

Since there is no tool to list volumes and their state, it’s easy to leave them on disk even after all processes exited and all containers are removed.

You can use below script to cleanup all the things:

#!/bin/bash

	# remove exited containers:
	docker ps --filter status=dead --filter status=exited -aq \| xargs -r docker rm -v

	# remove unused images:
	docker images --no-trunc \| grep '<none>' \| awk '{ print $3 }' \| xargs -r docker rmi

	# remove unused volumes:
	find '/var/lib/docker/volumes/' -mindepth 1 -maxdepth 1 -type d \| grep -vFf <(
	docker ps -aq \| xargs docker inspect \| jq -r '.[] \| .Mounts \| .[] \| .Name \| select(.)'
	) \| xargs -r rm -fr

org.hibernate.HibernateException: Error accessing stax stream

2017-10-20T22:47:00.003+05:30

Couple of things you should check:

If you are using Eclipse IDE make sure that it is not giving you any error indicators for hibernate.cfg.xml file.
Make sure to close the tag <hibernate-configuration>
Missing !character in the DOCTYPE
Not able to accesshibernate-configuration-3.0.dtd from the url provided.
Space before <?xml version='1.0' encoding='utf-8'?>

Docker: Error: Unauthorized: incorrect username or password on mac OS

2017-10-20T17:07:00.004+05:30

Getting the following error when I run anything...

docker: Error response from daemon: Get https://registry-1.docker.io/v2/library/mysql/manifests/latest: unauthorized: incorrect username or password

Here is how I solved it:

Login into www.docker.com with my username and password
Note down the docker ID which is different than your loginID/email
In Terminal now enter the command "docker login" it will ask you to enter your user and password
If the login is successfull you should get the following message:

Password:

Now you can run the commands and they should run fine.

What is Hibernate?

2017-10-18T18:46:00.000+05:30

It is an ORM Framework to develop persistence logic. It is:

Opensource
Lightweight
Provides abstraction over JDBC
No need for any containers (jsp, servlet etc etc).
Supports POJO and POJI model programming
Can be used individually as well as other technologies like Spring etc
Does not require you to implement any Hibernate API interfaces & neither it requires you to extend any hibernate clases. Hence we can say it is a loosely coupled framework
Allows us to develop Object Based Persistence logic and hence it is called ORM framework
Can change from one DB software to another without having to change any Objects etc
Transaction management, locking, caching etc is all in-built.

JDBC Limitations or Problems

2017-10-18T17:57:00.001+05:30

Changing DB Software is too complex since JDBC code is Database software dependent and is not at all portable.
Programmers need to convert Object notations to DB columns and visa versa.
Exception handling is mandatory and nothing is in built
Versioning and time stamping are not provided out of the box. Need to program and handle them manually.
No proper transaction management support.
Does not support Object level relationships like we have table relationships.
Too much boiler plate code (repeated code) needs to be written.

Concepts of Coding in AWS Lambda

2017-10-14T12:38:00.002+05:30

Languages supported (as on Oct 2017):
C#, Java 8, Node.js (4.3 & 6.10), Python (2.7 & 3.6).

Coding Pattern/Syntax: There are 4 main components we need to remember

Handler – This is your main function which will be called by AWS Lambda when execution starts. All of the event data is been passed to this handler function as the first parameter. The handler processes the event data and may invoke other methods or functions in your code.

Context – This is the second object passed to the lambda function. This is used by our code to interact with AWS Lambda. For e.g., you can figure out the execution time that remains. In node.js there are additional methods available that allows callback.

Logging – AWS Lambda writes logs to CloudWatch

Exceptions – You can throw or raise exceptions which AWS lambda will pass on to the caller.

AWS Cognito: ErrorNotAuthorizedException: Unable to verify secret hash for client 7cakk6gknpt4cvqifdfbt9qm91

2017-10-14T00:10:00.000+05:30

To get rid of this Error uncheck "Generate Client Secrets" when you create the App Client in AWS Cognito. That should solve the problem. Atleast that solves it for me.

How to Install and Configure Ansible on ubuntu 14.04?

2016-03-21T12:22:00.003+05:30

Ansible builds for ubuntu are available in PPA. That means all you need to do is add PPA to you repo's and then use ubuntu's default package installer (apt-get).

To configure the PPA on your machine and install ansible run these commands:

$ sudo apt-get install software-properties-common
$ sudo apt-add-repository ppa:ansible/ansible
$ sudo apt-get update
$ sudo apt-get install ansible

You can also install via pip as below:

$ sudo pip install ansible

What are Ansible Raw modules? Where and when they are used?

2016-03-21T12:12:00.003+05:30

Ansible’s raw modules are generally used to executing commands on managed machines which does not have basic requirements for Ansible on it.

For e.g., Running Ansible requires Python 2.4 with python-simplejson on the remote machine. If none of these are available on remote machines then how would you run Ansible on those hosts? The solution is to use Ansible’s Raw module to first install python and python-simplejson on the managed machine and then you can use all other Ansible modules/features.

Example of running a Raw module:

ansible myhost --sudo -m raw -a "yum install -y python2 python-simplejson"

What is SELinux?

2016-03-21T12:07:00.003+05:30

SELinux is a flexible but mandatory enterprise grade security implementations (from NSA) in Linux kernel itself. This provides support for enforcing different but mandatory access control policies.

You can see /etc/selinux/config file to see if SELinux is enabled or not for your system. The variable SELINUX in this file defines the mode (3 modes) in which SELinux has been operation. These 3 different modes of SELinux are namely:

Disabled: Completely disables the SELinux.
Permissive: enables the SELinux code, but causes it to operate in a mode where accesses that would be denied by policy are permitted but audited
Enforcement: Enables the SELinux code and causes it to enforce access denials as well as auditing them.

SELinux was the NSA's attempt at porting the Orange Book requirements to Linux. There are other popular solutions as well for implementing access controls in linux like AppArmor, GrSecurity etc.

Troubleshooting Best Practices

2015-10-04T17:43:00.001+05:30

No reboots first - Come out of this mindset ASAP.
Divide the problem into smaller logical parts
Understand the systems you are going to troubleshoot carefully
Refer problem history to see if something similar occurred in the past too for which we already have a solution.
Generally first thing we should look at is “What changed recently?”
Document each and every Problem and steps you take to solve it
Communicate and do not shy away in involving other team members for help.

Fix "ValueError: unknown locale: UTF-8" when running AWS CLI

2014-12-10T12:15:00.000+05:30

I was getting following error when running AWS CLI.

MacBook-Pro:python sgoswami$ aws ec2 describe-instances
Traceback (most recent call last):
File "/usr/local/bin/aws", line 15, in <module>
import awscli.clidriver
File "/Library/Python/2.7/site-packages/awscli/clidriver.py", line 31, in <module>
from awscli.help import ProviderHelpCommand
File "/Library/Python/2.7/site-packages/awscli/help.py", line 20, in <module>
from docutils.core import publish_string
File "/Library/Python/2.7/site-packages/docutils/core.py", line 20, in <module>
from docutils import frontend, io, utils, readers, writers
File "/Library/Python/2.7/site-packages/docutils/frontend.py", line 41, in <module>
import docutils.utils
File "/Library/Python/2.7/site-packages/docutils/utils/__init__.py", line 20, in <module>
import docutils.io
File "/Library/Python/2.7/site-packages/docutils/io.py", line 18, in <module>
from docutils.utils.error_reporting import locale_encoding, ErrorString, ErrorOutput
File "/Library/Python/2.7/site-packages/docutils/utils/error_reporting.py", line 47, in <module>
locale_encoding = locale.getlocale()[1] or locale.getdefaultlocale()[1]
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.py", line 496, in getdefaultlocale
return _parse_localename(localename)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.py", line 428, in _parse_localename
raise ValueError, 'unknown locale: %s' % localename
ValueError: unknown locale: UTF-8

All that I did to fix it was:

export LC_ALL=en_US.UTF-8

export LANG=en_US.UTF-8

What are manifest and site.pp files?

2014-05-11T14:15:00.001+05:30

Manifest is puppet’s term for files containing configuration information. Manifest files have a suffix of .pp. the puppet language is written into these files.

The site.pp file tells Puppet where and what configuration to load for our clients. These are stored in a directory called manifests under the /etc/puppet directory.

You can easily change the name and location of manifest directory and site.pp file by defining/changing the "manifestdir" and "manifest" configuration item in puppet.conf file. These can be found in the [main] are of puppet.conf file.

How to generate puppet.conf file?

2014-05-11T13:44:00.000+05:30

$ cd /etc/puppet/
$ puppet master --genconfig > puppet.conf

The above command will generate the puppet.conf file automatically.

The puppet.conf file is like an INI-style file and divided into various sections namely, [agent], [master], [main].

How to upgrade from JDK 1.6 to 1.7 on ubuntu?

2014-04-29T10:28:00.000+05:30

Its simple... Use the command

sudo apt-get install openjdk-7-jre

This will install JDK 1.7 on your ubuntu box. But there is a catch.. if you still run java -version, it will tell you that it is still using JDK 1.6.

So your system has actually been updated with JDK 1.7 but it is still not in use. There is a special script in ubuntu which allows you to check different versions present on the system and let's you choose the correct one you want to activate.

Here is how you can update that:

sudo update-alternatives --config java

This command will give you an output similar to below.. and asking you to choose which one you want to use..

There are 2 choices for the alternative java (providing /usr/bin/java).

Selection Path Priority Status
------------------------------------------------------------
* 0 /usr/lib/jvm/java-6-openjdk-amd64/jre/bin/java 1061 auto mode
1 /usr/lib/jvm/java-6-openjdk-amd64/jre/bin/java 1061 manual mode
2 /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java 1051 manual mode

Press enter to keep the current choice[*], or type selection number:

Simple.. Isn't it?

Puppet server hostname does not match certificate - can't authenticate.

2014-04-07T16:11:00.000+05:30

err: Could not retrieve catalog from remote server: Server hostname 'puppet' did not match server certificate; expected master

The agent configuration file seems to be communicating with a different server (master) name than that for which the puppet master has the certificate for. The only way to solve this is by re-generating the puppetmaster server certificate with the correct hostname as that defined in the agent configuration file.

To quickly test, another way is to change /etc/hosts file of agent and map the hostname and IP of the puppetmaster server.

Exiting; no certificate found and waitforcert is disabled

2014-04-07T13:56:00.001+05:30

When running command:

sudo puppet agent --test

Exiting; no certificate found and waitforcert is disabled

To solve this follow below instructions:

1. Login to the puppetmaster.
2. Run command "sudo puppet cert --list". It will list down your agent machine who have requested a certificate.
3. Sign the requests using command "sudo puppet cert --sign "ip-10-134-174-211.ap-southeast-1.compute.internal"" Where the IP-10.... is the hostname of you agent machine.

This should all be needed to resolve the issue.

err: Could not request certificate: Connection timed out - connect(2)

2014-04-07T13:47:00.003+05:30

When running below command on the agent machine:

sudo puppet agent --test

err: Could not request certificate: Connection timed out - connect(2)

Points to the problem that the Ports used for communication between the agent and master are not Open. Please open port 8140 TCP and check it again.

Recovering a Locked forgot administrator password for Windows EC2 Instance

2014-04-06T21:47:00.002+05:30

Its simple...

Create a new scratch EC2 instance running 2003 server, and log into it.
Get a note of volume attached to it and Shut down your lost instance (the one you can't log into) from the AWS console.
On the EC2 console, go to Elastic Block Store/Volumes and find the volume you find in step 2 and Detach it.
On that same volume, attach it to your new 2k3 EC2 instance. You can see it appear in Disk Manager and you'll get a new drive letter.
Go to NewDriveLetter:\Program Files\Amazon\Ec2ConfigService\Settings\ and edit config.xml with notepad
The first setting in the file is for the Ec2SetPassword plugin; change the State field from Disabledto Enabled. Save & close.
On the AWS console, Detach the volume from the new instance
Attach the volume to your original lost instance, as /dev/sda1 (Root volume)
Start the instance. After a few minutes, run the Get Windows Password function which will have the newly generated password.

Now if you have mapped that instance to C-name remember to change the corresponding URL because the ec2-url will change after the restart.

An interesting thing about AWS ELB's

2014-04-06T21:45:00.000+05:30

In last couple of days I discovered that ELB's are not designed to handle sudden spikes in traffic. What was most disturbing was that:

1. When there is a sudden spike, the ELB breaks down. An returns HTTP 404 messages.
2. The ELB gets into a bad state and remains like that for about 90 minutes.
3. The solution to this suggested by AWS Support team was to simply move out of ELB and have your own load balancer.
4. If it is just a testing spike, AWS can prewarm the ELB to handle that load but just for few weeks.

HAproxy or Nginx seems to be my obvious choice now if I have to move away from ELB. Doing High availability with HAProxy or even Nginx seems to be a little tricky but that's Okay if we have to move away from ELB.

#!/bin/bash

	# remove exited containers:
	docker ps --filter status=dead --filter status=exited -aq \| xargs -r docker rm -v

	# remove unused images:
	docker images --no-trunc \| grep '<none>' \| awk '{ print $3 }' \| xargs -r docker rmi

	# remove unused volumes:
	find '/var/lib/docker/volumes/' -mindepth 1 -maxdepth 1 -type d \| grep -vFf <(
	docker ps -aq \| xargs docker inspect \| jq -r '.[] \| .Mounts \| .[] \| .Name \| select(.)'
	) \| xargs -r rm -fr