Biju's Blog

BigQuery SQL

2021-05-26T16:09:00.001-05:00

Google bigquery-public-data

SELECT 
date,
channelGrouping,
totals.visits,
totals.transactions,
totals.transactionRevenue
FROM `bigquery-public-data.google_analytics_sample.ga_sessions_20170801`
WHERE channelGrouping in ('Organic Search', 'Direct')
ORDER BY transactionRevenue desc 
LIMIT 100

Deploy WordPress on AWS using Terraform and Chef

2019-08-20T00:00:00.000-05:00

Here is an example code for deploying WordPress on AWS using Terraform and Chef. Completely hands-free.

https://github.com/bijujo/aws_terraform_chef

Terraform script deploys following resources in us-west-2 region

VPC
Public Subnet
Internet Gateway
Security Group allowing ports 22, 80 and 443. Presently open to 0.0.0.0/0
Route table and route table association
RHEL7.6 EC2 instance with a user data script for cloud-init

Variables such as VPC/Subnet CIDR's, AMI, region, instance type, etc are specified in vars.tf file

Successful terraform execution displays public IP of the instance that is hosting WordPress

During EC2 deployment, userdata script performs the following steps

Chefdk installed
Chef recipes needed for installation of apache, php, mariadb and wordpress are cloned from my GitHub repo
Chef-solo executes all recipes specified in the run list
Recipes also create wordpress db, set root password for mariadb and configure wordpress config file for DB connection.

Prerequisites

Download latest version of terraform executable to the workstation/server from where you plan to execute the script
Install AWS CLI and configure credentials using "aws configure". Credentials are needed for terraform to initialize s3 backend
git clone the repo
terraform init, plan, apply

Convert SSL PEM certificate to PFX format using OpenSSL

2018-08-22T11:19:00.002-05:00

Azure application gateway need SSL certificate in
[root@server1]# openssl pkcs12 -export -out certificate.pfx -inkey privatekey.key -in certificate.srt
Enter Export Password:
Verifying - Enter Export Password:

[root@server1]#

AWS/Azure Metadata

2018-04-19T10:36:00.000-05:00

How to find out details about AWS account / Azure Subscription from within a guest OS

AWS

curl http://169.254.169.254/latest/dynamic/instance-identity/document

Azure

curl -H Metadata:true "http://169.254.169.254/metadata/instance?api-version=2017-08-01"

Docker/Container Installation and management

2017-02-23T22:59:00.001-06:00

Why is Docker/Container interesting?

First time when I heard about Docker/Container, I was thinking, oh another virtualization technology to the already overcrowded space. But once you started playing around you will see some real good interesting things. Few important points I noticed,

Lightweight. People who are familiar with Solaris zones already know this. Major portion of the OS (think about kernel and other supporting files/directories) is just one instance and is shared by all containers. So the container images are very small
Ease of installing an application in a docker image format. For example, if you want to install an nginx webserver or Jenkins server in RHEL/CentOS in the old fashion, you will have to run many commands to get the package, install it, configure it etc. In case of setting it up as a Docker, it all comes down to a single command (see below for some examples)
Ease of moving Docker images between hosts

This is a technology that is still getting features added and not mature as compared to other virtualization technologies like VMware ESX, KVM or XEN. But I feel there will be a huge demand for it in the future.

Docker engine installation

Here is an example of how to setup a Docker host and run few containers. I am using CentOS 7 as host and running a ubuntu, nginx and Jenkins containers. Of course for a production environment, you will need much more than basic Docker hosts. Most importantly a management platform like Kubernetes (Openshift) or Docker swarm. I have not yet tested any of these.

Install CentOS 7. I selected minimal install type.
Login to the server as root. Run all of the following commands as root.
Update the system

yum update -y

Leave SELinux in Enforcing mode
Leave firewalld.service running
Install Docker Engine

curl -fsSL https://get.docker.com/ | sh

Enable Docker service

systemctl enable docker

Start Docker service

systemctl start docker

We can get details about the Docker engine by running docker info command

docker info

Once Docker service is started, it will create a new interface called "docker0". You can verify this by running "/sbin/ip a"

3: docker0: mtu 1500 qdisc noqueue state DOWN

link/ether 02:42:75:d3:f4:6f brd ff:ff:ff:ff:ff:ff

inet 172.17.0.1/16 scope global docker0

valid_lft forever preferred_lft forever

"/usr/bin/docker" is the main command that will be using to manage Docker containers. Run docker --help to see list of all options available.
Run "docker ps -a" to list all Docker containers. Since we have not installed any containers yet, container list will be empty
Run "docker images" to see list of images locally available. It will be empty initially.
Lets start by downloading and running a ubuntu container on top of out CentOS host

[root@dockerhost ~]# docker run -it ubuntu bash

Unable to find image 'ubuntu:latest' locally

latest: Pulling from library/ubuntu

8aec416115fd: Pull complete

695f074e24e3: Pull complete

946d6c48c2a7: Pull complete

bc7277e579f0: Pull complete

2508cbcde94b: Pull complete

Digest: sha256:71cd81252a3563a03ad8daee81047b62ab5d892ebbfbf71cf53415f29c130950

Status: Downloaded newer image for ubuntu:latest

root@f02c14538663:/#

As you can see from the output, since ubuntu container image is not locally available, it has downloaded the image and installed it. And see the change in prompt. We are now within the container.

root@f02c14538663:/# df -h

Filesystem Size Used Avail Use% Mounted on

overlay 17G 1.3G 16G 8% /

tmpfs 912M 0 912M 0% /dev

tmpfs 912M 0 912M 0% /sys/fs/cgroup

/dev/mapper/cl-root 17G 1.3G 16G 8% /etc/hosts

shm 64M 0 64M 0% /dev/shm

tmpfs 912M 0 912M 0% /sys/firmware

root@f02c14538663:/# ps -aef

UID PID PPID C STIME TTY TIME CMD

root 1 0 0 03:50 ? 00:00:00 bash

root 11 1 0 04:22 ? 00:00:00 ps -aef

root@f02c14538663:/#

We can come out of the container without stopping it by pressing Ctrl p + Ctrl q

If we press Ctrl C, container will get stopped
Open another session to the CentOS host and run "docker ps" command. This will give details of the running container

[root@dockerhost ~]# docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

f02c14538663 ubuntu "bash" 33 minutes ago Up 33 minutes serene_tesla

[root@dockerhost ~]#

Rename a Container

As you can see Docker has assigned an arbitery name to the cotainer. In this case it is "serene_tesla". We can specify a name for the container as part of the docker run command with --name option or we can rename it later using "docker rename" command

[root@dockerhost ~]# docker rename serene_tesla ubuntu_test

[root@dockerhost ~]# docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

f02c14538663 ubuntu "bash" 51 minutes ago Up 51 minutes ubuntu_test

[root@dockerhost ~]#

To get more details about the container use, "docker inspect" command. We can specify the or the to specify the conatiner

docker inspect f02c14538663

To connect to a running container, run "docker attach" command

[root@dockerhost ~]# docker attach ubuntu_test

root@f02c14538663:/#

You will notice that there are only very limited command available within the container. For example, "ip" command is not available. But you can install the corresponding package by running apt-get install (Since this is ubuntu container)

root@f02c14538663:/# ip a

bash: ip: command not found

root@f02c14538663:/# apt-get install iproute2

Once the package is installed, ip command is available

root@f02c14538663:/# ip a

1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

valid_lft forever preferred_lft forever

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

4: eth0@if5: mtu 1500 qdisc noqueue state UP group default

link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0

inet 172.17.0.2/16 scope global eth0

valid_lft forever preferred_lft forever

inet6 fe80::42:acff:fe11:2/64 scope link

valid_lft forever preferred_lft forever

root@f02c14538663:/#

Create an NGINX webserver container

[root@dockerhost ~]# docker run --name mynginxweb1 -p 9000:80 -d -i nginx

Unable to find image 'nginx:latest' locally

latest: Pulling from library/nginx

5040bd298390: Pull complete

31123d939af1: Pull complete

23f1bdd267a9: Pull complete

Digest: sha256:4296639ebdf92f035abf95fee1330449e65990223c899838283c9844b1aaac4c

Status: Downloaded newer image for nginx:latest

512b7409f290983bf5c4026eadf9305d3ff818d167db407c5d5be45dcd833f5e

[root@dockerhost ~]# docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

512b7409f290 nginx "nginx -g 'daemon ..." 2 minutes ago Up 2 minutes 443/tcp, 0.0.0.0:9000->80/tcp mynginxweb1

776f46b6cfae ubuntu "bash" 2 days ago Up 2 days myubuntu

[root@dockerhost ~]#

In this example

--name mynginxweb1 - Name of the container

-p 9000:80 - We are mapping port 80 of the container to port 9000 of the host. To access the webserver we just need to point to the url docker_host_ip:9000

docker inspect mynginxweb1 will give more details about the container

Mount directory from host

Create an NGINX webserver container by specifying mount directory from host so that we have access to the NGINX folder/files directly from the host

In the first example, the container is using all standard files that come as part of the image. In this second example, we are adding little more flexibility by mounting host directories to the container.

[root@dockerhost ~]# docker run --name mynginxweb2 -v /nginx/html:/usr/share/nginx/html -v /nginx/etc/nginx:/etc/nginx -v /nginx/logs:/var/log/nginx -p 9001:80 -d -i nginx

[root@dockerhost ~]# docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

ab8afcd15598 nginx "nginx -g 'daemon ..." 15 minutes ago Up 15 minutes 443/tcp, 0.0.0.0:9001->80/tcp mynginxweb2

512b7409f290 nginx "nginx -g 'daemon ..." 46 minutes ago Up 46 minutes 443/tcp, 0.0.0.0:9000->80/tcp mynginxweb1

776f46b6cfae ubuntu "bash" 2 days ago Up 2 days myubuntu

[root@dockerhost ~]#

In this example we are telling docker to mount the folder /nginx/html on host as /usr/share/nginx/html within the container. Similarly /nginx/etc/nginx is mounted as /etc/nginx and /nginx/logs is mounted as /var/log/nginx within the container. Now for example if we want to make changes to the index.html file, we can make changes to "/nginx/html/index.html". Also to see access and error logs, we can look at the folder /nginx/logs

As you can see, I am forwarding port 80 of container to port 9001 of the host. Port 9000 is already used for the first nginx container.

[root@dockerhost ~]# ls /nginx/logs/

access.log error.log

[root@dockerhost ~]# cat /nginx/html/index.html

Welcome to mynginxweb2

[root@dockerhost ~]#

Run the following command to list details of the mounts

[root@dockerhost ~]# docker inspect -f '{{ .Mounts }}' mynginxweb2

[{bind /nginx/etc/nginx /etc/nginx true } {bind /nginx/logs /var/log/nginx true } {bind /nginx/html /usr/share/nginx/html true }]

[root@dockerhost ~]#

Configure data volumes

In the above example, we saw the option of doing bind mount of host directory in the container. Another option to manage storage is to use volume option.

[root@dockerhost ~]# docker run --name myubuntu3 -v /data3 -d -t -i ubuntu bash

1bf259beacdfbe068166dee7c9d1572d6ab99ecab8305c51f6be9d0ce5c1b47c

[root@dockerhost ~]#

[root@dockerhost ~]# docker attach myubuntu3

root@1bf259beacdf:/#

root@1bf259beacdf:/# df -h

Filesystem Size Used Avail Use% Mounted on

overlay 17G 1.6G 16G 10% /

tmpfs 912M 0 912M 0% /dev

tmpfs 912M 0 912M 0% /sys/fs/cgroup

/dev/mapper/cl-root 17G 1.6G 16G 10% /data3

shm 64M 0 64M 0% /dev/shm

tmpfs 912M 0 912M 0% /sys/firmware

root@1bf259beacdf:/#

We can see a new filesystem within the container called /data3.

root@1bf259beacdf:/data3# echo "Test file created in myubuntu3" > myubuntu3_file

root@1bf259beacdf:/data3# ls

myubuntu3_file

The advantage of using bind mount or volume is that even when the container is stopped or deleted, the data is still available.

[root@dockerhost volumes]# docker stop myubuntu3

myubuntu3

[root@dockerhost volumes]# docker rm myubuntu3

myubuntu3

[root@dockerhost volumes]# cat /var/lib/docker/volumes/f1f1ca5e0d757014326da683649914694df5b20a332693557b209c835e69aa2c/_data/myubuntu3_file

Test file created in myubuntu3

[root@dockerhost volumes]#

Build a custom Docker image

Custom Docker images can be built using "docker build" command (from the specification in Dockerfile) or by "docker commit" command based on an existing image.

Using Dockerfile. In this example, we are creating a custom ubuntu image with a /appdata volume. When we create a container using this image, /appdata volume will appear as a filesystem within the container.

[root@dockerhost ~]# docker images

REPOSITORY TAG IMAGE ID CREATED SIZE

nginx

ubuntu latest f49eec89601e 5 weeks ago 129 MB

[root@dockerhost ~]# cat Dockerfile

FROM ubuntu

RUN useradd bijujo

VOLUME /appdata

[root@dockerhost ~]#

Dockerfile supports many more configuration keywords. These are few examples.

[root@dockerhost ~]# docker build -t my_ubuntu .

Sending build context to Docker daemon 23.55 kB

Step 1/3 : FROM ubuntu

---> f49eec89601e

Step 2/3 : RUN useradd bijujo

---> Running in 28fa2d1d173b

---> 03964f50ccdf

Removing intermediate container 28fa2d1d173b

Step 3/3 : VOLUME /appdata

---> Running in 45aba3dc0faa

---> acd468a6475a

Removing intermediate container 45aba3dc0faa

Successfully built acd468a6475a

[root@dockerhost ~]#

[root@dockerhost ~]# docker images

REPOSITORY TAG IMAGE ID CREATED SIZE

my_ubuntu latest acd468a6475a 16 seconds ago 130 MB

nginx latest db079554b4d2 12 days ago 182 MB

ubuntu latest f49eec89601e 5 weeks ago 129 MB

[root@dockerhost ~]#

[root@dockerhost ~]# docker run --name myubuntu1 -d -t -i my_ubuntu bash

8daebfa0d4f966733f9fa5d7bb5d6a2ee2baa948bbde6923ab30edfa726d146b

[root@dockerhost ~]# docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

8daebfa0d4f9 my_ubuntu "bash" 4 seconds ago Up 3 seconds myubuntu1

ab8afcd15598 nginx "nginx -g 'daemon ..." 45 hours ago Up 44 hours 443/tcp, 0.0.0.0:9001->80/tcp mynginxweb2

776f46b6cfae ubuntu "bash" 3 days ago Up 2 hours myubuntu

[root@dockerhost ~]# docker attach myubuntu1

root@8daebfa0d4f9:/#

root@8daebfa0d4f9:/# df -h

Filesystem Size Used Avail Use% Mounted on

overlay 17G 1.6G 16G 10% /

tmpfs 912M 0 912M 0% /dev

tmpfs 912M 0 912M 0% /sys/fs/cgroup

/dev/mapper/cl-root 17G 1.6G 16G 10% /appdata

shm 64M 0 64M 0% /dev/shm

tmpfs 912M 0 912M 0% /sys/firmware

root@8daebfa0d4f9:/# cd /appdata/

root@8daebfa0d4f9:/appdata# ls

root@8daebfa0d4f9:/appdata# echo "File created on myubuntu1" > myubuntu1_file

root@8daebfa0d4f9:/appdata# ll

total 4

drwxr-xr-x. 2 root root 28 Feb 28 22:47 ./

drwxr-xr-x. 1 root root 61 Feb 28 22:46 ../

-rw-r--r--. 1 root root 26 Feb 28 22:47 myubuntu1_file

root@8daebfa0d4f9:/appdata#

On the host, if you search for "myubuntu1_file" you will see the file under /var/lib/docker/volumes path

[root@dockerhost ~]# find / -name myubuntu1_file

/var/lib/docker/volumes/aa4240428453e4224dc9fb2adef6b263a1c0da8867b35665444fccf372743b4a/_data/myubuntu1_file

[root@dockerhost ~]#

Using Docker commit option to build a custom image.

We noticed earlier that the default ubuntu image doesn't have "ip" or "ifconfig" commands since corresponding packages are not installed. We installed it using apt-get. If we want to make sure that all future ubuntu containers that we want to build should have these commands, we can create a custom image based on the container with these packages already installed.

In this example, "myubuntu" is the container that we installed iproute2 package so that we can run "ip" commands. We are building a new Docker image called "ubuntu_ip" using the container "myubuntu"

[root@dockerhost ~]# docker images

REPOSITORY TAG IMAGE ID CREATED SIZE

my_ubuntu latest acd468a6475a 21 minutes ago 130 MB

nginx latest db079554b4d2 12 days ago 182 MB

ubuntu latest f49eec89601e 5 weeks ago 129 MB

[root@dockerhost ~]# docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

ab8afcd15598 nginx "nginx -g 'daemon ..." 46 hours ago Up 44 hours 443/tcp, 0.0.0.0:9001->80/tcp mynginxweb2

776f46b6cfae ubuntu "bash" 3 days ago Up 2 hours myubuntu

f02c14538663 ubuntu "bash" 5 days ago Up 3 minutes ubuntu_test

[root@dockerhost ~]# docker commit myubuntu ubuntu_ip

sha256:ff22b997d3869a4246aea0df97afb8d459b4f53c0350cd269d730380b69fedc3

[root@dockerhost ~]# docker images

REPOSITORY TAG IMAGE ID CREATED SIZE

ubuntu_ip latest ff22b997d386 4 seconds ago 177 MB

my_ubuntu latest acd468a6475a 22 minutes ago 130 MB

nginx latest db079554b4d2 12 days ago 182 MB

ubuntu latest f49eec89601e 5 weeks ago 129 MB

[root@dockerhost ~]# docker run --name myubuntu2 -d -t -i ubuntu_ip bash

8a5e1e31bddd096900a3f2a75ef01f709753c4a411d09046003623415b3f5619

[root@dockerhost ~]# docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

8a5e1e31bddd ubuntu_ip "bash" 13 seconds ago Up 12 seconds myubuntu2

ab8afcd15598 nginx "nginx -g 'daemon ..." 46 hours ago Up 44 hours 443/tcp, 0.0.0.0:9001->80/tcp mynginxweb2

776f46b6cfae ubuntu "bash" 3 days ago Up 2 hours myubuntu

f02c14538663 ubuntu "bash" 5 days ago Up 4 minutes ubuntu_test

[root@dockerhost ~]# docker attach myubuntu2

root@8a5e1e31bddd:/#

root@8a5e1e31bddd:/# ip a

1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

valid_lft forever preferred_lft forever

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

41: eth0@if42: mtu 1500 qdisc noqueue state UP group default

link/ether 02:42:ac:11:00:04 brd ff:ff:ff:ff:ff:ff link-netnsid 0

inet 172.17.0.4/16 scope global eth0

valid_lft forever preferred_lft forever

inet6 fe80::42:acff:fe11:4/64 scope link

valid_lft forever preferred_lft forever

root@8a5e1e31bddd:/#

Configure Docker containers to start automatically during host server reboot

Containers by default will NOT start when host operating system/docker service starts.

[root@dockerhost ~]# docker inspect -f '{{ .HostConfig.RestartPolicy}}' mynginxweb1

{no 0}

[root@dockerhost ~]#

To configure it to start automatically we can change the "restart" parameter.

[root@dockerhost ~]# docker update --restart=unless-stopped mynginxweb2

Use the following command to verify the parameter is changed.

[root@dockerhost ~]# docker inspect -f '{{ .HostConfig.RestartPolicy}}' mynginxweb2

{unless-stopped 0}

[root@dockerhost ~]#

Reboot the host. Once the host is up run "docker ps" command to verify that container is started.

Install a Jenkins Server Container

In this example, we are going to name the container "myjenkins" and going to forward port 8080 of the container to port 9002 of the Docker host.

[root@dockerhost ~]# docker run --name myjenkins -d -p 9002:8080 jenkins

Unable to find image 'jenkins:latest' locally

latest: Pulling from library/jenkins

5040bd298390: Already exists

fce5728aad85: Pull complete

76610ec20bf5: Pull complete

60170fec2151: Pull complete

e98f73de8f0d: Pull complete

11f7af24ed9c: Pull complete

c39ff935bbe4: Pull complete

6e03296a7302: Pull complete

401a90041717: Pull complete

a24a58f3e779: Pull complete

3f05866f4c9a: Pull complete

194637a1b12c: Pull complete

f5405da54999: Pull complete

66c5531af30f: Pull complete

c61bd2df6e6a: Pull complete

85969c669ffc: Pull complete

a22b8d2b56ba: Pull complete

11b90bc5b10c: Pull complete

351c841a8598: Pull complete

Digest: sha256:2bfbdb2c2606a3fa036bd7c0392003cd5aeb45978cc7fa05ba14b0412b4e9a1c

Status: Downloaded newer image for jenkins:latest

eb263883a450fb3db12eb8a46a735c364485b14dbf9d51464b4c254ab96d6845

[root@dockerhost ~]# docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

eb263883a450 jenkins "/bin/tini -- /usr..." 30 seconds ago Up 28 seconds 50000/tcp, 0.0.0.0:9002->8080/tcp myjenkins

8a5e1e31bddd ubuntu_ip "bash" 33 minutes ago Up 33 minutes myubuntu2

ab8afcd15598 nginx "nginx -g 'daemon ..." 46 hours ago Up 45 hours 443/tcp, 0.0.0.0:9001->80/tcp mynginxweb2

776f46b6cfae ubuntu "bash" 4 days ago Up 3 hours myubuntu

f02c14538663 ubuntu "bash" 5 days ago Up 37 minutes ubuntu_test

[root@dockerhost ~]#

To configure Jenkins, we will need the default password. We can use "docker logs" commands to see the information that was generated during docker run

[root@dockerhost ~]# docker logs myjenkins

Connect to the Docker Host IP:9002 and you will be prompted to enter the administrative password. Once that is provided you will be provided option to select the plugins to install and create username and password and will finally be landed to the homepage

Get Shell access to Container using docker exec

If you want to get into bash shell within a running container ( for exampel an nginx or a jenkins), you can run

docker exec -it <container_id> /bin/bash

[root@dockerhost ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
94745bbf0904 corbinu/docker-phpmyadmin "/bin/sh -c phpmya..." 35 hours ago Up 35 hours 0.0.0.0:9004->80/tcp root_phpmyadmin_1
2a2c1cfce06f wordpress "docker-entrypoint..." 35 hours ago Up 35 hours 0.0.0.0:9003->80/tcp root_wordpress_1
b39c905e7c33 mariadb "docker-entrypoint..." 35 hours ago Up 35 hours 3306/tcp root_wordpress_db_1
eb263883a450 jenkins "/bin/tini -- /usr..." 2 days ago Up 2 days 50000/tcp, 0.0.0.0:9002->8080/tcp myjenkins
8a5e1e31bddd ubuntu_ip "bash" 2 days ago Up 2 days myubuntu2
ab8afcd15598 nginx "nginx -g 'daemon ..." 4 days ago Up 4 days 443/tcp, 0.0.0.0:9001->80/tcp mynginxweb2
776f46b6cfae ubuntu "bash" 6 days ago Up 2 days myubuntu
f02c14538663 ubuntu "bash" 8 days ago Up 2 days ubuntu_test
[root@dockerhost ~]#

To connect get shell prompt to ab8afcd15598, run

[root@dockerhost ~]# docker exec -it ab8afcd15598 /bin/bash

root@ab8afcd15598:/# df -h

Filesystem Size Used Avail Use% Mounted on

overlay 17G 4.4G 13G 26% /

tmpfs 912M 0 912M 0% /dev

tmpfs 912M 0 912M 0% /sys/fs/cgroup

/dev/mapper/cl-root 17G 4.4G 13G 26% /etc/nginx

shm 64M 0 64M 0% /dev/shm

tmpfs 912M 0 912M 0% /sys/firmware

root@ab8afcd15598:/# ps -aef

UID PID PPID C STIME TTY TIME CMD

root 1 0 0 Feb27 ? 00:00:00 nginx: master process nginx -g daemon off;

nginx 5 1 0 Feb27 ? 00:00:00 nginx: worker process

root 6 0 0 14:40 ? 00:00:00 /bin/bash

root 16 0 0 14:53 ? 00:00:00 /bin/bash

root 58 0 0 15:01 ? 00:00:00 /bin/bash

root 63 58 0 15:01 ? 00:00:00 ps -aef

root@ab8afcd15598:/#

Press Ctrl p and Ctrl q to come out of the shell.

Instead of running "/bin/bash", we can simply run some commands and get the output from container also.

[root@dockerhost ~]# docker exec -it ab8afcd15598 uname -a

Linux ab8afcd15598 3.10.0-514.6.1.el7.x86_64 #1 SMP Wed Jan 18 13:06:36 UTC 2017 x86_64 GNU/Linux

[root@dockerhost ~]# docker exec -it ab8afcd15598 ps -aef

UID PID PPID C STIME TTY TIME CMD

root 1 0 0 Feb27 ? 00:00:00 nginx: master process nginx -g

nginx 5 1 0 Feb27 ? 00:00:00 nginx: worker process

root 6 0 0 14:40 ? 00:00:00 /bin/bash

root 16 0 0 14:53 ? 00:00:00 /bin/bash

root 58 0 0 15:01 ? 00:00:00 /bin/bash

root 88 0 0 15:04 ? 00:00:00 ps -aef

[root@dockerhost ~]#

Docker Compose

Docker compose is a way to organize multiple containers into a group. For example, take the case of an application that includes a database container, app container, and a web container. With docker compose we can group all three containers so that the startup and shutdown will be handled together and it also allows intra-container links.

There is a good writeup about how to setup a wordpress site using wordpress and mariadb containers.

https://www.digitalocean.com/community/tutorials/how-to-install-wordpress-and-phpmyadmin-with-docker-compose-on-ubuntu-14-04

docker-compose is not part of the docker-engine. It has to be installed separately.

[root@dockerhost ~]# curl -L "https://github.com/docker/compose/releases/download/1.11.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 600 0 600 0 0 959 0 --:--:-- --:--:-- --:--:-- 958
100 8053k 100 8053k 0 0 2203k 0 0:00:03 0:00:03 --:--:-- 3225k
[root@dockerhost ~]#

Make the file executable

[root@dockerhost ~]# chmod u+x /usr/local/bin/docker-compose

As given the tutorial link, I have created the following docker-compose.yml file. Please note that I am using different port numbers.

[root@dockerhost ~]# cat docker-compose.yml
wordpress:
image: wordpress
links:
- wordpress_db:mysql
ports:
- 9003:80
wordpress_db:
image: mariadb
environment:
MYSQL_ROOT_PASSWORD: password
phpmyadmin:
image: corbinu/docker-phpmyadmin
links:
- wordpress_db:mysql
ports:
- 9004:80
environment:
MYSQL_USERNAME: root
MYSQL_ROOT_PASSWORD: password

[root@dockerhost ~]#

Once docker-compose.yml file is created, run "docker-compose up". Specifying "-d" for detached mode will get us back to the # prompt once the command execution is complete. This command will download all three docker images will bring up three containers as specified in the .yml file.

[root@dockerhost ~]# docker-compose up -d

To see status of containers created using docker-compose

[root@dockerhost ~]# docker-compose ps
Name Command State Ports
-----------------------------------------------------------------------------------
root_phpmyadmin_1 /bin/sh -c phpmyadmin-start Up 0.0.0.0:9004->80/tcp
root_wordpress_1 docker-entrypoint.sh apach ... Up 0.0.0.0:9003->80/tcp
root_wordpress_db_1 docker-entrypoint.sh mysqld Up 3306/tcp

Group of containers can be stopped andstarted using docker-compose commands

[root@dockerhost ~]# docker-compose stop
Stopping root_phpmyadmin_1 ... done
Stopping root_wordpress_1 ... done
Stopping root_wordpress_db_1 ... done
[root@dockerhost ~]# docker-compose start
Starting wordpress_db ... done
Starting phpmyadmin ... done
Starting wordpress ... done
[root@dockerhost ~]#

Create Docker image from scratch

This is an example of how to create a Docker image from scratch. ie without downloading anything from Docker hub or RedHat registry.

Please refer http://developerblog.redhat.com/2014/05/15/practical-introduction-to-docker-containers/ for details

We are going to create a Docker image for CentOS 7. For this, I created a small VM and performed a minimal (base) install. Once OS installation is done, take tar image of the server excluding /proc and /sys

[root@localhost ~]# tar --numeric-owner --exclude=/proc --exclude=/sys -cvf centos7.tar /

This will create a tar file in your current working directory. Copy it over to the Docker host

[root@localhost ~]# scp centos7.tar <docker_host_ip>:/root/

In the Docker host, run "docker import" to create new image based on the tar file

[root@dockerhost ~]# cat centos7.tar |docker import - centos7

sha256:8b0047557a9d7cfa84b083759c7d980d077a597d4447493b40daa3ae3c3c3c84

[root@dockerhost ~]# docker images

REPOSITORY TAG IMAGE ID CREATED SIZE

centos7 latest 8b0047557a9d About a minute ago 998 MB

wordpress latest 6eaad793d5c8 2 days ago 400 MB

ubuntu_ip latest ff22b997d386 2 days ago 177 MB

my_ubuntu latest acd468a6475a 2 days ago 130 MB

mariadb latest 56741a13bbb9 3 days ago 393 MB

nginx latest db079554b4d2 2 weeks ago 182 MB

jenkins latest ee06af673de2 3 weeks ago 712 MB

ubuntu latest f49eec89601e 5 weeks ago 129 MB

corbinu/docker-phpmyadmin latest 5c663962b799 11 months ago 473 MB

Create new Container using the new Docker image

[root@dockerhost ~]# docker run --name mycentos7 -itd -i centos7 bash

871760f72097f83834d1c06694677a5430747e89ea7b283c6ad27a7d1b8bed18

[root@dockerhost ~]#

[root@dockerhost ~]# docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

871760f72097 centos7 "bash" 9 seconds ago Up 6 seconds mycentos7

573bffcf80a8 corbinu/docker-phpmyadmin "/bin/sh -c phpmya..." 3 hours ago Up 46 minutes 0.0.0.0:9004->80/tcp root_phpmyadmin_1

51ae49acb1bc wordpress "docker-entrypoint..." 3 hours ago Up 46 minutes 0.0.0.0:9003->80/tcp root_wordpress_1

5a0f0c97fe64 mariadb "docker-entrypoint..." 3 hours ago Up 46 minutes 3306/tcp root_wordpress_db_1

eb263883a450 jenkins "/bin/tini -- /usr..." 2 days ago Up 2 days 50000/tcp, 0.0.0.0:9002->8080/tcp myjenkins

8a5e1e31bddd ubuntu_ip "bash" 2 days ago Up 2 days myubuntu2

ab8afcd15598 nginx "nginx -g 'daemon ..." 4 days ago Up 4 days 443/tcp, 0.0.0.0:9001->80/tcp mynginxweb2

776f46b6cfae ubuntu "bash" 6 days ago Up 2 days myubuntu

f02c14538663 ubuntu "bash" 8 days ago Up 2 days ubuntu_test

[root@dockerhost ~]# docker attach mycentos7

[root@871760f72097 /]# df -h

Filesystem Size Used Avail Use% Mounted on

overlay 17G 6.7G 11G 40% /

tmpfs 912M 0 912M 0% /dev

tmpfs 912M 0 912M 0% /sys/fs/cgroup

/dev/mapper/cl-root 17G 6.7G 11G 40% /etc/hosts

shm 64M 0 64M 0% /dev/shm

tmpfs 912M 0 912M 0% /sys/firmware

[root@871760f72097 /]#

Docker Swarm

Docker swarm allows configuring Docker hosts in a cluster. A docker swarm consists for manager nodes and worker nodes. In this example we will setup a three node swarm (docker1, docker2 and docker3). docker1 is manager node. docker2 and 3 are worker nodes. By default manager node also act as worker node. There are specific firewall port requirements, but in this testing I have disabled firewalld service in all three nodes. Please refer below URL for details

https://docs.docker.com/engine/swarm/key-concepts/

Install docker engine on all three nodes

[root@docker1 ~]# curl -fsSL https://get.docker.com/ | sh

Enable and start docker service on all three nodes

[root@docker1 ~]# systemctl enable docker; systemctl start docker

[root@docker1 ~]#

On the host that you are planning to make as manager (In this example docker1) run following command. <advertise-addr> is it's own IP address

[root@docker1 ~]# docker swarm init --advertise-addr 192.168.91.70

Swarm initialized: current node (6cn44zmuce73la97tyqrcqjhd) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join \

    --token SWMTKN-1-4czy3m569ucgt695birdkokcwvt4802q2sb9h8ctl2wzdcv6n8-1vzmja59q31psrgqnideee3ys \

    192.168.91.70:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

[root@docker1 ~]#

On the worker nodes, run the docker swarm join command

[root@docker2 ~]# docker swarm join --token SWMTKN-1-4czy3m569ucgt695birdkokcwvt4802q2sb9h8ctl2wzdcv6n8-1vzmja59q31psrgqnideee3ys 192.168.91.70:2377

This node joined a swarm as a worker.

[root@docker2 ~]#

[root@docker3 ~]# docker swarm join --token SWMTKN-1-4czy3m569ucgt695birdkokcwvt4802q2sb9h8ctl2wzdcv6n8-1vzmja59q31psrgqnideee3ys 192.168.91.70:2377

This node joined a swarm as a worker.

[root@docker3 ~]#

Once both worker nodes joined to cluster, run "docker node ls" on manager node to see the cluster status

[root@docker1 ~]# docker node ls

ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS

6cn44zmuce73la97tyqrcqjhd *  docker1   Ready   Active        Leader

j57lkcyj4vc7u4vyney4hgj5t    docker3   Ready   Active

z5b1ii3kwq07ge7t2o4i1nyxb    docker2   Ready   Active

[root@docker1 ~]#

Add nginx containers to docker swarm

On the manager host, run the following command. With this command we are creating a service called "mynginxweb1" with "2" instances of nginx containers running. Port number 80 of container is mapped to port 9000 of ALL three nodes of the cluster. That is an important point. As you can see below, one of the node has no container, but still the host is listening on port 9000 and we will be able to get nginx homepage by pointing to port 9000 of any three of the nodes.

[root@docker1 ~]# docker service create --replicas 2 --name mynginxweb1 --publish 9000:80 nginx

pfbziahyqkrzr5b9o87zo4fjn

[root@docker1 ~]#

[root@docker1 ~]# docker ps

CONTAINER ID        IMAGE                                                                           COMMAND                  CREATED             STATUS              PORTS               NAMES

6f7dcbecaa19        nginx@sha256:52a189e49c0c797cfc5cbfe578c68c225d160fb13a42954144b29af3fe4fe335   "nginx -g 'daemon ..."   37 seconds ago      Up 37 seconds       80/tcp, 443/tcp     mynginxweb1.1.kdy4bddg0p0vbkvp6jqwkdvii

[root@docker1 ~]# netstat -na|grep 9000

tcp6       0      0 :::9000                 :::*                    LISTEN

[root@docker1 ~]#

[root@docker2 ~]# docker ps

CONTAINER ID        IMAGE                                                                           COMMAND                  CREATED              STATUS              PORTS               NAMES

79c51dfffdcf        nginx@sha256:52a189e49c0c797cfc5cbfe578c68c225d160fb13a42954144b29af3fe4fe335   "nginx -g 'daemon ..."   About a minute ago   Up About a minute   80/tcp, 443/tcp     mynginxweb1.2.3wq177je1sebuwkiwa5alr5xr

[root@docker2 ~]# netstat -na|grep 9000

tcp6       0      0 :::9000                 :::*                    LISTEN

[root@docker2 ~]#

[root@docker3 ~]# docker ps

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

[root@docker3 ~]# netstat -na|grep 9000

tcp6       0      0 :::9000                 :::*                    LISTEN

[root@docker3 ~]#

We can increase the number of instances by using "docker service scale" option. In this example we are increasing the number of instances from 2 to 4. As a result two containers are created in docker3 host.

[root@docker1 ~]# docker service ls

ID            NAME         MODE        REPLICAS  IMAGE

pfbziahyqkrz  mynginxweb1  replicated  2/2       nginx:latest

[root@docker1 ~]# docker service scale mynginxweb1=4

mynginxweb1 scaled to 4

[root@docker1 ~]# docker service ls

ID            NAME         MODE        REPLICAS  IMAGE

pfbziahyqkrz  mynginxweb1  replicated  4/4       nginx:latest

[root@docker1 ~]# docker ps

CONTAINER ID        IMAGE                                                                           COMMAND                  CREATED             STATUS              PORTS               NAMES

6f7dcbecaa19        nginx@sha256:52a189e49c0c797cfc5cbfe578c68c225d160fb13a42954144b29af3fe4fe335   "nginx -g 'daemon ..."   12 minutes ago      Up 12 minutes       80/tcp, 443/tcp     mynginxweb1.1.kdy4bddg0p0vbkvp6jqwkdvii

[root@docker1 ~]#

[root@docker3 ~]# docker ps

CONTAINER ID        IMAGE                                                                           COMMAND                  CREATED             STATUS              PORTS               NAMES

c26c669471b2        nginx@sha256:52a189e49c0c797cfc5cbfe578c68c225d160fb13a42954144b29af3fe4fe335   "nginx -g 'daemon ..."   19 seconds ago      Up 16 seconds       80/tcp, 443/tcp     mynginxweb1.3.xhmdkvj5wqy0jw97z116gnkbv

6ab47c7015ce        nginx@sha256:52a189e49c0c797cfc5cbfe578c68c225d160fb13a42954144b29af3fe4fe335   "nginx -g 'daemon ..."   19 seconds ago      Up 17 seconds       80/tcp, 443/tcp     mynginxweb1.4.hw0hbbbunwhcfabgett3p2f4z

[root@docker3 ~]#

-- Add Topics

Backup and Restore of containers
Transfer container image between hosts

IP Subnets

2017-01-24T12:26:00.002-06:00

Default Subnets

Class A 1 - 126 /8
Class B 128 - 191 /16
Class C 192 - 223 /24
Class D 224 - 239 Used for Multicast
Class E 240 - 255 Reserved

Private IP address range

10.0.0.0 - 10.255.255.255 255.0.0.0 or /8 Class A

172.16.0.0 - 172.31.255.255 255.255.0.0 or /16 Class B

192.168.0.0 - 192.168.255.255 255.255.255.0 or /24 Class C

RHEL7/CentOS 7 Kickstart - asknetwork / Supply network parameters during installation

2016-11-14T16:03:00.002-06:00

With Kickstart, most of the time it will be handsfree installation where all of the configuration parameters are predefined in Kickstart configuration files and network configuration details are provided by DHCP. But in some cases where you may want to specify network parameters (IP, Subnet Mask, Default Gateway, DNS servers) during installation.

RHEL5/6 Kickstart has "asknetwork" parameter that can be used for this purpose. With this keyword during installation, system will prompt you to enter network details. In RHEL7, "asknetwork" is deprecated. Alternate option available is to specify the network parameters at the boot prompt itself as below

boot: linux ip=ip_address::gateway_address:netmask:fqdn_of_host:primary_ethernet_device:none dns=dns_server_ip

example

boot: linux ip=192.168.91.92::192.168.91.2:255.255.255.0:centosnew.test.com:ens33:none dns=192.168.91.2

Please note there are two "::" between IP address and Gateway.

RHEL/CentOS 7 - analyze server startup performance using systemd-analyze

2016-11-13T17:12:00.002-06:00

If you have a RHEL/CentOS 7 server that's taking more than the normal time to finish boot process, /usr/bin/systemd-analyze will be very handy for troubleshooting.

Two most useful options are "systemd-analyze plot" and "systemd-analyze blame".

systemd-analyze plot will generate SVG graphic file showing service startup details. I created the ouput file with .html extension just to make it easy to open with firefox or Chrome browser.

/usr/bin/systemd-analyze plot > centos7_boot-details.html

To illustrates the delay a service can cause during server startup, I created a systemd service called "bad-service" which simply sleeps for 30 seconds. Details of the service below.

[root@centos7compute1 ~]# cat /etc/systemd/system/bad-service.service
[Unit]
Description=Test systemd service
#After=default.target
After=sshd.service

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/bad_script.sh

[Install]
#WantedBy=default.target
WantedBy=multi-user.target

[root@centos7compute1 ~]# cat /usr/local/sbin/bad_script.sh
#!/bin/bash
echo "Bad script started at `date`" > /var/log/bad_script.log
sleep 30
echo "Bad script ended at `date`" >> /var/log/bad_script.log
[root@centos7compute1 ~]#
[root@centos7compute1 ~]# systemctl status bad-service.service
bad-service.service - Test systemd service
   Loaded: loaded (/etc/systemd/system/bad-service.service; enabled)
   Active: inactive (dead) since Sun 2016-11-13 01:06:01 EST; 16h ago
Process: 1361 ExecStart=/usr/local/sbin/bad_script.sh (code=exited, status=0/SUCCESS)
Main PID: 1361 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/bad-service.service

Nov 13 01:06:01 centos7compute1.test.com systemd[1]: Started Test systemd service.
[root@centos7compute1 ~]#

/usr/bin/systemd-analyze blame will display services sorted by time it takes to init.

[root@centos7compute1 ~]# /usr/bin/systemd-analyze blame
         30.154s bad-service.service
          6.422s network.service
          5.268s tuned.service
          2.912s openvswitch-nonetwork.service
          2.624s postfix.service
          2.590s avahi-daemon.service
          2.304s libvirtd.service
          2.261s systemd-logind.service
          1.922s iptables.service
          1.898s kdump.service
          1.441s rsyslog.service
          1.361s proc-fs-nfsd.mount
          1.196s iprinit.service
          1.193s iprupdate.service
          1.188s lvm2-monitor.service
          1.174s rhel-loadmodules.service
          1.108s var-lib-nfs-rpc_pipefs.mount

RHEL 6.x Disable Transparent Huge Pages (THP)

2015-02-16T13:28:00.000-06:00

Oracle recommends disabling transparent hugepages in RHEL 6.x and use standard hugepages.

To disable transparent hugepages in RHEL 6.x, append "transparent_hugepage=never" to the correct kernel line in /boot/grub/grub.conf file.

To verify whether THP is enabled or disabled, use the following commands.

Transparent Hugepages (THP) enabled

sh-4.1# cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
[always] madvise never
sh-4.1# grep -i AnonHugePages /proc/meminfo
AnonHugePages: 399360 kB
sh-4.1#
sh-4.1#
sh-4.1# grep transparent_hugepage grub.conf
kernel /vmlinuz-2.6.32-504.3.3.el6.x86_64 ro root=/dev/mapper/rootvg-root_lv rd_NO_LUKS rd_LVM_LV=rootvg/swap_lv LANG=en_US.UTF-8 rd_LVM_LV=rootvg/root_lv rd_NO_MD SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=128M rhgb quiet rd_NO_DM rhgb quiet
sh-4.1#

Transparent Hugepages (THP) disabled

sh-4.1# cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
always madvise [never]
sh-4.1#
sh-4.1# grep -i AnonHugePages /proc/meminfo
AnonHugePages: 0 kB
sh-4.1#
sh-4.1#
sh-4.1# grep transparent_hugepage /boot/grub/grub.conf
kernel /vmlinuz-2.6.32-504.3.3.el6.x86_64 ro root=/dev/mapper/rootvg-root_lv rd_NO_LUKS rd_LVM_LV=rootvg/swap_lv LANG=en_US.UTF-8 rd_LVM_LV=rootvg/root_lv rd_NO_MD SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=128M rhgb quiet rd_NO_DM rhgb quiet transparent_hugepage=never

sh-4.1#

Unix / Linux convert upper case to lower or lower to upper

2015-02-09T17:02:00.000-06:00

Here is an example to convert contents of a file to all uppercase or all lowercase

A file called testfile that has both upper and lower case letters

sh-3.2$ cat testfile
ABCDEFGH
IJKlmnOPQ
abcdefgh
ijkLMNopQR
sh-3.2$

To convert it to all lowercase

sh-3.2$ tr '[A-Z]' '[a-z]' < testfile
abcdefgh
ijklmnopq
abcdefgh
ijklmnopqr
sh-3.2$
sh-3.2$

To convert all to uppercase

sh-3.2$ tr '[a-z]' '[A-Z]' < testfile
ABCDEFGH
IJKLMNOPQ
ABCDEFGH
IJKLMNOPQR
sh-3.2$
sh-3.2$

To conver to uppercase and redirect the o/p to a file called uppercase

sh-3.2$ tr '[a-z]' '[A-Z]' < testfile > uppercase
sh-3.2$
sh-3.2$ cat uppercase
ABCDEFGH
IJKLMNOPQ
ABCDEFGH
IJKLMNOPQR
sh-3.2$

Linux free memory explanation

2015-01-29T14:58:00.000-06:00

From RedHat documentation. Explanation of free memory usage.

# free -m
             total       used       free     shared    buffers     cached
Mem:          1000        900        100          0        350        350
-/+ buffers/cache:        200        800

In this example, as far as applications are concerned the system is using only 200MB of memory and has 800MB free and available for use if needed (as long as no anonymous memory maps or shared memory regions are in place).

For example,

Total Physical Memory = 1000 M
Physically Used Memory = 900 M
Actual used memory = 200 M
buffers = 350 M
cached = 350 M
Physically Free Memory = 100 M
Memory free for Applications = 800 M

The items to note here are:

= + + = 200 + 350 + 350 = 900 M

= - - - = 1000 - 200 - 350 - 350 = 100 M

= - = 1000 - 200 = 800 M

= - - = 900 - 350 - 350 = 200 M

Puppet and Satellite 6 - Notes

2015-01-26T14:53:00.000-06:00

Just some notes from my Satellite 6 POC.

I did the Satellite 6 install by downloading the DVD image and mount it.

katello-installer --capsule-dhcp true --capsule-dhcp-interface eth1 --capsule-tftp true --capsule-tftp-servername lmpocsatl01 --capsule-puppet --capsule-puppet-ca

Good part with katello-installer, if you have not included any capsule module during first run of katello-installer, you can rerun katello-installer with the additional module. This will not overwrite the modules that is already installed, but just adds the additional module.

Example to add dhcp and tftp, just run

katello-installer --capsule-dhcp true --capsule-dhcp-interface eth0 --capsule-tftp true --capsule-tftp-servername lmpocsatl01

While installing, make sure /var has sufficient space. Lot of contents goto /var
add user script

Monitor -> Content Dashboard - Error - You do not currently have access to any organizations. Please contact an administrator to get permission to access an organization - To check with RedHat

Some useful awk, cut, sed examples

2015-01-26T12:44:00.000-06:00

You want to get a unique list of clients connected to a specific port of a server. Use the following command

netstat -na|grep 22|grep ESTABLISHED|awk '{print $2}'|cut -d. -f 1,2,3,4|sort -n|uniq

In this example, awk '{print $2}' is getting the second field of the netstat output

cut -d. -f 1,2,3,4 is getting fields 1,2,3 and 4 with "." field separator

Cut the last character in a string using sed. For example when working with emcpower pseudo devices, you want to get rid of the last character.

To remove "a" from "emcpower71a"

[root@server1]# echo emcpower71a|sed s'/.$//'

emcpower71

[root@server1]#

To get specific columns from /etc/shadow. For example you want to share information with an auditor about password controls, but you don't want to share the encrypted password field. (This is tested in Linux)

cat /etc/shadow | awk 'BEGIN {FS=":" };{ print $1":" $3":" $4":" $5":" $6":" $7}'

RedHat Satellite 5.x. Register/Unregister thru command line non-interactively

2015-01-21T22:58:00.000-06:00

Registration

To register clients non-interactively, we have to first create activation keys. We have to create keys for each of the channel that the clients will be subscribed to.

For example if you expect new clients of RHEL 5.10, RHEL 6.5 and RHEL 6.6, you should create three activation keys corresponding to each channell.

To create activation keys, click on Systems --> Activation Keys --> Create New Key. Give a description to identify which version the key corresponds to. Example OrgName_RHEL_6.5. Select the base channel the key is associated with. Select proper entitlements and click "Create Activation Key" in the bottom right corner.

Once it's created, select the key that you just created from the main activation keys window. If you would like your systems to register to child channel, select that from the "Child Channels" tab. For example we attach all our clients to Tools child channel. Click on "Update Key". Note down the "Key" value. An example from our server. We will need this information while registering the clients.

Key:	2-

Repeat the same steps of each of the versions that you are planning to use.

Once the keys are created, you can either use command line similar to below to below for registering clients. Assuming satellite.example.com is the satellite server.

mkdir -p /usr/share/rhn/
wget http://satellite.example.com/pub/RHN-ORG-TRUSTED-SSL-CERT -O /usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT
perl -npe 's/RHNS-CA-CERT/RHN-ORG-TRUSTED-SSL-CERT/g' -i /etc/sysconfig/rhn/*
if [ -f /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release ]; then
rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
fi
/usr/sbin/rhnreg_ks --serverUrl=https://satellite.example.com/XMLRPC --sslCACert=/usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT --activationkey=2-6b32e442d5e30f7fc153283b1ba33c45

___________________

To make it more easy, you can create a small script that will execute the proper activation command based on server version. Here is an example

#!/bin/bash
mkdir -p /usr/share/rhn/
wget http://satellite.example.com/pub/RHN-ORG-TRUSTED-SSL-CERT -O /usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT
perl -npe 's/RHNS-CA-CERT/RHN-ORG-TRUSTED-SSL-CERT/g' -i /etc/sysconfig/rhn/*
if [ -f /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release ]; then
rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
fi
version=`cat /etc/redhat-release|awk '{print $7}'`
case $version in
5.10)
rhnreg_ks --force --serverUrl=https://satellite.example.com/XMLRPC --sslCACert=/usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT --activationkey=2-fcea66d0e624763f6c040111b9c1e86f
;;
5.11)
rhnreg_ks --force --serverUrl=https://satellite.example.com/XMLRPC --sslCACert=/usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT --activationkey=2-bbe80dfd5a1708cab147c68d85b11f62
;;
6.4)
rhnreg_ks --force --serverUrl=https://satellite.example.com/XMLRPC --sslCACert=/usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT --activationkey=2-d5846e0a8dda484fe1364b28c85467e3
;;
6.5)
rhnreg_ks --force --serverUrl=https://satellite.example.com/XMLRPC --sslCACert=/usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT --activationkey=2-6b32e442d5e30f7fc153283b1ba33c45
;;
7.0)
rhnreg_ks --force --serverUrl=https://satellite.example.com/XMLRPC --sslCACert=/usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT --activationkey=2-42341d5466973937fa52c84b62a35e14
;;
*)
echo "Unsupported version"
;;
esac

################################################################################

Unregistering / Client Deletion

We will use use one python script to get sysid of the client that we want to unregister and another python script to perform the deletion. These python scripts are NOT developed by me. I got the scripts from two different blogs and I just created a small wrapper around it. Thanks to the developers of these python scripts.

We have to run satellite_unregister.sh on the client that we want to unregister from Satellite. I would suggest putting the script in an NFS share, mount it on the clients are execute it.

satellite_client_list.py will get sysid of the client that we have to unregister

satellite_client_unregister.py will do the unresgiter

sh-3.2# cat satellite_unregister.sh
#!/bin/bash
# Get System ID of the client and unregister it from Satellite
#
/root/scripts/unregister/satellite_client_list.py|grep `hostname`|awk '{print $3}' > /tmp/sysid
SYSID=`cat /tmp/sysid`
/root/scripts/unregister/satellite_client_unregister.py $SYSID
#

sh-3.2#
sh-3.2# cat satellite_client_list.py
#!/usr/bin/python

import xmlrpclib

SATELLITE_URL = "https://satellite.example.com/rpc/api"
SATELLITE_LOGIN = "satellite_user_name"
SATELLITE_PASSWORD = "password"

client = xmlrpclib.Server(SATELLITE_URL, verbose=0)
key = client.auth.login(SATELLITE_LOGIN, SATELLITE_PASSWORD)
list = client.system.listUserSystems(key)

for group in list:
sysname = group.get('name')
sysid = group.get('id')
print '%s - %s ' %(sysname, sysid)

client.auth.logout(key)
sh-3.2#

sh-3.2# cat satellite_client_unregister.py
#!/usr/bin/python

import xmlrpclib
import sys

SATELLITE_URL = "http://satellite.example.com/rpc/api"
SATELLITE_LOGIN = "satellite_user_name"
SATELLITE_PASSWORD = "password"

client = xmlrpclib.Server(SATELLITE_URL, verbose=0)
key = client.auth.login(SATELLITE_LOGIN, SATELLITE_PASSWORD)

for system in sys.argv:
# SKIP the first element of the args as it's always the script name
if system == sys.argv[0]:
continue

print 'asking RHN to delete :%s' %(system)
sysid = int(system)
client.system.deleteSystems(key,sysid)

client.auth.logout(key)
sh-3.2#
sh-3.2#

Verizon FIOS with your own wireless router

2015-01-13T21:20:00.000-06:00

Recently I switched from Timewarner to Verizon FIOS. With timewarner, I was using timewarner supplied cable modem along with CISCO E1000 wireless router. Once I disconnect timewarner, I returned the modem.

My apartment was already wired for FIOS. Verizon was charging around $6/month rent for their modem/router and my cisco router was lying idle so decided to start using my own router. These are the steps. Please note that I am using only internet. If you want to use TV/DVR, you will need the verizion supplied router or a similar one. This procedure assumes that you have basic knowledge of how to connect/disconnect ethernet/coaxial cables. Do at your own risk.

Switch off the verizon supplied router/modem and disconnect the co-axial cable connection between ONT and router
Connect the cisco wi-fi router Wan port to the Ethernet port of OTN using a RH-45 cable and poweron
Ethernet port of OTN will be disabled by default. Call verzion support and ask them to enable the Ethernet port. They will enable ethernet port and will disable co-axial port.
SSID of CISCO router will be available in Wi-fi networks. Connect to that network and you are all set.
Ask verizon to send return kit and once you have it return the verizon router.

vCAC/vRA Linux post install script

2015-01-13T20:32:00.000-06:00

As part of provisioning of Linux VM's thru vCAC we created a template with base OS install. We will do customization and different agent installation using post install script. vCAC workflow will execute the post install script using the mechanism explained below.

All of our agents (Backup, Monitoring, RedHat Satellite registration etc) are kept in an NFS share that is accessible from various subnets. We created a single post_install.sh that will call individual silent agent install script. So, for example the post install script is available in

nfs_server:/share/scripts/post_install.sh

The Linux template contains a master script file in /root/scripts/master.sh. The purpose of this file is to do two functions. Mount the nfs share and execute the /share/scripts/post_install.sh script.

Contents of /root/scripts/master.sh

mount nfs_server:/share /mnt
/mnt/scripts/post_install.sh

Now create a workflow in vCAC and setup these values

As part of VM deployment, once the VM is created from template and is accessible on network, vCAC will call the /root/scripts/master.sh script. This will mount the nfs share and execute /mnt/scripts/post_install.sh

Advantage with this approach is that to add or remove components from post install (for example you want to add a new agent software as part of build) we don't have to make any modifications to template or vCAC workflow. We just have to update /share/scripts/post_install.sh that is available as NFS share.

Unix / Linux. Sort based on column

2014-12-05T09:36:00.003-06:00

Here is an example file "test".

If you would like to sort the contents of test based on column three, use

sort -nk3 test

To sort in reverse order

sort -nrk 3 test

sh-3.2$

sh-3.2$ cat test

10 12 14

11 13 10

3 8 19

5 4 25

25 38 9

sh-3.2$

sh-3.2$ sort -nk 3 test

25 38 9

11 13 10

10 12 14

3 8 19

5 4 25

sh-3.2$

sh-3.2$ sort -nrk 3 test

5 4 25

3 8 19

10 12 14

11 13 10

25 38 9

sh-3.2$

Install and configure bind DNS server in Linux

2014-12-02T20:23:00.000-06:00

Following are the steps to setup a DNS server using bind

In this example we will setup internal domain test.com.

rhel6 is the hostname of the RHEL server and 192.168.91.61 is the IP address.

Install the bind package. yum install bind

Update /etc/named.conf Below is sample file from my server. Changes that I made to default file are highlighted in yellow.

[root@rhel6 ~]# cat /etc/named.conf

// named.conf

// Provided by Red Hat bind package to configure the ISC BIND named(8) DNS

// server as a caching only nameserver (as a localhost DNS resolver only).

// See /usr/share/doc/bind*/sample/ for example named configuration files.

options {

listen-on port 53 { 127.0.0.1;192.168.91.61; };

listen-on-v6 port 53 { ::1; };

directory "/var/named";

dump-file "/var/named/data/cache_dump.db";

statistics-file "/var/named/data/named_stats.txt";

memstatistics-file "/var/named/data/named_mem_stats.txt";

allow-query { localhost;192.168.91.0/24; };

recursion yes;

dnssec-enable yes;

dnssec-validation yes;

dnssec-lookaside auto;

/* Path to ISC DLV key */

bindkeys-file "/etc/named.iscdlv.key";

managed-keys-directory "/var/named/dynamic";

};

logging {

channel default_debug {

file "data/named.run";

severity dynamic;

};

zone "." IN {

type hint;

file "named.ca";

};

zone"test.com" IN {

type master;

file "fwd.test.com";

allow-update { none; };

};

zone"91.168.192.in-addr.arpa" IN {

type master;

file "rev.test.com";

allow-update { none; };

};

include "/etc/named.rfc1912.zones";

include "/etc/named.root.key";

[root@rhel6 ~]#

Create forward and reverse files in /var/named/

In this example, filenames are /var/named/fwd.test.com and /var/named/rev.test.com. See that I have added some example DNS entries here. ubuntu, ubuntu2, Win2012, Win2008 and bsd1. You need to update these files based on your servers name and IP address. In the reverse file, note the "." after hostname. This is important.

[root@rhel6 ~]# cat /var/named/fwd.test.com

$TTL 86400

@ IN SOA rhel6.test.com. root.test.com. (

2014051001 ; serial

3600 ; refresh

1800 ; retry

604800 ; expire

86400 ; minimum

)

@ IN NS rhel6.test.com.

rhel6 IN A 192.168.91.61

ubuntu IN A 192.168.91.60

ubuntu2 IN A 192.168.91.62

win2012 IN A 192.168.91.63

win2008 IN A 192.168.91.64

bsd1 IN A 192.168.91.65

[root@rhel6 ~]# cat /var/named/rev.test.com

$TTL 86400

@ IN SOA rhel6.test.com. root.test.com. (

2014051001 ; serial

3600 ; refresh

1800 ; retry

604800 ; expire

86400 ; minimum

)

@ IN NS rhel6.test.com.

rhel6 IN A 192.168.91.61

60 IN PTR ubuntu.test.com.

61 IN PTR rhel6.test.com.

62 IN PTR ubuntu2.test.com.

63 IN PTR win2012.test.com.

64 IN PTR win2008.test.com.

65 IN PTR bsd1.test.com.

[root@rhel6 ~]#

Verify the configuration files using /usr/sbin/named-checkzone

[root@rhel6 ~]# /usr/sbin/named-checkzone test.com /var/named/fwd.test.com

zone test.com/IN: loaded serial 2014051001

[root@rhel6 ~]# /usr/sbin/named-checkzone test.com /var/named/rev.test.com

zone test.com/IN: loaded serial 2014051001

[root@rhel6 ~]#

Set named to start automatically on boot

/sbin/chkconfig named on

Start named service

[root@rhel6 ~]# /sbin/service named restart

Stopping named: [ OK ]

Starting named: [ OK ]

[root@rhel6 ~]#

Now the server is ready. You can test by configuring a client to use this DNS server. Here is an example from an ubuntu server.

root@ubuntu:~# cat /etc/network/interfaces

# This file describes the network interfaces available on your system

# and how to activate them. For more information, see interfaces(5).

# The loopback network interface

auto lo

iface lo inet loopback

# The primary network interface

auto eth0

iface eth0 inet static

address 192.168.91.60

netmask 255.255.255.0

gateway 192.168.91.2

#dns-nameservers 192.168.91.2

dns-search test.com

dns-nameservers 192.168.91.61

root@ubuntu:~#

root@ubuntu:~# nslookup ubuntu2

Server: 192.168.91.61

Address: 192.168.91.61#53

Name: ubuntu2.test.com

Address: 192.168.91.62

root@ubuntu:~#

For adding new server entries to DNS update both fwd and rev file and reload named service.

To enable or disable verbose logging --> /usr/sbin/rndc querylog

CentOS repo for RHEL

2014-11-28T21:58:00.004-06:00

As you know, updates from RedHat are available only if you are a paying customer. What if you want to install some packages in your test RHEL server. You can get compatible packages from CentOS repository by setting up a repo file as below.

Example for RHEL 6.

vi /etc/yum.repos.d/centos.repo

[centos]
name=CentOS $releasever - $basearch
baseurl=http://ftp.heanet.ie/pub/centos/6/os/$basearch/
enabled=1
gpgcheck=0

__________________________________________________________

RHEL 5

vi /etc/yum.repos.d/centos.repo

[centos]
name=CentOS $releasever - $basearch
baseurl=http://ftp.heanet.ie/pub/centos/5/os/$basearch/
enabled=1
gpgcheck=0

Linux - How to remove old eth interface references

2014-11-19T14:11:00.000-06:00

If you have added/removed NIC to RHEL/CentOS VM's you may end up getting eth1, eth2, eth3 etc instead of eth0. To keep the configuration standard you may want to make sure the first NIC is always getting eth0. Perform the following steps.

cd /etc/udev/rules.d/

mv 70-persistent-net.rules /root/

cd /etc/sysconfig/network-scripts/

edit ifcfg-ethX file and replace the following line

DEVICE=ethX to DEVICE=eth0

Reboot the VM

You will be able to see eth0 in ifconfig -a o/p

Sendmail/Postfix mail domainname masquerade.

2014-04-22T12:15:00.001-05:00

Mails which are sent from Unix/Linux servers may not get delivered to corporate mail accounts if the from domainname is not a valid one. For example if you sent a mail from root account of server1, the from address may appear as root@server1.localdomain

Domain masquerade is the solution for this. Using this option we are basically making the from address a valid domainname. For example root@yourdomain.com

Solaris sendmail

svcadm disable sendmail

svcadm disable sendmail-client

cd /etc/mail/cf/cf

cp -p sendmail.mc myfile.mc

vi myfile.mc

Add these lines after line starting with define

MASQUERADE_AS(`yourdomain.com')dnl
FEATURE(`masquerade_envelope')dnl
FEATURE(`masquerade_entire_domain')dnl
FEATURE(`always_add_domain')dnl
MASQUERADE_DOMAIN(`yourdomain.com')dnl

/usr/ccs/bin/make myfile.cf

sendmail -C myfile.cf -v user@yourdomain.com < /dev/null - To test that domainname is correctly appearing using the new .cf file

cp myfile.cf /etc/mail/sendmail.cf

svcadm enable svc:/network/smtp:sendmail

svcadm enable svc:/network/sendmail-client:default

mailx -s "Test 1" -v user@yourdomain.com < /dev/null

____________________________________________________________

For Linux postfix

postfix stop

cd /etc/postfix

cp -p main.cf main.cf.bak

vi main.cf and add the below line to mydomain section

#mydomain = domain.tld
mydomain = yourdomain.com

save and exit

postfix start

mailx -s "Test 1" -v user@yourdomain.com < /dev/null

Sun Oracle ILOM web pages not displayed in Firefox

2014-04-16T11:16:00.002-05:00

On newer versions of Firefox for Linux, if you open Sun Oracle ILOM, webpages will not be completely displayed.

https://bugzilla.mozilla.org/show_bug.cgi?id=694931

Workaround is to create/append /.mozilla/firefox/ccpzkekr.default/chrome/userContent.css file with following lines.

Example. For root user

______________________________________________________
cat /root/.mozilla/firefox/ccpzkekr.default/chrome/userContent.css

@namespace url(http://www.w3.org/1999/xhtml);
#mainpage { visibility: visible !important; }

______________________________________________________

ESX/ESXi host in a VMware High Availability cluster fails to enter maintenance mode and stops at 2%

2014-03-10T16:59:00.000-05:00

In my ESXi 5.5 Lab I was trying to put a host into maintenance mode, but the progress was staying at 2%.

I was able to manually vmotion the VM's between the ESXi hosts.

As per VMware dodumentation, I tried disabling Admission control and HA itself, with no luck.

Finally it was resolved by clicking on "Override Storage Recommendation" and "Apply Recommendations" in the DRS tab.

VxVM data copy thru volume mirror and diskgroup split

2014-01-24T14:43:00.000-06:00

Solaris 10/ VxVM 5.0 MP3RP4

Requirement is to take a clone of data from Server A and mount it in Server B. Our initial plan was to clone the disk at SAN level. But later we decided to get this done at VxVM level. SAN team presented new LUN's to both source and target server. The process is to add the new LUN to source diskgroup, mirror volumes, disassociate the plex, create new volume from the disassociated plex, split the diskgroup, deport the diskgroup and finally import it and mount in target server. Below are the steps.

Source diskgroup in Server A is testdg
Present new LUN to both servers (source and target)
Label the disk and initialize it
vxdg -g testdg adddisk testdgmirror= - Add new disk to diskgroup
vxassist -g testdg mirror testvol1 testdgmirror - Mirror the volume to be copied over
vxplex -g testdg dis testvol1-02 - Disassociate the plex that is created on new disk
vxmake -g testdg -U fsgen vol testvol1new plex=testvol1-02 - Create a new volume using the disassociated plex that is in DISABLED state
vxdg split testdg testdg1 testvol1new - Split the diskgroup. This command will create a new diskgroup "testdg1" that has volume testvolnew
Deport the diskgroup from Server A
Import the diskgroup on ServerB
Start the volumes
Mount the filesystem

How to get X Windows after doing sudo su – to another user

2013-10-14T16:22:00.000-05:00

[user1@server1 ~]$ echo $DISPLAY

localhost:10.0

[user1@server1 ~]$ xauth list

server1/unix:10 MIT-MAGIC-COOKIE-1 ea1897dde6549a56d7323640ed37111b

Run /usr/openwin/bin/xclock and make sure it's displayed

Now sudo su - to become root or any other user

[root@server1 ~]# xauth add server1 unix:10 MIT-MAGIC-COOKIE-1 ea1897dde6549a56d7323640ed37111b

xauth: creating new authority file /root/.Xauthority

[root@server1 ~]# DISPLAY=localhost:10.0;export DISPLAY

[root@server1 ~]# xclock