YoYoClouds

5 reasons why AWS rules!

2018-05-07T12:50:00.000+05:30

5 Reasons Why AWS Rules

In this article, we will look at 5 reasons why AWS is the place to go for all your cloud services. Read on to find out more!

Cloud computing has evolved a lot over the years and has almost become a mainstream part of everyday society. From storing large amounts of data to having burst compute capacity at your fingertips and enterprise software applications available on demand any time anywhere, these are just some of the key benefits that clouds provide today. At the forefront of this new way of computing is Amazon Web Services (AWS).

AWS is a comprehensive public cloud computing platform made up of remote computing services called web services that make up the platform. Once a part of the e-commerce giant Amazon.com, it wasn't until 2006 that AWS became a separate entity of its own.

With data centers located in USA, Europe, Brazil, Singapore, Japan, China, and Australia, AWS operates globally to offer a variety of web-based products and services on an on-demand and pay-per-use basis. AWS provides a variety of mechanisms which end users can connect to and leverage its services. The most common form of interaction is the web-based dashboard, AWS Management Console.

5 reasons that keep AWS on top

AWS boasts an array of Infrastructure as a Service (IaaS) and Platform as a Service (PaaS) capabilities, with a diverse customer base and a strong network of software vendors who have integrated their solutions with AWS. Here is our list of top 5 reasons why AWS rules the cloud market:

Deployment Speed

We all have gone through the pain of long wait for provisioning a hosted web service. However, AWS shrinks this waiting time from hours to mere minutes. With multiple strategies for infrastructure provisioning ranging from Amazon EC2, Amazon EBS, and Amazon S3 to Amazon RDS, AWS offers promptness and quick availability as well as greater flexibility to deploy your website or application.

AWS also enables automation in its deployment services, namely:

AWS Elastic Beanstalk
AWS CloudFormation
AWS OpsWorks
AWS CodeCommit
AWS CodePipeline
AWS CodeDeploy
Amazon EC2 Container Service

Scalable and Reliable

With AWS, you can scale your application with ease during periods of increased demand. AWS also prides itself on the fact that it is reliable and provides a better experience whilst keeping the costs under control.

AWS OpsWorks, Auto Scaling, and Elastic Load Balancing are some of the AWS tools can help you scale your business up or down on demand. AWS provides on-demand services giving you the opportunity to flex your resources as and when needed.

Flexible

One of the best features of AWS is its flexibility. It lets you select from a plethora of services such as:

Operating system
Programming language
Web application platform
Database

AWS provides a virtual environment that lets you load the software and services required by your application. It simplifies the migration process for already existing applications and provides options for building new ones. With faster scalability, de-facto standard APIs, and mind-blowing deployment speed, you can have a completely customized ready-to-use solution in under 10 minutes.

On the Go Pricing

We all know that hosting a service on cloud sounds like a pretty massive investment, but this is resolved by AWS’s pricing structure. AWS offers “pay for what you use”. This means that you pay only for the resources you use. This is a refreshing approach from AWS, making it more user-friendly, and allowing you to choose your infrastructure and service needs with zero capital expenditure. This cost-effectiveness has really helped startups grow their business on AWS. Users can even try the AWS Free Tier.

With AWS Free Tier, you can enjoy hands-on experience with the AWS platform, products, and services for free. This no-cost option includes offers that expire after 1 year of signing up and others that never expire. This is one of the best ways for a new user to try out AWS.

Secure

Security is a core requirement for any application whether it is hosted in an on-premise data center or a cloud such as AWS. It is a fundamental service that protects your applications and data from a variety of cyber-attacks, security breaches, accidental or deliberate data deletions, theft, and much more.

The obvious doubt lingering in your mind while considering AWS must be is it really secure? Is it safe to use AWS? The answer is a big yes!

Apart from private subnets, multi-factor authentication (MFA), Isolate GovCloud, and encrypted data storage, AWS uses different layers of security to safeguard its resources:

Physical data center security – The data centers are housed at non-disclosed locations and their entry is strictly controlled, managed, logged and audited on a regular basis
Virtualization and OS security – AWS regularly patches and updates virtualization operating systems against a variety of cyber-attacks
Regulatory compliance – AWS is certified for security and data protection in accordance with various industry and government requirements

AWS also enhances its security with IAM (Identity and Access Management). Using this authentication greatly reduces the risk.

If this article has got you hooked on AWS, you should definitely read my fourth book titled: AWS Administration - The Definitive Guide

This book is for those who want to learn and leverage the rich plethora of services provided by AWS. It guides you through some of the most popular AWS services, such as EC2, Elastic Beanstalk, EFS, CloudTrail, Redshift, EMR, Data Pipeline, and IoT using a simple, real-world, application-hosting example. Each chapter is designed to provide you with maximum information about each AWS service, coupled with easy to follow, hands-on steps, best practices, tips, and recommendations. By the end of the book, you will be able to create a highly secure, fault-tolerant, and scalable environment for your applications to run on.

NetApp ONTAP 9 Lab: The Complete Guide

2016-10-22T20:20:00.001+05:30

NetApp ONTAP 9 Lab: The Complete Guide

Admit it, there's nothing like building your own super lab with cool software! And things get even better when someone goes out all the way to create an equally super lab guide!

That's exactly what my good friend Neil Anderson has done.. You can now download a copy of his Build Your Own NetApp ONTAP 9 Lab absolutely free!

The guide talks about setting up your very own NetApp ONTAP 9 (Previously called "Clustered DataNTAP") using VMware Workstation. It walks you through the basic topology of the Lab and the components required to set it all up. I guess the best part of the entire guide is the meticulous attention to detail, explaining each step of the way with screenshots.. that really makes things a lot simpler and easier to follow (I mean you really can't go wrong with this!)

Here is a gist of what the guide has to offer:

Introduction
Lab Topology Diagram
Lab Topology Notes
IP Addressing Tables
Lab Notes
VMware Workstation Player Install
VyOS Router Build
NetApp Simulator Build
Windows Server Build
Linux Build
SuperPutty Install

If you follow thguide down to the letter you can have your entire Get your copy today right HERE!

Setting up a Secure FTP Server on AWS

2016-09-16T14:18:00.002+05:30

Setting up a Secure FTP (SFTP) Server on AWS

So here's a quick and easy guide to setting up a simple Secure FTP Server on a Ubuntu 14.04 AWS EC2 instance.

Secure FTP Servers are an ideal mechanism for transfering files to and fro between trusted users. For this tutorial, I'm using a simple Ubuntu 14.04 AMI running on a t2medium instance type.

AMI: Ubuntu Server 14.04 LTS (HVM), SSD Volume Type - ami-2d39803a

Instance Hostname: US-EAST-FTPSERVER-01

Username: ubuntu

Key name: myppkfile.ppk

Important: Make sure you update your OS using the sudo apt-get update command before following this tutorial

# sudo apt-get install vsftpd

Create a new user (in my case its called as the blueuser)

#sudo adduser blueuser

Also create the SSH directory for your newly created user (NOTE: The SSH directory is a hidden directory):

# sudo mkdir /home/blueuser/.ssh

With this done we now need to a keypair for our blueuser. There are many ways to do this, but for now 'm going to use the AWS Console itself and create one for me quickly. From the EC2 management dashboard, select the Pairs option and click on New Key pair. provide a suitable name (blue-ftpserver) and click Create. Save the .pem locally on your machine.

Next, we need to create a public key from our newly downloaded key pair. I'm using PuttyGen for the same. Click on Load to load your blue-ftpserver.pem file.

It prompt you with a notice. Click on OK

Now here's the important part. Click on Save Public Key option as shown below. Provide a suitable name for your public key and save it locally on your system.

Next, open your saven public key using any editor (I'm using Sublime Text) and add the following two text as shown below:

First, add the text "ssh-rsa" to the beginning of the key and then add the key's name to the end of the key as shown. There are no quotations whatsoever. Save the file.

Next, in your FTP Server, create the authorized_keys file for your user:

# sudo vi /home/blueuser/.ssh/authorized_keys

Copy and paste the entire content of the Public Key file that we modified earlier here. Save and exit the file.

Next, run the following commands to set the correct permissions on your folders:

# sudo chmod 700 /home/blueuser/.ssh/

# sudo chmod 600 /home/blueuser/.ssh/authorized_keys

# sudo chown -R blueuser:blueuser /home/blueuser/.ssh/

Now, we test. To do so, transfer the PEM file the FTP Server instance using any tool such as WinSCP. This is the PEM file that was created using the AWS Management Console (blue-ftpserver)

Once transferred run the following command to verify whether your new user can login to the FTP server securely using the PEM file.

# sftp -o IdentityFile=blue-ftpserver.pem blueuser@ec2.compute-1.amazonaws.com

If all goes well, your newly created user should get authenticated and you should see the FTP Server prompt as shown below. If for some reason you get a permission denied error, then please check the contents of your user's authorized_keys file. make there are no new line characters in it.

You can also use this newly created user to SSH your Ubuntu instance as well. Type in the following command:

# ssh -i blue-ftpserver.pem blueuser@ec2.compute-1.amazonaws.com

Your new user should get ssh access to the FTP server instance as shown below.

But to get really secure, you may want to disable this SSH access for your new user and restrict it only for FTP access. To do so, run the following commands as shown:

Create a group dedicated for FTP users:

# sudo groupadd sftponly

Add your newly created user to this group:

# sudo adduser blueuser sftponly

Edit the sshd_config file and append the following data towards the end of the file:

# sudo vi /etc/ssh/sshd_config

# Paste the following content towards the end of file

Match group sftponly

ChrootDirectory /home/%u

X11Forwarding no

AllowTcpForwarding no

ForceCommand internal-sftp

Save and exit the file.

Apply the necessary permissions and restart the ssh service for the changes to take effect:

# sudo chown root:root /home/blueuser

# sudo chown -R blueuser:blueuser /home/blueuser/.ssh

# sudo /etc/init.d/ssh restart

Try logging on the FTP serer now using the same username and PEM file as earlier. Your user should have SFTP access however the SSH access should be blocked out as shown below:

Well, there you have it.. a really simple way to setup a secure FTP server the AWS environment. Stay tuned for more such tutorials coming your way soon!

Infographic: 10 Questions CIOs Should Ask Before Moving To Cloud

2016-09-13T18:20:00.002+05:30

With so many organizations looking to reap the advantages of Clouds, it is equally important to understand and answer some other important issues such as potential risks, hidden costs, cloud readiness etc.

Infographic created by SingleHop

Analyzing Apache Logs using ELK

2016-08-24T19:14:00.000+05:30

Analyzing Apache Logs using ELK

In our previous post, we looked at few simple steps to get an ELK stack up and running on an EC2 instance in AWS. In this tutorial I'll be showing you how to setup a simple Logstash pipeline to send your Apache web server logs over to the ELK instance.

To get started, you will need an AWS instance with Apache web server installed in it. In my case, I have launched an instance (hostname:: client1) in the same region and AZ as that of my ELK server instance.

The first thing that you should always do before you start working with your EC2 instances is to update its packages. In my case, Im using a Ubuntu 14.04 AMI so the commands will be Ubuntu native.

# sudo apt-get update

Next, install the Apache web server packages if not already installed.

# sudo apt-get install apache2

With Apache installed, let's run a simple test to check whether Apache's error and access logs are working as expected or not. In the same terminal, type in the the following command to simulate someone accessing your web server:

# curl localhost

NOTE: You can alternatively open up a web browser and type in your Apache Web Server instance's Public IP address or DNS as well.

You should see a similar output with a bunch of <html> tags. This is actually the welcome page of your Apache web server.

Next up, tail into your Apache web server's access logs and you should see one entry created as shown below:

# sudo tail /var/log/apache2/access.log

This means that our Apache web server and its logs are working.

Next up, we install Logstash on our client Apache instance. Logstash will basically act as a spout consuming the apache access logs from the client, processing them based on certain filters and then forwarding them to our previously created ELK instance for further analysis.

# sudo apt-get install openjdk-7-jre-headless

NOTE: You can optionally install Oracle Java here as well instead of using the default OpenJDK packages.

Verify whether Java was successfully installed or not:

# java -version

With Java installed, we can now move forward with Logstash's installation. First we add the

# echo "deb http://packages.elastic.co/logstash/1.5/debian stable main" | sudo tee -a /etc/apt/sources.list

NOTE: You can substitute 1.5 with 2.1 to install the latest Logstash packages.

Make sure you update the packages on the system using the update command:

# sudo apt-get update

Install the Logstash package:

# sudo apt-get install logstash

For this tutorial, we will use Logstash to forward the Apache web server instance's access.log file to our ELK server.

Here's a snippet of the configuration pipeline:

# sudo vi /etc/logstash/conf.d/apache-access.conf

input {

file {

path => '/var/log/apache2/access.log'

}

filter {

grok {

match => { "message" => "%{COMBINEDAPACHELOG}" }

}

output {

stdout { codec => rubydebug }

}

The snippet file is pretty straightforward. The conf file will watch the Apache access log file (path) for events, parses them with a special filter pattern using grok (COMBINEDAPACHELOG) and will print those events to standard console output (stdout).

Run Logstash using the following command:

# sudo /opt/logstash/bin/logstash -f \

/etc/logstash/conf.d/apache-access.conf

You should see the message as "Logstash startup completed".

To test whether its working, open up a new terminal window for your Apache web server instance and try the "curl localhost" command once again. You should see logs getting printed on the terminal as shown below:

Now to send these logs over to the ELK server instance, all you need to do is edit the apache-access.conf file and update the output section as shown below:

output {

elasticsearch { "host" => "<IP_ADDRESS>" } #Provide the IP address of your ELK instance

}

Restart Logstash to make the changes take effect:

# sudo service logstash restart

Open up Kibana and hit the Discovery tab as shown below. You should now see your Apache web server access logs streaming in as shown below:

Well that's about it from this post for now.. in the next post, I'll be showing you how to setup a Production scale ELK server on AWS using EC2 instances, so stick around.. more coming your way soon!

Setup a simple ELK on AWS in under 10 minutes!

2016-08-23T19:51:00.001+05:30

Setup a simple ELK on AWS in under

10 minutes!

So here I am... back at it again after a good long break! For this tutorial, Im going to show you how easy and simple it is to setup a basic working ELK on AWS!

ELK? Not heard of it? well if you haven't heard of it or tried it out yet, I suggest you do! ELK is actually an acronym for ElasticSearch, Logstash and Kibana.. fancy names but trust me.. together they form a really powerful log analysis tool.

ElasticSearch: Built on top of Apache Lucene, ElasticSearch is the work engine behind ELK that performs real-time data extractions and analysis on structured as well as unstructured data. To know more read HERE
Logstash: Logstash is a tool that can ingest logs, process them and forward them to another system such as ElasticSearch. Logstash comes with a huge supply of inputs, filters, codecs and outputs that can be used to consume virtually any type of logs from web servers, to syslogs, to error lgos etc. To know more read HERE
Kibana: Kibana is a visualization tool that can be used to visualize trends, patterns, read and interpret your log data. To know more read HERE

In this tutorial, I'll be walking you simple steps to get started with a single node ELK setup on AWS infrastructure.

Source: https://www.cartoonstock.com/cartoonview.asp?catref=jhen301

To get started, you will need an AWS account. You can signup for AWS and get one year of services for free* T&C apply of course!

For this particular tutorial, Im using a Ubuntu AMI and running this setup on a t2.medium instance (CPU: 2 -- RAM: 4 GB -- HDD: 20 GB).

Make sure the instance that you launch has a Public IP attached to it. Login to the instance using the username "ubuntu" and the PPK file as the password.

Run the following command as shown:

Update the packages using the following command:

# sudo apt-get update

Next, install Java. You can either go for Oracle supplied Java or even the OpenJDK version.

# sudo apt-get install openjdk-7-jre-headless

Once installed, check the java version using the following command:

# java -version

With the basics out of the way, we now go ahead and install ElasticSearch out first. To do so, type in the following command to import ElasticSearch's public GPG key:

# wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Next, we add the ElasticSearch to the source list:

# echo "deb http://packages.elastic.co/elasticsearch/1.7/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list

NOTE: You can replace the 1.7 with 2.x to obtain the latest version of ElasticSearch.

Finally, run the update command again:

# sudo apt-get update

Next, install the ElasticSearch packages using the following command:

# sudo apt-get install elasticsearch

With the installation complete, start the service using the following command:

# sudo service elasticsearch restart

Test the install by querying the localhost at port 9200. You should see the output as shown below:

# curl localhost:9200

Enable ElasticSearch to start on boot using the following command:

# sudo update-rc.d elasticsearch defaults 95 10

We now move to Logstash. Installing Logstash is a simple and straightforward process as well. First we add the

# echo "deb http://packages.elastic.co/logstash/1.5/debian stable main" | sudo tee -a /etc/apt/sources.list

NOTE: You can substitute 1.5 with 2.1 to install the latest Logstash packages.

Make sure you update the packages on the system using the update command:

# sudo apt-get update

Install the Logstash package:

# sudo apt-get install logstash

Enable Logstash to start on boot using the following command:

# sudo update-rc.d logstash defaults 97 8

For this tutorial, we will use Logstash to forward the instance's syslogs to ELK.

The Logstash configuration file defines our Logstash pipeline.

Here's a snippet of the configuration pipeline:

input {

}

filter {

}

output {

}

NOTE: The filter section is optional.

# sudo vi /etc/logstash/conf.d/10-syslog.conf

input {

file {

type => "syslog"

path => [ "/var/log/messages", "/var/log/*.log" ]

}

output {

stdout {

codec => rubydebug

}

elasticsearch {

host => "<PRIVATE_IP>" # Provide the Private IP address of your ELK instance here

}

Restart the Logstash service for the configurations to take effect:

# sudo service logstash restart

We now move on too the final part of the ELK setup and that is the installation and configuration of Kibana. To do so, we first download the Kibana setup using the following command:

# wget https://download.elastic.co/kibana/kibana/kibana-4.1.1-linux-x64.tar.gz

You will also need to download Kibana's service file as well. This will help in starting Kibana's service much easily:

# wget https://raw.githubusercontent.com/akabdog/scripts/master/kibana4_init -O kibana4

Extract Kibana's contents to a folder using the following commands:

# sudo tar -xzf kibana-4.1.1-linux-x64.tar.gz

# sudo mkdir -p /opt/kibana

# sudo mv kibana-4.1.1-linux-x64/* /opt/kibana

Finally, copy Kibana's service file and enable it to start on boot using the following command:

# sudo mv kibana4 /etc/init.d/

# sudo chmod +x /etc/init.d/kibana4

# sudo update-rc.d kibana4 defaults 96 9

# sudo service kibana4 start

Kibana will start on port 5601. To test the install, point your ELK instance's public IP address and port 5601 to a browser. You should see Kibana's initial configuration screen as shown below:

Here, you need to configure the default index pattern for ElasticSearch to run indexes and analysis against. Select the "Index contains time-based events" checkbox as shown. Also, type in "logstash-*" in the Index name or pattern field and set the Time-field name value as "@timestamp"

Click the "Create" button to complete the setup.

You should see logs from your ELK instance popping up in Kibana's "Discover" tab as shown below:

There you have it! a simple ELK on a single EC2 instance!

Coming up next, Configuring ELK clients and how to setup ELK on a production scale as well, so stay tuned for much more!!

Infographic: Manage the Surge In Unstructured Data

2015-10-12T10:39:00.000+05:30

Manage the Surge In Unstructured Data

We produce at least 2 to 3 quintillion bytes (That's 18 zeros!!) of raw, unstructured data per day from Facebook, to online shopping, to Googling etc and this number is growing exponentially each and every day! Storing, analyzing and managing this vast amount of data is a tough challenge even for most IT organizations.
Here's a look at what Unstructured Data is and how can it impact your business / organization? Infographic Source

Taking the correct steps to ensure your infrastructure can handle unstructured data + your typical systems (VoIP, etc.) is a key piece of preparing for the 21st century. XO Communications

Getting started with CouchDB-Part 3

2015-04-13T10:00:00.000+05:30

Getting started with CouchDB-Part 3

Basic Replication and Clustering

In the previous POST, we saw few simple commands using which you can create, list, populate and delete documents and databases. In this post, I'll be walking you through simple steps using which you can replicate and cluster CouchDB Databases.

Replication
The replication is an incremental one way process involving two databases (a source and a destination).

The aim of the replication is that at the end of the process, all active documents on the source database are also in the destination database and all documents that were deleted in the source databases are also deleted (if exists) on the destination database.

NOTE: The replication process only copies the last revision of a document, so all previous revisions that were only on the source database are not copied to the destination database.
Replication in CouchDB is peer-based and involves a source and a destination database, which can be one the same or on different CouchDB instances.

A replication job is triggered by storing a special replication document in the replicator (_replicate) database. Its status can be inspected through the "_active_tasks" API. A replication can be stopped by deleting the document, or by updating it with its cancel property set to true.
Let us look at how to perform replication on the same CouchDB Instance:

First, create two separate databases on the same CouchDB instance. In this case, we have created "db-01" and "db-02"

# curl -X PUT http://127.0.0.1:5984/db-01

# curl -X PUT http://127.0.0.1:5984/db-02

Next, create an empty document on either of the database. Note that the database where the document is created will now become the source from where the contents will be replicated over to the target database.

# curl -X POST http://127.0.0.1:5984/db-01 \
-H "Content-Type: application/json" \
-d '{}'

You can populate the document later once the replication has started.

Next, setup the replication process itself. Here, "db-01" is the source and "db-02" is the target database

# curl -X POST http://127.0.0.1:5984/_replicate \
-H "Content-Type: application/json" \
-d \
'{
"source":"db-01",
"target":"db-02",
}'

To make sure the replication has indeed started, try fetching something from the target database. You should see the "doc_count" as 1 now, meaning there is one document in this database.

# curl -X GET http://127.0.0.1:5984/db-02

You can optionally set the replication to take place continuously as well by setting the "continuous" attribute to "true" as shown:

# curl -X POST http://127.0.0.1:5984/_replicate \
-H "Content-Type: application/json" \
-d \
'{
"source":"db-01",
"target":"db-02",
"continuous":true
}'

Note the "_local_id" attribute value provided.

You can list all current running tasks to check if the replication is working using the "_active_tasks" command as shown:

# curl -X GET http://127.0.0.1:5984/_active_tasks

You should see an output similar to the one shown below:

[
{"pid":"<0.16302.1297>",
"checkpoint_interval":5000,
"checkpointed_source_seq":1,
"continuous":true,
"doc_id":null,
"doc_write_failures":0,
"docs_read":0,
"docs_written":0,
"missing_revisions_found":0,
"progress":100,"
replication_id":"fb577a13ef44561e67e4a849ca81ec4e+continuous",
"revisions_checked":1,
"source":"db-01",
"source_seq":1,
"started_on":1427875738,
"target":"db-02",
"type":"replication",
"updated_on":1427875833}
]

To cancel an active replication task, either delete the document, or update it with its "cancel" property set to "true". Remember to pass the "_local_id" of the replication task as shown:

# curl -X POST http://127.0.0.1:5984/_replicate \
-H "Content-Type: application/json" \
-d \
'{
"_local_id":"96b5d289afa4d7c82622f7d28e7528a4+continuous",
"cancel":true
}'

CouchDB also allows you to replicate your database on more than one CouchDB instance. In this example, I have CouchDB running on two servers; DB-HOST-1 and DB-HOST-2. To begin with, create a database on each of the two servers:

# curl -X PUT http://192.168.0.40:5984/db-01

Follow the same process on the second CouchDB Instance:

# curl -X PUT http://192.168.0.45:5984/db-02

Next, as we did before, create an empty document on either of the database you select as the source:

# curl -X POST http://192.168.0.40:5984/db-01 \
-H "Content-Type: application/json" \
-d '{}'

You can populate the document later once the replication has started.

Next, setup the replication process itself. Here, "db-01" is the source and "db-02" is the target database

# curl -X POST http://127.0.0.40:5984/_replicate \
-H "Content-Type: application/json" \
-d \
'{
"source":"http://192.168.0.40:5984/db-01",
"target":"http://192.168.0.45:5984/db-02",
"continuous":true
}'

You can optionally use CouchDB's UI, Futon as well to setup a replication process.

Access a web browser and type in the following:

http://<COUCHDB_SERVER_IP>:5984/_utils

From the "Overview" page, select the "Replicator" option from the right-hand side options tab as shown. When you click the Replicator you will be presented with the "Replicator screen". This allows you to start replication between two databases by filling in or select the appropriate options within the form provided.

To start a replication process, either the select the "local database" or enter a "remote database" name into the corresponding areas of the form. To enable continuous replication, select the "Continuous" checkbox.

To start the replication process, click the "Replicate" button. The replication process should start and will continue in the background. If the replication process will take a long time, you can monitor the status of the replication using the "Status" option under the Tools menu.

That's all there is for this post.. stay tuned for lots more tutorials coming your way soon!

Cheers!

Getting started with CouchDB-Part 2

2015-04-10T10:37:00.001+05:30

Getting started with CouchDB-Part 2

Working with CouchDB- Basic Commands

In PART 1 of this series of tutorials, we learned some quick facts about CouchDB and also saw an easy way to install and configure it. In this post, I'll be walking you through some basic commands and also show you how to work with Futon; a web interface for managing CouchDB.

Part 1: Installation and Configuration
Part 2: Running basic commands
Part 3: Clustering
Part 4: Load balancing

Commands:
First off, let's check whether our CouchDB instance is up and running or not. For that, simply run a curl command as shown:

# curl -X GET http://192.168.0.40:5984

You should get a response similar to this:
{"couchdb":"Welcome","uuid":"e833c5cfb43d96001433cae620e74595","version":"1.6.1","vendor":{"name":"The Apache Software Foundation","version":"1.6.1"}}

Next, let's create a simple Database.

# curl -X PUT http://192.168.0.40:5984/<DATABASE_NAME>

You should see a response as: {"ok":true}

This means you successfully created a new database on CouchDB.

A few important pointers to remember while creating a database:

Use Lowercase characters (a-z)

You can use Digits (0-9) while naming your database but the first character has to be an alphabet

You can use any of the characters _, $, (, ), +, -, and / while naming your database but the first character has to be an alphabet

If you try to add a database with the same name, then obviously CouchDB will not allow you to do that and prompt you that "the file already exists"

You can obtain additional details about your database by simply querying against it:

# curl -X GET http://192.168.0.40:5984/<DATABASE_NAME>

This will provide you the following output:

{

"db_name":"stuff",

"doc_count":0,

"doc_del_count":0,

"update_seq":0,

"purge_seq":0,

"compact_running":false,

"disk_size":79,

"data_size":0,

"instance_start_time":"1427795965351454",

"disk_format_version":6,

"committed_update_seq":0

}

CouchDB also allows you to list all your databases in one go using some built in commands, such as "_all_dbs"

# curl -X GET http://192.168.0.40:5984/_all_dbs

The output shows you a list of all the databases present on this particular CouchDB Instance. Also worth noting here is the presence of two CouchDB internal databases called as "_users" and "_replicator". We will be talking about these two in the coming tutorials.

["_replicator","_users","oranges","stuff"]
NOTE: Additional to "_all_dbs", CouchDB also provides the following interfaces to query your Couch Instance:

/_active_tasks: Lists actively running tasks

/_db_updates: Lists all database events

/_log: Prints CouchDB's logs

/_replicate: Used during the replication process (more on this in next post)

/_restart: Restarts CouchDB Instance

/_stats: Prints statistics of the CouchDB server

/_utils: Accesses the built-in Futon administration interface for CouchDB

/_uuids: Used to request one or more UUIDs from CouchDB

/_config: Prints entire CouchDB configuration info

You can find the complete list HERE

Now that our database is ready, lets populate it with a "Document". Documents are the primary unit of data in CouchDB and consist of any number of fields and attachments. Documents also include metadata that’s maintained by the database system. Document fields are uniquely named and contain values of varying types (text, number, boolean, lists, etc), and there is no set limit to text size or element count.

To add a document to a database, we use the same curl command but with a "-d" paramenter to pass our data, for e.g:

# curl -X POST http://192.168.0.40:5984/stuff \

-H 'Content-type: application/json' \

-d '{

"fruit":"apple",

"car":"fiat",

"random stuff":"meeoooowwww"

You should see a similar response once you have added your document. The output contains two important fields:

"id" --> Basically a unique ID that identifies your document.
"rev" --> Revision ID. Revisions are important in CouchDB. You cannot update a document in the database without knowing the current revision of the document. This is a failsafe to ensure that you do not just overwrite the document with a new version.

{"ok":true,"id":"0ca55157fc3fdd5528f7d98934002b2b","rev":"1-c31f8f2f547922f63fd3195547db7180"}

To list the document's contents, you simply use the GET command along with the "Document's ID" as shown:

# curl -X GET http://192.168.0.40:5984/<DATABASE_NAME>/<DOCUMENT_ID>

If you wish to update a particular entry from your document, you can do that as well by providing the document's "Revision ID" along with the field that you wish to update as shown:

# curl -X POST http://192.168.0.40:5984/stuff \
-H 'Content-type: application/json' \
-d '{
"car":"BMW",
"-rev":"1-c31f8f2f547922f63fd3195547db7180"
}'

Deleting a Database is simple as well, just pass the database name with the DELETE parameter as show:

# curl -X DELETE http://192.168.0.40:5984/stuff

The best part of CouchDB is that it comes with its own web-based UI as well called Futon. It provides a basic interface to the majority of the functionality, including the ability to create, update, delete and view documents and views, provides access to the configuration parameters, and an interface for initiating replication.

To access Futon, simply open up a browser and type in the following:

http://<COUCHDB_SERVER_IP>:5984/_utils

You will be presented with a similar "Home" screen or the "Overview" tab that basically shows you a list of databases that you have created. To the right of the UI, there is a "Tools" section, where you can select different options such as "Configuration", "Replication" and even view the current "Status" of your CouchDB instance.

To create a new database, click the "Create Database" button. You will be prompted for the database name, as shown

Provide a suitable name for your database and click "Create" once done.

This will automatically take you to your newly created database. Here, you can create documents, manage the database's security and even delete the database if required.

For now, let us create a sample document for our database. Click "New Document"

The document gets an "_id" field by default. You can change its value to something a bit more meaningful if you wish. Click on "Add Field" to add a new row in your document. Once done, remember to save the changes by selecting the "Save Document" button.

Fulton will automatically change the "Revision ID/ _rev" in case you edit a field of the document. you can traverse between the different revisions of the document by using the tabs as shown below:

Optionally you can also view the "Source" of your document by selecting the "Source" tab as shown. This will provide you the contents of your document in a JASON format.

In the next PART of this series, we will be looking at some simple ways to create a Cluster between two CouchDB instances so stay tuned for much more coming your way soon!

Cheers!

Getting started with CouchDB-Part 1

2015-04-09T09:25:00.002+05:30

Getting started with CouchDB-Part 1

Installation and Configuration

In this series of simple tutorials, I'm going to show you how you can get started with a popular and upcoming NoSQL database called CouchDB.

Part 1: Installation and Configuration
Part 2: Running basic commands
Part 3: Clustering

Introducing CouchDB
A few interesting facts that you should know about CouchDB before we begin:

CouchDB is a JSON document-oriented database written in Erlang.
It is a highly concurrent database designed to be easily replicable, scale horizontally across numerous devices and be fault tolerant.
It is an open source Apache foundation project.
It allows applications to store JSON documents via its RESTful interface.
It makes use of map/reduce to index and query the database.
Follows ACID Semantics.
Eventual Consistent.
Provides Mobile access.

Setup for this guide:
For this set of tutorials, I'll be running CouchDB on a set of CentOS 6.5 64 Bit Virtual Machines. You can run CouchDB off physical commodity-based hardware as well.

OS: CentOS 6.5 64 bit
HOSTNAME: DB-HOST-1
IPADDRESS: 192.168.0.40

Installation
CouchDB can be installed both via packages as well as from source. Installation from packages is quite simple in case you are using a Debian/ ubuntu system. All you need to do is run the following command on the terminal:

# sudo apt-get install couchdb

Notice that the packages in the repositories are not always the latest version. So in this case we will be installing CouchDB from source.

First off, install some pre-requisite software. This will be required to build and install CouchDB from its source.

# yum install libicu-devel curl-devel ncurses-devel libtool libxslt fop java-1.6.0-openjdk java-1.6.0-openjdk-devel unixODBC unixODBC-devel openssl-devel

Next, we install Erlang. CentOS unfortunately does not provide any official packages for Erlang, so you will have to build it from source.

First, go to www.erlang.org/download.html and download the latest source code.

# wget http://www.erlang.org/download/otp_src_17.4.tar.gz

Unpack the archive

# tar -zxvf otp_src_17.4.tar.gz

CD into the new directory and run the following command:

# cd otp_src_17.4

# ./configure

Make sure you dont get any gcc compiler-based errors here. Next, run the make command:

# make

And finally the make install

# make install

If all goes well, your CentOS system should have Erlang successfully installed on it by now. To test whether it did go well, just type in "erl" to enter the Erlang shell prompt. You should see a similar output as shown.

# erl

Next, we need to install something called as SpiderMoney JavaScript Engine.
Mozilla's SpiderMoney JavaScript Engine is required by CouchDB in order to compile successfully.

CouchDB requires Mozilla's SpiderMoney version 1.8.5, which you can download from their FTP:

# wget http://ftp.mozilla.org/pub/mozilla.org/js/js185-1.0.0.tar.gz

unpack the archive:

# tar -zxvf js185-1.0.0.tar.gz

CD into the following directory and run the following command:

# cd js-1.8.5/js/src/

# ./configure

Next, run the make command

# make

And finally the make install:

# make install

With our pre-requisites met, we can now finally go ahead and install CouchDB on our host. First off, downlaod the latest source of CouchDB:

# wget http://apache.arvixe.com/couchdb/source/1.6.1/apache-couchdb-1.6.1.tar.gz

Unpack the archive using the following command:

# tar -zxvf apache-couchdb-1.6.1.tar.gz

Follow the same 3-step build and install process as we performed for Erlang and SpiderMonkey. CD into the new directory and run the configure command:

# cd apache-couchdb-1.6.1

# ./configure

Next, run the make command

# make

And finally the make install:

# make install

Create a user for CouchDB to use:

# adduser --no-create-home couchdb

Change the owner on the following files to the newly created user:

# chown -R couchdb:couchdb \

/usr/local/var/lib/couchdb \

/usr/local/var/log/couchdb \

/usr/local/var/run/couchdb

Create a symlink for the couchdb init script to /etc/init.d:

# ln -sf /usr/local/etc/rc.d/couchdb /etc/init.d/couchdb

Add couchdb's init script in chkconfig and enable it so that it starts CouchDB on boot:

# chkconfig --add couchdb

# chkconfig couchdb on

By default, CouchDB can be accessed only from the server it is installed on itself. If you'd like to access it from the web, you'll have to edit the following configuration file:

# vi /usr/local/etc/couchdb/local.ini

Should you need to access couchdb from the web, in the [httpd] section, look for a setting called bind_address and change it to 0.0.0.0 - this will make CouchDB bind all available addresses. You can optionally change the port number as well if you wish to do so.

[httpd]

port = 5984

bind_address = 0.0.0.0

Once done, start the couchdb service

# service couchdb start

A simple test to check whether CouchDB is indeed running or not, you can simply run couchdb on the command line. it will show you the following output: "Apache CouchDB is running, time to relax".

To additionally verify its working, simply query the CouchDB port using curl:

# curl http://localhost:5984

You should see a similar output:
{

"couchdb":"Welcome",

"uuid":"763ba4330a0f38c42bdb79b037ef4cc4",

"version":"1.6.1",

"vendor":{"name":"The Apache Software Foundation","version":"1.6.1"}

}

You can pass your Server's IP address as well to test the connectivity:

# curl http://192.168.0.40:5984

With this we come to an end of the installation and configuration process. In PART 2, we will be looking at how to run few simple commands in CouchDB, so stay tuned for a lot more coming your way soon!!

cheers!

Install and configure VirtualBox on CentOS/ RedHat

2015-03-02T14:33:00.001+05:30

Install and configure VirtualBox on CentOS/ RedHat

Here's a quick, easy to follow guide for installing and configuring VirtualBox to run on a CentOS/ RedHat machine.

This tutorial was tested on a CentOS 6.5 64 Bit machine.

To get started, first off install the correct EPEL repo. You can find the exact EPEL repo link suited for your machines here:
--http://dl.fedoraproject.org/pub/epel
--http://mirror.steadfast.net/epel

Since Im using a CentOS 6.5 machine, my EPEL repo command is as follows:

# rpm -Uvh http://mirror.steadfast.net/epel/6/x86_64/epel-release-6-8.noarch.rpm

Once the EPEL repo is installed, create a new .repo file for downloading the VirtualBox RPM pacakge:

# vi /etc/yum.repo.d/virtualbox.repo

[virtualbox]

name=Oracle Linux / RHEL / CentOS-$releasever / $basearch - VirtualBox

baseurl=http://download.virtualbox.org/virtualbox/rpm/el/$releasever/$basearch

enabled=1

gpgcheck=1 gpgkey=http://download.virtualbox.org/virtualbox/debian/oracle_vbox.asc

Save the file and exit the editor.

Run a repolist to verify and list the newly added repos.

# yum repolist

Install few additional packages before we install VirtualBox.

# yum install gcc make patch dkms qt libgomp

Install the development libraries and kernel headers as well

# yum install kernel-headers hernel-devel fontforge binutils glibc-headers glibc-devel

Configure the correct value for the KERN_DIR environment variable. In my case, the Kernel Source files were present at /usr/src directory.

# export KERN_DIR=/usr/src/kernels/2.6.32-504.8.1.el6.x86_64/

Once all done, simply install VirtualBox using the following command:

# yum install VirtualBox-4.3

After installation finishes, rebuild the kernel modules using following command:

# service vboxdrv setup

Once done, you are now ready to start using VirtualBox on your CentOS/ RedHat machine. For the CentOS desktop, select Applications -> System Tools -> Oracle VM VirtualBox

Alternatively, you can also run VirtualBox from the terminal using the following command:

# virtualbox &

Automating IT infrastructure with Ansible- PART 4

2015-02-24T09:43:00.000+05:30

Automating IT infrastructure with Ansible- PART 4

- MySQL Playbook

In our previous Ansible tutorials, we have looked at how to create simple and re-usable playbooks from scratch. In this tutorial, I'll be demonstrating how to install and configure MySQL Database on either a RedHat or a Debian based system by using just a single playbook.

NOTE: This is a continuation post so if you haven't gone through the previous tutorials, here's your chance to catch up on them:

>> Automating IT infrastructure with Ansible- PART 1 - Installing and configuring Ansible

>> Automating IT infrastructure with Ansible- PART 2 - Writing simple playbooks

>> Automating IT infrastructure with Ansible- PART 3 - Working with conditions

>> Automating IT infrastructure with Ansible- PART 4 - MySQL Playbook (YOU ARE HERE)

Create the Playbook

First of, lets create the required folders for our playbook.

The folder structure of this playbook looks something like this:

bootstrap/
|___ hosts ~~> Contains the inventory info
|___ main.yml ~~> master main.yml file
|___ ntp/ ~~> ntp role <Added in PART 3>

|___ mysql/ ~~> mysql role

|__ defaults

|__ files/

|__ handlers/

|__ meta/

|__ README.md

|__ tasks/
|__ templates/
|__ vars/

In my earlier tutorial we accomplished this manually by creating each individual folder such as tasks, templates, vars etc. In this case, let's use Ansible's command to do the work for us:

# ansible-galaxy init <playbook_name>

This command initializes and creates all the required sub-folders and their respective main.yml files as shown.

Next, create the vars files that will hold our MySQL-related variables. AS done in PART 3, I have created two variable files namely RedHat.yml and Debian.yml with the following contents:

# RedHat distribution based Variables

mysql_service: mysqld
mysql_conf_path: /etc/my.cnf
mysql_datadir: /var/lib/mysql
mysql_socket: /var/lib/mysql/mysql.sock
mysql_log: /var/log/mysqld.log
mysql_pid: /var/run/mysqld/mysqld.pid

Similarly, for the Debian-based variables:

# Debian based distribution Variables

mysql_service: mysql
mysql_conf_path: /etc/mysql/my.cnf
mysql_datadir: /var/lib/mysql
mysql_socket: /var/lib/mysqld/mysqld.sock
mysql_log: /var/log/mysql/error.log
mysql_pid: /var/run/mysqld/mysqld.pid

Once both the variable files are created, populate the defaults/main.yml with few default values as well.

---
# defaults file for mysql

mysql_port: 3306
dbuser: cloud
dbname: testdb

dbpassword: test

In this case, we are setting the MySQL port value along with a sample database name, its username and password.

Next off, create the templates/my.cnf.j2 file. The reason we are creating this as a template is because the values in my.cnf file varry from RedHat to Debian distributions.

[mysqld]
datadir={{ mysql_datadir }}
socket={{ mysql_socket }}

user=mysql
symbolic-links=0
port={{ mysql_port }}

[mysqld_safe]
log-error={{ mysql_log }}
pid-file={{ mysql_pid }}

Moving on, create the main tasks file under tasks/main.yml as shown:

---
# tasks file for mysql

- name: Include OS-specific variables
include_vars: "{{ ansible_os_family }}.yml"

- name: Install MySQL (RedHat)
yum: name={{ item }} state=installed
with_items:
- mysql-server
- MySQL-python
when: ansible_os_family == 'RedHat'

- name: Install MySQL (Debian)
apt: name={{ item }} state=installed
with_items:
- mysql-server
- python-mysqldb
when: ansible_os_family == 'Debian'

- name: Create Mysql configuration file
template: src=my.cnf.j2 dest={{ mysql_conf_path }}

- name: Start Mysql Service
service: name={{ mysql_service }} state=started enabled=yes

- name: Create dummy Database
mysql_db: name={{ dbname }} state=present

- name: Create dummy Database User
mysql_user: name="{{ dbuser }}" password="{{ dbpassword }}" priv=*.*:ALL host='%' state=present

Lets have a close look at what we have configured:
1) include_vars: "{{ ansible_os_family }}.yml"
The include_vars will search for a .yml file in the vars directory. In this case, the include_vars will call two files: RedHat.yml and Debian.yml using the {{ ansible_os_family }} variable/ fact.

2/ 3) when: ansible_os_family == 'RedHat'
"when" statement is a condition control statement in Ansible and is used to execute a particular task only when a particular condition is met. If the condition is not met, this task is simply avoided.

4) {{ mysql_conf_path }}
This value will be picked up from the two variable files and depending on the base OS, the actual value will be populated.
RedHat-based OS: /etc/my.cnf
Debian-based OS: /etc/mysql/my.cnf

5) {{ mysql_service }}
Similar to the previous point, this will start up the My SQL service and enable it to start on boot as well.

6) mysql_db
This is one of Ansible's many core modules. Using this module you can basically create, add and delete databases on remote hosts. You can read more about it HERE

7) mysql_user
Yet another Ansible core module, this helps you to create/ remove as well as assign privileges to your database users. You can read more about this module HERE

Edit the bootstrap main.yml (master yml file) and make sure the mysql role is added to it as shown:

Run the Playbook
Once all the required files and settings are completed, simply execute the playbook as shown:

# ansible-playbook -i hosts main.yml

You should see the playbook install MySQL server on your remote hosts, copy the my.cnf file over to the correct location, start the MySQL service and populate the database as well.

Here's the directory structure of the final playbook.

Lots more still to come on Ansible so stay tuned for a lot more!!

Automating IT infrastructure with Ansible- PART 3

2015-02-18T09:19:00.000+05:30

Automating IT infrastructure with Ansible- PART 3

- Working with conditions

In Part 1 and Part 2 of this tutorial series we saw how to install and configure Ansible in a few easy steps along with how to get started with writing simple playbooks.

This tutorial will look into some of Ansible's conditional statements and also provide you with a quick insight on how to develop playbooks that support multiple operating systems.

To get started, let us first understand the scenario. I have 2 remote Hosts in my environment: Host 1 (CentOS :: 192.168.0.15) and Host 3 (Ubuntu :: 192.168.0.25). Both these systems are out of sync and I want to sync them up to a NTP server. The solution is simple... write a playbook that will:
1) Set the correct timezone (based on what the end users specify)
2) Check whether the NTP packages are installed on both the systems or not. If not then it will install the NTP packages based on that specific OS type.
3) Add the NTP servers to the /etc/ntp.conf file based on the specific OS type
4) Start the NTP service
5) Sync the NTP date and time to a common NTP server

So without further adieu, let's get started!! First up, create a folder for hosting our playbooks. Here, I'm using the following folder structure for this tutorial:

/etc/ansible/playbooks/bootstrap/ntp

NOTE: Here, bootstrap is my master playbook, and the ntp acts as a Role. You can alternatively create your NTP directory as a separate playbook as well.

Create the Playbook
The folder structure of this playbook looks something like this:

bootstrap/
|___ hosts ~~> Contains the inventory info
|___ main.yml ~~> master main.yml file
|___ ntp/ ~~> ntp role
|__ defaults/
|__ handlers/
|__ tasks/
|__ templates/
|__ vars/

First off, lets create some OS specific variables for our play. In this case, Im going to create two files, namely RedHat.yml and Debian.yml and populate them with the NTP related info as shown. But why name these files RedHat and Debian??

Ansible provides and exploits some really cool variables/ facts such as :

{{ ansible_distribution }}-{{ ansible_distribution_version }}
{{ ansible_distribution }}
{{ ansible_os_family }}

In non-ansible terms, these three actually mean:

CentOS-6.5

CentOS

RedHat

[or]

Ubuntu-12.04

Ubuntu

Debian
We will be using {{ ansible_os_family }} to call/ import the different variable files in our main.yml. But first, create and assign the variables as shown below:

# For RedHat:
ntp_package:
name: ntp
state: installed
ntp_service: ntpd
ntp_config: /etc/ntp.conf
ntp_servers:
- 0.centos.pool.ntp.org iburst
- 1.centos.pool.ntp.org iburst
- 2.centos.pool.ntp.org iburst
- 3.centos.pool.ntp.org iburst

# For Debian:
ntp_package:
name: ntp
state: installed
ntp_service: ntp
ntp_config: /etc/ntp.conf
ntp_servers:
- 0.ubuntu.pool.ntp.org
- 1.ubuntu.pool.ntp.org
- 2.ubuntu.pool.ntp.org
- 3.ubuntu.pool.ntp.org

Save and close the files when done.

Next, we move on to the main tasks file for our playbook. Create the ntp/tasks/main.yml file as shown:

---
# tasks file for ntp

- name: Include OS-specific variables
include_vars: "{{ ansible_os_family }}.yml"

- name: Set the correct timezone
file: src=/usr/share/zoneinfo/{{ ntp_timezone }} dest=/etc/localtime state=link force=yes

- name: Install NTP (RedHat)
yum: name={{ ntp_package.name }} state={{ ntp_package.state }}
when: ansible_os_family == 'RedHat'

notify:

- Sync NTP

- name: Install NTP (Debian)
apt: name={{ ntp_package.name }} state={{ ntp_package.state }}
when: ansible_os_family == 'Debian'

notify:

- Sync NTP

- name: Configure NTP
template: src={{ item }} dest={{ ntp_config }}
with_first_found:
- "../templates/{{ ansible_distribution }}-{{ ansible_distribution_version }}.ntp.conf.j2"
- "../templates/{{ ansible_distribution }}.ntp.conf.j2"
- "../templates/{{ ansible_os_family }}.ntp.conf.j2"
- "../templates/ntp.conf.j2"

notify:

- Start NTP Service

Lets have a close look at what we have configured:
1) include_vars: "{{ ansible_os_family }}.yml"
The include_vars will search for a .yml file in the vars directory. In this case, the include_vars will call two files: RedHat.yml and Debian.yml using the {{ ansible_os_family }} variable/ fact.

2) {{ ntp_timezone }}
Here, we are using the {{ ntp_timezone }} variable as a "role default variable". These variables are created in the /defaults/main.yml file and can be easily overridden by a variable declared in the /vars directory.

3) {{ ntp_package.name }}
We earlier declared a variable called {{ ntp_package }} in our two variable files. This is just a easier way of calling the sub-attributes of the same variable.

4) when: ansible_os_family == 'RedHat'
"when" statement is a condition control statement in Ansible and is used to execute a particular task only when a particular condition is met. If the condition is not met, this task is simply avoided.

5) {{ item }}.... with_first_found
{{ item }} statements can be used as loops to iterate through various items in a list. In this case, we are iterating through a files list and using with_first_found parameter. This parameter is useful when you want to iterate through a list of files and use the first available file from the list.

Moving on, create the handlers as well in the /handlers/main.yml file. Make sure the names of both the handlers match with the [notify] attribute in the /tasks/main.yml file.

Create a j2 template file that will populate the NTP servers list based on the different OS used.

In this template file we have made use of the [for] loop.

{% for ntp_server in ntp_servers %}
server {{ ntp_server }}

{% endfor %}

Each [for] loop ends with a [endfor] statement. In this case, the particular [for] statement will loop and print the NTP servers list from the two variable files that we created in the earlier steps

Dont forget to add the custom variables in the defaults/main.yml file.

Edit the bootstrap main.yml (master yml file) and make sure the ntp role is added to it as shown

Run the playbook
Once the playbook is created, we can now run it against all the hosts in our inventory. First off, let's check the actual date and time on our remote hosts using Ansible itself.

# ansible all -m shell -a "date"

As you can see, both the servers are out of sync, in both time, date and timezone.

Run the playbook.

# ansible-playbook -i hosts main.yml

Run the date command once the playbook completes its execution. The servers are now synced to a particular NTP server and are showing the correct time, date and timezone details as required.

Here's the directory structure of the final playbook.

So, let's quickly recap what all we got to learn from this tutorial:
-- We got to see how to create playbooks that have multi-OS support
-- We learnt about using ansible facts and variables in the form of {{ ansible_os_family }}
-- We looked at the [when] statement and its uses
-- We saw two loop statements as well in the form of {{ item }} and {{% for %}}

Still more to come on Ansible so stay tuned for further posts!!

Cheers!

Automating IT infrastructure with Ansible- PART 2

2015-02-13T13:54:00.000+05:30

Automating IT infrastructure with Ansible- PART 2

Writing simple Playbooks

In PART 1 of this series of tutorials on Ansible, we saw how simple and easy it is to get Ansible up and running on a host and run few simple Ansible commands here and there.

In PART 2, we are going to look at how to write simple playbooks, specifically we will learn about: Tasks, Handlers, Roles, Templates and Variables

For this scenario, the playbook that we are going to create will simply install and configure Apache web server (httpd) on the remote host. The main purpose of this tutorial however is to learn how to write playbooks effectively and correctly. So without further adieu, lets get started!

Creating the Playbook
I'm going to assume you have already installed and configured the Ansible server and a few hosts in your environment. If not, refer the PART 1 of this tutorial.

For this tutorial, we will first create a separate directory under Ansible's default directory (/etc/ansible) and perform all our tasks from here. Start off by creating a simple Master Playbook as shown below:

# cd /etc/ansible

# mkdir -p playbooks/http-playbook

# vi main.yml

Tasks
Let's create a simple play which will check and install http packages on the remote host. Here, this play will run the [task] named "Install Apache httpd" on [all] hosts listed in the hosts inventory file.

---
## PLAYBOOK TO INSTALL AND CONFIGURE APACHE HTTP ON CENTOS

- host: all
tasks:
- name: Install Apache httpd
yum: pkg=httpd state=installed

Save the file and exit the editor.

Run the playbook. you should see the following sample output as shown below:

# ansible-playbook main.yml

Handlers
Handlers are similar to tasks that we saw in the earlier step. The only difference is that a Handler will be executed only when it is called by an event. For e.g in the same main.yml file, lets add a handler that will start the httpd service. But, if you see closely, the handler will only be called AFTER the initial install [tasks] completes. The handler will be called by the [notify] directive.

NOTE that the name of the notify and the handler must be the same.

---
## PLAYBOOK TO INSTALL AND CONFIGURE APACHE HTTP ON CENTOS

- host: all
tasks:
- name: Install Apache httpd
yum: pkg=httpd state=installed

notify:
- Start Httpd

handlers:
- name: Start httpd
service: name=httpd state=started

Run the playbook again. This time, you should see a NOTIFIED message as shown below.

Roles
Roles are just a good way of organizing and redistributing multiple, related tasks. For example, you can have a web role which contains tasks to install and configure a http server, you can have a DB role for containing all your database tasks and handlers etc.

Let's go ahead and create a role for our small playbook. Let's call the role as web. Within this role, we will create separate sub-directories each with its own purpose.

# mkdir -p roles/web

# cd /roles/web

# mkdir -p {handlers,tasks,templates,vars}

-Handlers: Will store all handlers for our web role
-Tasks: Will store the tasks that are required to be performed on the web role
-Templates: Will contain a .j2 file (Python's Jinja2 template engine)
-Vars: Will contain some custom variables that will be used ONLY for the web role

here's what the directory structure now looks like:

IMP NOTE: Within each directory, Ansible will search for and read any Yaml file called main.yml automatically.

Let's first edit our main.yml and add the web role in it. Notice that we have wiped out all the rest of the content as now that content will be placed in separate files and folders.

---
## PLAYBOOK TO INSTALL AND CONFIGURE APACHE HTTP ON CENTOS

- host: all
roles:
- web

Save the file and exit the editor.

let us go ahead and populate the tasks and handlers now. First up, the tasks:

# vi roles/web/tasks/main.yml

---
- include: install-httpd.yml

The [include] directive is used to call other Yaml files. In this case, we will use our main.yml to call another yaml file that will actually install the http packages.

Create the install-httpd.yml file and populate it with the following contents:

# vi roles/web/tasks/install-httpd.yml

---
# these tasks install http and the php modules.

- name: Install httpd packages
yum: name={{ item }} state=installed
with_items:
- httpd
- php

notify:
- Start Httpd

Notice here, we did not use the [hosts] or the [tasks] directive. We directly go ahead and create the play. Towards the end, we have also mentioned the [notify] directive. As you must have guessed, this will call the corresponding handler which will start the httpd service.

Create the corresponding handler in the handlers directory.

# vi roles/web/handlers/main.yml

---
# handler to start httpd service

- name: Start Httpd
service: name:httpd state=started

By now, your playbook should look something like this:

Run the playbook. You should see now that both the http and php packages were installed and the httpd service was started by the handler as well.

Templates
Templates are files that are based on Python's Jinja2 template engine. Each template file ends with a .j2 extension. You can put any content in these files for example a index.html files content etc, but the real power of these files comes when you use variables in it. You can use Ansible's [facts] and even call custom variables in these template files.

In the below example, I have used two facts/ variables:
{{ ansible_eth0.ipv4.address }} ~~> IP Address of the remote host
{{ ansible_nodename }} ~~> Hostname of the remote host

NOTE: Ansible by default gathers “facts” about the machines under management, and these facts can be accessed in Playbooks and in templates. To see a list of all of the facts that are available about a machine, you can run the “setup” module as an ad-hoc action:
# ansible -m setup hostname

# vi roles/web/templates/index.html.j2

Save and exit the editor once you have created your template file.

Next, we will need a mechanism by which we can transfer these template files from their source directory (/roles/web/templates) to a destination directory on the remote host.

For that, let's create another yaml file in the tasks directory as follows:

# vi roles/web/tasks/copy-template.yml

---

- name: Create the index.html file over at the httpd server
template: src=index.html.j2 dest=/var/www/html/index.html

save and exit the editor. Here, we have used a [template] module that will copy the .j2 file from the source templates directory over to the destination server.

Dont forget to add our copy-template.yml file in the main.yml tasks file.

# vi roles/web/tasks/main.yml

- include: copy-template.yml

Run the playbook. You should see the task of creating the index.html file over at the httpd server as shown below:

Want to see what got populated in the index.html file? Simply open a browser up and type in the IP address of the Httpd Server, you should see the hostname ({{ ansible_nodename }}) and IP address ({{ ansible_eth0.ipv4.address }}) printed as shown below:

Variables
As the name implies, you can use both Ansible facts and custom made variables in your playbooks. However, the important thing to note here is how to declare the variables?? Variables can be defined in 5 different ways:

1) Variables defined in the play under vars_files attribute:

vars_files:

- "/path/to/var/file"

2) Variables defined in <role>/vars/main.yml

3) Variables passed in on the command line

# ansible-playbook main.yml -e "http-port=80"

4) Variables defined in the play under vars

vars: http_port: 80

5) Variables defined in group_vars/ directory
Let's have a look at this with an example. In this case, I'm using a httpd.conf file as a template stored at roles/web/templates directory. In this file, I'll call a custom variable {{ http_port }}. This variable will be defined in the /group_vars directory as explained:

NOTE: Variable names should never include dashes, they will be interpreted as a subtraction. Use underscores instead!

# vi roles/web/templates/httpd.conf.j2

In the conf file, I have replaced the Listening port value with the custom variable {{ http_port }}

I have also used the custom variable and Ansible Facts in the ServerName and ServerAdmin fields as shown

To declare the variable, create a group_vars directory outside the roles folder.

# mkdir group_vars

# vi group_vars/all.yml

in the all.yml file, declare the custom variable as shown. This Variable will now be available to ALL the roles that are included in this playbook. If you wish to keep this variable restricted only to the web role, then create the all.yml file in the roles/web/vars directory.

---
# Global Custom Variables
http_port: 80

Don't forget to call the [template] module for the httpd.conf file as well.

# vi roles/web/tasks/copy-template.yml

- name: Create the httpd.conf file
template: src=httpd.conf.j2 dest=/etc/httpd/conf/httpd.conf

Run the playbook. You should see the httpd.conf task as shown below:

by the end of this tutorial, your directory structure should look something like this.

Using this similar style, you can create roles such as Database, Loadbalancer etc and create plays and templates for each of them. Ansible truly has a lot of potential and is really easy and quick to learn. Hope this tutorial helps you out the way it helped me! Stay tuned for a lot more coming your way soon..

Cheers!!

Automating IT infrastructure with Ansible- PART 1

2015-02-10T16:48:00.000+05:30

Automating IT infrastructure with Ansible- PART 1

Ansible is a really powerful and useful tool when it comes to automating IT infrastructure. It is an easy to use configuration management system that can be used to automate and organize your system configuration tasks for a large network of computers.

How Does Ansible Work?
Ansible works by configuring client machines from an computer with Ansible components installed and configured.

It communicates over normal SSH channels in order to retrieve information from remote machines, issue commands, and copy files. Because of this, an Ansible system does not require any additional software to be installed on the client computers.

This is one way that Ansible simplifies the administration of servers. Any server that has an SSH port exposed can be brought under Ansible's configuration umbrella, regardless of what stage it is at in its life cycle.

Any computer that you can administer through SSH, you can also administer through Ansible.

Ansible takes on a modular approach, making it easy to extend to use the functionalities of the main system to deal with specific scenarios. Modules can be written in any language and communicate in standard JSON.

Configuration files are mainly written in the YAML data serialization format due to its expressive nature and its similarity to popular markup languages. Ansible can interact with clients through either command line tools or through its configuration scripts called Playbooks.

Setup for this guide:

For this series of tutorials, I'll be using mostly this particular setup consisting of three VMs. Each VM has CentOS 6.5 64 Bit installed on it.

Ansible Host: 192.168.0.10
Host1: 192.168.0.15
Host2: 192.168.0.20

NOTE: For a majority of these tutorials, I have disabled the firewall and Selinux on all the machines.

Installation
Installing Ansible is a very easy process. Unlike most configuration management tools, Ansible is only required to be setup on on master server and there are many different ways to do so. Many Linux distributions have a package available in their 3rd party repos which can easily be installed, however a quick alternative is to just pip install it or grab the latest copy from github.

To install using your package manager, on RHEL/CentOS Linux based systems you will most likely need the EPEL repo

Type in the following command to install the EPEL repo on your host:

# yum install http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

Next, install the Ansible packages:

# yum install ansible

Ansible has a default inventory file used to define which servers it will be managing. After installation, there's an example one you can reference at /etc/ansible/hosts.

My hosts file looks something like this. Here, I have declared a group called as [My-Hosts] and within this group we have the two hosts added. You can either use IP addresses or even FQDNs to specify the hosts.

Once we have an inventory configured, we can start running Tasks against the defined servers.

Ansible will assume you have SSH access available to your servers, usually based on SSH-Key.

Run the following command to test your Ansible config. Note for the time being, we will use the --ask-pass parameter to prompt for entering the password of the remote hosts.

# ansible -m ping My-Hosts --ask-pass

Here, we are attempting a simple ping command on our inventory group called My-Hosts. If all things are configured correctly, you should be prompted to enter the Hosts password and shown the output as pong. In some cases, you may be prompted to install the sshpass program. you can do that by simply running a yum install sshpass on your ansible server:

The correct output will look something like this:

You can also ping individual hosts by specifying the host's IP address or FQDN in place for the inventory group.

Generating SSH Keys
With the basic configurations done, lets start with creating and configuring some SSH keys so we can avoid the --ask-pass options:
In the Ansible server, run the following commands to create a SSH keypair.

# ssh-keygen -t rsa

Now copy over the keys over to the hosts. There are a variety of ways to do this, but let's have a look how we can perform these steps using Ansible itself:

# ansible all -m copy -a "src=/root/.ssh/id_rsa.pub dest=/tmp/id_rsa.pub" --ask-pass

Here, all indicates to use all the hosts specified in the hosts inventory file. Copy is an ansible module. We will use many such modules to perform our tasks using Ansible.

You should get an output similar to the one shown below:

Next, add the public keys over to the remote servers:

# ansible all -m shell -a "cat /tmp/id_rsa.pub >> /root/.ssh/authorized_keys" --ask-pass

Let's try pining our two hosts again now, but this time without the --ask-pass parameter. You should see the following pong output:

# ansible My-Hosts -m ping

Try running some other command with ansible. Here, we want to list the hostname of our hosts in the inventory.

# ansible My-Hosts -m shell -a "hostname"

Here, shell is another ansible module that you can use to pass shell commands to your remote hosts. The -a attribute allows you to pass commands to the shell module. For E.G. here, we pass the hostname command. You can alternatively pass commands such as yum install httpd as well.

With this, you should be ready to write and create your own playbooks. This is something we will look at in the next Part of this tutorial series.. till then stay tuned!!

Quick Start: Setup CentOS 7 as a SAMBA Server

2015-01-14T13:15:00.000+05:30

Quick Start: Setup CentOS 7 as a SAMBA Server

In this tutorial, I'm going to show you simple steps using which you can setup your own SAMBA Server on a CentOS 7 system.

SAMBA is an open source software that provides file and print services to various SMB/CIFS clients. It allows users to share files, folders, and printers between Linux server and Windows clients.

Prerequisites
CentOS 7 Minimal ISO
Make sure your system has a static IP address assigned to it and a fully qualified domain name.

Samba System Details:
Base OS: CentOS 7 64 Bit (Minimal ISO)
HOSTNAME: master.cloud.com
IPADDRESS: 192.168.0.15

Client System Details:
Base OS: Ubuntu 14.04 64 Bit
HOSTNAME: client1.cloud.com
IPADDRESS: 192.168.0.101

Steps:
Install the samba packages

# yum install samba*

Next, create a directory which will be shared across the network. In this case, I'm creating a directory /network/share. Here, the directory "share" will appear as a shared folder on the network

# mkdir -p /network/share

# chmod -R 0777 /network/share/

Edit the samba configuration file and make the following changes to it:

# vi /etc/samba/smb.conf

# Add the following lines under [global] section

unix charset = UTF-8 d

os charset = CP932

# Change the to windows default "WORKGROUP"

workgroup = WORKGROUP

# Un-comment and set the IP Range for localhost and your network

hosts allow = 127. 192.168.0.
# Un-comment max protocol

max protocol = SMB2

# Un-comment, and change the value of security field to "user"

security = user

# Add the following line

map to guest = Bad User

# At the end of the file, add the details for your network shared folder. In my case, the name of the folder is [share] and it is located at /network/share directory

[share]

path = /network/share

writable = yes

browsable = yes

guest ok = yes

guest only = yes

create mode = 0777

directory mode = 0777

Save the file and quit the editor once the changes are done.

Start and enable the samba services

# systemctl start smb

# systemctl start nmb

# systemctl enable smb

# systemctl enable nmb

Test whether your samba share has been successfully configured or not by using the "testparm" command as shown

# testparm

Pres the enter key to see a detailed description of the shared services

You should see something similar to the screenshot below. A summary of your [global], [homes], [printers] and shared folder [share] displayed.

Add the following rules and reload the firewall:

# firewall-cmd --permanent --add-port=137/tcp

# firewall-cmd --permanent --add-port=138/tcp

# firewall-cmd --permanent --add-port=139/tcp

# firewall-cmd --permanent --add-port=445/tcp

# firewall-cmd --permanent --add-port=901/tcp

# firewall-cmd --reload

Testing the setup
Here, I'll be testing the samba shared folder on a Ubuntu Client. This Ubuntu machine is on the same network as the Samba Server.

NOTE: You can test the samba connectivity using a Linux or a Windows machine as well. The steps remain the same more or less ;)

Select the "Connect to Server" option. In the dialog box, provide the samba server's IP address as shown and click "Connect":

smb://<SAMBA_SRVR_IP_ADDRESS>

You should see the "share" directory as shown below. To further test this directory, go ahead and create a temporary folder in it.

Here, I create a temp directory called "TestFolder" within the "share" directory

If you check in your samba server, you should see the "TestFolder" is automatically created under the /network/share directory as shown below.

Securing Samba shared folders
The example above create a less secure shared folder that is accessible by anyone on the network. If you wish to make your shared folder a bit more secure, you can provide user credentials for accessing it as well.

Follow these simple steps to create a secure shared folder:

# Add a user for accessing samba shared folder
# useradd -s /sbin/nologin testuser

# Create a group for samba users
# groupadd smbgroup

# Add the user to the samba group
# usermod -a -G smbgroup testuser

# Set the password for your samba user
# smbpasswd -a testuser

Now, create another directory (secure_share) within the /network folder. This new directory will have secured access only to the samba user that we created in the earlier steps.

# cd /network

# mkdir secure_share

# chmod -R 0755 secure_share

# chown -R testuser:smbgroup secure_share

Add the new folder's description in the samba configuration file as performed in the earlier steps:

# vi /etc/samba/smb.conf

[secure_share]

path = /network/secure_share

writable = yes

browsable = yes

guest ok = no

valid users - @smbgroup

Save and exit the editor once the changes are made

Run the "testparm" command once again. You should see the newly added secured folder's description as shown below:

# testparm

Test the secured folder by adding it to your client machine. This time, when you try to access the folder, it will prompt you to enter a "username", "domain" and "password". Provide the details as required and hit "connect" when done.

You should have access to your secured folder now. To further test the folder, create a temp directory (SecureFolder) in it.

You can check the validity of the new folder creation by logging on the samba server and verifying whether the new folder was created or not.

NOTE: The new folder is created and owned by the "testuser" that we created and assigned earlier.

That's it for now, bust stay tuned for much more coming your way soon!

Cheers!!

Quick Start: Setup CentOS 7 as a LAMP Server

2015-01-13T15:28:00.003+05:30

Quick Start: Setup CentOS 7 as a LAMP Server

In this tutorial, I'm going to show you simple steps using which you can setup your own LAMP (Linux-Apache-MySQL-PHP) on a CentOS 7 system.

Prerequisites
CentOS 7 Minimal ISO
Make sure your system has a static IP address assigned to it and a fully qualified domain name.

System Details:
Base OS: CentOS 7 64 Bit (Minimal ISO)
HOSTNAME: master.cloud.com
IPADDRESS: 192.168.0.15

Steps:

LINUX
Start by Installing a CentOS 7 system. You can refer THIS GUIDE for the same.

APACHE WEB SERVER
Once the base OS is ready, we now install the Apache web server package.

# yum install httpd

The httpd package downloads some additional dependencies as well. Hit "Y" and download all the required RPMs as shown.

Next, edit the "httpd conf" file

# vi /etc/httpd/conf/httpd.conf

In this file, you only need to edit at two places:

# Replace ServerAdmin root@localhost
ServerAdmin root@<YOUR_CENTOS_IP_ADDRESS>

# Replace ServerName localhost:80
ServerName <YOUR_CENTOS_IP_ADDRESS>:80

NOTE: ServerName attribute is commented out by default, so remember to uncomment it out first.

Save and exit the editor once the changes are made.

Start and enable the httpd service

# systemctl start httpd

# systemctl enable httpd

Enable the HTTP port (Port 80) on the firewall and reload the firewall.

# firewall-cmd --permanent --add-port=80/tcp

# firewall-cmd --reload

Test your Web Server by typing in the CentOS system's IP address in a browser. You should see the following web page if your settings are done correctly.

http://<YOUR_CENTOS_IP_ADDRESS>

MySQL
Next up we install the MySQL Database. An important point to note here is that with CentOS 7, MariaDB is shipped as the default MySQL Engine unlike the earlier versions where MySQL was the de facto choice.

In this guide, I'll show you how to install both MariaDB and MySQL Server.

First off, its MariaDB

# yum install mariadb mariadb-server

This will install the necessary DB packages for MariaDB, rest of the configuring steps remain the same for MySQL so I'll now show you how to install MySQL Community Edition on a CentOS 7 system.

First off, we download some prerequisites:

# yum install wget

Next, download the MySQL Community Release RPM using the following command:

# wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm

Install the downloaded RPM

# rpm -ivh mysql-community-release-el7-5.noarch.rpm

The RPM installs two repos at /etc/yum.repos.d directory, both will be used to install the MySQL Community Server packages

Once prepped, you can now install the MySQL packages as required

# yum install mysql-server mysql

Start and enable the MySQL service

# systemctl start mysqld

# systemctl enable mysqld

Run the mysql_secure_installation script to set the root password and other necessary parameters

# mysql_secure_installation

Test your MySQL server by logging on the DB:

# mysql -u root -p

PHP
The final install is PHP. Run the following command to install the basic PHP packages:

# yum install php php-mysql

Hit "Y" to download and install the packages

Next, create a simple file to test whether PHP was successfully installed or not

# vi /var/www/html/info.php

<?php phpinfo(); ?>
Restart the httpd service before you proceed.

# systemctl restart httpd

Open up a browser and type in the following. You should see the PHP info page as shown below:

http://<CENTOS_SERVER_IP_ADDRESS>/info.php

Thats it for this post.. but stay tuned for much more coming your way soon..

Cheers.

Quick Start: Setup CentOS 7 as a DHCP Server

2015-01-08T12:44:00.000+05:30

Quick Start: Setup CentOS 7 as a DHCP Server

In our EARLIER POST, we saw how to configure a DNS server on a CentOS 7 system. In this post, we will look at few simple steps that will help you create and configure a functional DHCP Server on your CentOS 7 system.

System Details:
Base OS: CentOS 7 64 Bit (Minimal ISO)
HOSTNAME: master.cloud.com
IPADDRESS: 192.168.0.15

Steps:
First of all, make sure your system has a static IP address assigned to it and a fully qualified domain name.

Next, install the DHCP packages

# yum install dhcp

Edit the dhcpd.conf file and paste the contents as shown below:

# vi /etc/dhcp/dhcpd.conf

# DHCP Server Configuration file.
# see /usr/share/doc/dhcp*/dhcpd.conf.example
# see dhcpd.conf(5) man page
#
# option definitions common to all supported networks...

# specify domain name
option domain-name "cloud.com";
# specify name server's hostname or IP address
option domain-name-servers master.cloud.com;
# default lease time
default-lease-time 600;
# max lease time
max-lease-time 7200;
# this DHCP server to be declared valid
authoritative;

# specify network address and subnet mask
subnet 192.168.0.0 netmask 255.255.255.0 {

# specify the range of lease IP address
range dynamic-bootp 192.168.0.101 192.168.0.150;

# specify broadcast address
option broadcast-address 192.168.0.255;

# specify default gateway
option routers 192.168.0.1;
}

Edit the values highlighted in RED and replace them with your own values. Save and exit the editor once done.

Start and enable the DHCP Service

# systemctl start dhcpd

# systemctl enable dhcpd

You can test your DHCP Server by spinning up another system on the same network. By default, the Network Interface is always set to DHCP so if your settings are right, you should automatically see the IP address for your new system as shown below.

There you have it! A simple DHCP server based on CentOS 7.. Hope this tutorial helps you out.. stay tuned for lots more coming your ways soon!!

Cheers..

Quick Start: Setup CentOS 7 as a DNS Server

2015-01-07T13:32:00.000+05:30

Quick Start: Setup CentOS 7 as a DNS Server

In this tutorial, I'll be showing you simply steps using which you can create and configure a functional DNS Server on your CentOS 7 system.

System Details:
Base OS: CentOS 7 64 Bit (Minimal ISO)
HOSTNAME: master.cloud.com
IPADDRESS: 192.168.0.15

Steps:
First of all, make sure your system has a static IP address assigned to it and a fully qualified domain name.

Next, install the DNS packages:

# yum install bind bind-utils

Edit the "named.conf" file and edit the content as shown below:

# vi /etc/named.conf

# Replace the content in RED with your values
options {
listen-on port 53 { 127.0.0.1; 192.168.0.15;}; ## DNS Server IP
# listen-on-v6 port 53 { ::1; };
directory "/var/named";
dump-file "/var/named/data/cache_dump.db";
statistics-file "/var/named/data/named_stats.txt";
memstatistics-file "/var/named/data/named_mem_stats.txt";
allow-query { localhost; 192.168.0.0/24;}; # IP Subnet

# Add the following lines before the include statement as shown

zone "cloud.com" IN {

type master;

file "forward.cloud";

allow-update { none; };

};

zone "0.168.192.in-addr.arpa" IN {

type master;

file "reverse.cloud";

allow-update { none; };

};

Save the file and exit the editor.
Next, we create the Zone files. There will be two files created, one for the forward lookup and one for the reverse lookup. These files are created in the /var/named directory and their names should be same as mentioned in the named.conf file (Earlier steps)

Forward Zone Lookup File:

# vi /var/named/forward.cloud

Add the following content as shown. Replace the content marked in RED with your matching requirements.

$TTL 86400

@ IN SOA master.cloud.com. root.cloud.com. (

2011071001 ;Serial

3600 ;Refresh

1800 ;Retry

604800 ;Expire

86400 ;Minimum TTL

)

@ IN NS master.cloud.com.

@ IN A 192.168.0.15

@ IN A 192.168.0.102

@ IN A 192.168.0.103

master IN A 192.168.0.15

client1 IN A 192.168.0.101

client2 IN A 192.168.0.102

Save and exit the editor.

Reverse Zone Lookup File:

# vi /var/named/reverse.cloud

$TTL 86400

@ IN SOA master.cloud.com. root.cloud.com. (

2011071001 ;Serial

3600 ;Refresh

1800 ;Retry

604800 ;Expire

86400 ;Minimum TTL

)

@ IN NS master.cloud.com.

@ IN PTR cloud.com

master IN A 192.168.0.15

client1 IN A 192.168.0.101

client2 IN A 192.168.0.102

15 IN PTR master.cloud.com

101 IN PTR client1.cloud.com

102 IN PTR client2.cloud.com

Save and exit the editor.

Enable and start the DNS service:

# systemctl enable named

# systemctl start named

Enable the ports on the Firewall and reload the firewall

# firewall-cmd --permanent --add-port=53/tcp

# firewall-cmd --reload

Verify your DNS:

Check DNS default configuration file for errors:

# named-checkconf /etc/named.conf

Next, check both the forward and reverse lookup files for any errors as well.

# named-checkzone cloud.com /var/named/forward.cloud

# named-checkzone cloud.com /var/named/reverse.cloud

Once checked, modify the network interface file and add the DNS details as shown below:

DNS1=192.168.0.15

NOTE: In my case, I am using multiple DNSs. You can assign multiple DNS values using as shown in the image below.

Alternatively, you can even add the DNS entry in the /etc/resolv.conf file as well for name resolution.

# vi /etc/resolv.conf

nameserver 192.168.0.15

Restart your network to apply the changes

# systemctl restart network

Test the DNS:

Run the following commands to verify whether the DNS can resolve the hostnames correctly or not.

# dig master.cloud.com

You can alternatively run nslookup to verify your Zones

# nslookup cloud.com

There you have it.. a simple and easy to use DNS server on a CentOS 7 System.. Hope this post helps you out! Stay tuned for a lot more coming your way soon!!

Cheers.

Getting started with CentOS 7

2015-01-06T15:51:00.001+05:30

Getting started with CentOS 7

In this tutorial I'll be walking you through simple and easy steps to get started with CentOS 7 as a VM on VirtualBox.

I'm using a CentOS 7 x86-64 BIT Minimal ISO. You can download a copy of the same from HERE.

NOTE: The same steps can be used to install and configure a RHEL 7 as well.

System Details:

HOSTNAME: master.cloud.com
IP ADDRESS: 192.168.0.15
NETMASK: 255.255.255.0
GATEWAY: 192.168.0.1
DNS: 8.8.8.8

On booting form the ISO, select "Install CentOS 7" to get started

Next, select the language that you will use during the entire installation process. Click "Continue" once done

You will now be brought to the "Installation Summary" page. Here, you can select on various options to configure settings for your CentOS system. To begin with, lets set the "Date and Time"

Here, select the "Region" and the "City" as per your requirement. You can optionally even set the "Network Time" as ON. This will pole the time from the nearest network (NTP) server. Once set, click on the "DONE" button on the top left-hand corner to save the settings and return to the "Installation Summary" page.

Similarly, you can select the "Installation Destination" option to select and configure your disk partitions

Here, make sure you select your disk before proceeding further. You can tell if your disk is selected or not by checking for a tick mark across your disk as shown in the image below.

You can provide a custom partition for your CentOS system as well by selecting the "I will configure partitioning" option. Click "Done" once completed

The following screen only appears if you have selected the custom partitioning option earlier. Here, you can set the partitioning scheme as well as create and assign custom mount points based on your requirements.

Click on the "+" icon to get started

In the "Add a new Mount Point" dialog, select a "mount point" from the drop down and provide it with a desired capacity.

NOTE: All values entered in the "Desired Capacity" field are in MBs

Click on "Add mount point" when done

Your new mount point should be displayed as shown in the image below. You can further edit your mount point settings such as change its "File System" or "Desired Type" based on your requirement.

To add more partitions, simply click again on the "+" icon and follow the same procedure highlighted as above.

Add a "SWAP" partition

Add a "ROOT" partition as well. NOTE: If no value is provided in the "Desired Capacity" field, the remainder of free disk space is completely utilized for that partition.

Your final "Partitioning table" should like this by now:
/boot 100 MB
/ 30 GB
swap 1 GB

Click "Done" to complete the disk partitioning.

You will be prompted with a "Summary of changes". Verify and click "Accept changes" to continue

You can even edit the network changes for your system. To do so, select the "Network and Hostname" option as shown

By default, the netowrk is set to DHCP. You can change that by selecting the "Configure" option. You can also change the "Hostname" of your system as shown below.

The following dialog appears if you had selected the "configure" option as mentioned earlier. Here, you can set static IPs for your system as well as other options such as "Automatically connect to network", "All users can connect to the network" etc.

In the "IPv4 Settings" Tab, manually configure the IP based on your requirements as shown below. Once done, click on "Save" to continue.

You should see the network changes applied as shown below.

Once these basic requirements are met, you are now ready to begin with the installation of the OS. Click on "Begin Installation" to start

You can set the "Root Password" and "Add a New User" for your system while the installation continues in the background.

Provide a strong root password. The system will prompt you to click on the "DONE" button twice if you have entered a weak or a dictionary based password.

Follow the same process for adding a new user as well.

Provide a suitable username and a strong password. You can optionally set this user as an "Administrator" based on your requirements. Click "Done" when finished

Once the installation completes, you will be prompted to "Reboot" your system.

Once the system boots up, you can login to it by either the root user credentials or by specifying the user credentials that were set during the install process.

That's it for now... lot's more coming your way soon.. so STAY IN TOUCH!!

Cheers..

Working with Object Stores: S3CMD

2014-09-25T08:44:00.004+05:30

Working with Object Stores: S3CMD

In my previous article we looked at How to get started with Riak-CS, an elastic and open source based Object Store. In this section of the tutorial, I'll be walking you simple steps to setup S3CMD, a tool that can be used to query, add/ delete objects from the object store.

A little overview of S3CMD from its website:

"S3cmd is a free command line tool and client for uploading, retrieving and managing data in Amazon S3 and other cloud storage service providers that use the S3 protocol, such as Google Cloud Storage or DreamHost DreamObjects. It is best suited for power users who are familiar with command line programs. It is also ideal for batch scripts and automated backup to S3, triggered from cron, etc.S3cmd is written in Python. It's an open source project available under GNU Public License v2 (GPLv2) and is free for both commercial and private use. You will only have to pay Amazon for using their storage."

We will be using S3CMD to perform simple tasks on our Object Store such as create buckets, add objects to them, list objects and finally delete objects and buckets.

To get started, first we need to download the latest version of S3CMD (here, I'm using v1.5.0-rc1). Do note that the latest version contains support for Multi-part Upload, a unique technique using which large objects are broken up into smaller, more manageable chunks and then uploaded to the bucket.

Installation

The installation is a fairly straight forward process:

# wget http://sourceforge.net/projects/s3tools/files/s3cmd/1.5.0-rc1/s3cmd-1.5.0-rc1.tar.gz

Un-tar the S3CMD tar file:

# tar -xvzf s3cmd-1.5.0-rc1.tar.gz

Run the following command to install S3CMD tool.

# python setup.py install

Configuration
We now need to configure S3CMD to use our Riak-CS server as the Object Store. To do that interactively, type the following command:

# s3cmd --configure

You may get the following error at this point: "Import Error: Trying to import dateutil.parser"

To solve this error, simply install python-dateutil package as shown below:

# yum install python-dateutil

You may optionally install the python-magic package as well. This package is used by S3CMD to guess the MIME type of the file that you upload to the Object store.

There's no harm if you don't install this package, although you will have to content yourself by facing warning messages from S3CMD, stating: "WARNING: Module python-magic is not available. Guessing MIME types based on file extensions"

# yum install python-magic

Once the pre-requites are met, you can again try to configure S3CMD:

# s3cmd --configure

There are 4 default settings you should change:

Access Key — Use the Riak-CS user access key you generated above.
Secret Key — Use the Riak-CS user secret key you generated above.
Proxy Server — Use your Riak-CS Server IP.
Proxy Port — The default Riak-CS port is 8080

Test the configuration. Make sure you get the success messages as shown below. Finally save these settings.

NOTE: These settings will be saved at /root/.s3cfg

Testing
Now comes the fun part.. where we actually test out the Riak-CS using S3CMD

1) List S3CMD version:

# s3cmd --version

2) Create a Bucket:

Syntax: s3cmd mb s3://<unique_bucket_name>

# s3cmd mb s3://yoyo-01

3) List Buckets:

Syntax: s3cmd ls s3://<unique_bucket_name>

# s3cmd ls

4) Add objects in a Bucket:

Syntax: s3cmd put <filename> s3://<unique_bucket_name>

# s3cmd put /opt/testfile s3://yoyo-01

NOTE: Optionally, you can use "get" to download the object from the Bucket as well. Syntax remains the same.

You can verify whether the file was successfully uploaded or not by listing the contents of the bucket using the ls command followed by the Bucket Name:

# s3cmd ls s3://yoyo-01

5) Check size of Bucket:

Syntax: s3cmd du s3://<unique_bucket_name>

# s3cmd du s3://yoyo-01

6) Syncing Objects to your Bucket:

Here, I created a 1GB dummy file using the dd command. The purpose of this large file was to show you how multi-part upload works out using S3CMD.

Sync is a little different to put command as it is used to "sync" a remote directory with several files in it (preferably with nested sub-directories)

IMP: Each time a sync is executed, it will only upload files that don’t exist or different version at the destination. If the files in the bucket already exist or are the same, there will be no transfer.

Syntax: s3cmd sync <folder> s3://<unique_bucket_name>

# s3cmd sync -r /opt s3://yoyo-01

NOTE: -r means recursive, in this example, it will recursively scan the /opt directory and transfer the objects within it to the Riak-CS. Also note how the 1GB test file that we created in the earlier steps is getting uploaded. Its broken up into 15 MB chunks (this is a default setting and can be changed in the /root/.s3cfg file) and then uploaded to the Riak-CS storage.

The recursive (-r) can be used with other S3CMD commands as well, such as list (ls) to list the folders recursively.

# s3cmd ls -r s3://yoyo-01

7) Deleting Objects:

Syntax: s3cmd del s3://<unique_bucket_name>/<filename>

# s3cmd del s3://yoyo-01/testfile

8) Deleting Buckets:

Syntax: s3cmd rb s3://<unique_bucket_name>

# s3cmd rb s3://yoyo-01

NOTE: You cannot delete a bucket that has any content present in it. Make sure you delete the objects in the Bucket first before you try to remove the Bucket itself.

Well, that's all there is to this post folks! Stay tuned for much more coming your way soon!!

Working with Object Stores: RiakCS

2014-09-23T13:16:00.000+05:30

Working with Object Stores: Riak-CS

We have all heard of the amazing power and flexibility of Amazon's S3 (Simple Storage as a Service) offering.. a simple Object Storage that can be used to virtually to store any content you have up on Amazon's Cloud.

Lucky for us, we have similar products and solutions that can provide us the same S3-like feel on our own in-house infrastructure. Enter Riak-CS!!

I have been working on Riak-CS for quite a short while now and must say, it's really something we all IT admins must try out sometime in our lives... its a unique open source solution that can be used virtually in a same manner as S3, providing a list of features and functions that can be leveraged with real ease.

In this set of tutorials, Ill be showing how to setup Riak-CS on a single Linux box and then work with it using s3cmd tools.

Before we begin, here's a quick intro of the services that we will be working with... first up:

Riak

"Riak is a distributed NoSQL key-value data store that offers extremely high availability, fault tolerance, operational simplicity and scalability. Written in Erlang, Riak has fault tolerance data replication and automatic data distribution across the cluster for performance and resilience"

Riak CS

"Riak CS (Cloud Storage) is simple, open source storage software built on top of Riak. It can be used to build public or private clouds, or as reliable storage to power applications and services. It features:

--Highly available, fault-tolerant storage
--Large object support and multipart upload
--S3-compatible API and authentication
--Multi-tenancy and per-user reporting
--Simple operational model for adding capacity
--Robust stats for monitoring and metrics
--OpenStack integration with Object Storage API and Keystone authentication service"

Stanchion

"Manages requests involving globally unique system entities, such as  buckets and users sent to a Riak instance, for example, to create users or to create or delete buckets"

Setting up Riak-CS is fairly simple and straightforward. Here, Ill be using a CentOS 6.5 64 Bit machine with a hard disk of 50 GB. I have setup the hard disk as a LVM, so that in case the disk runs short of capacirty, I can easily increase its size by adding a new hard drive. To know more about LVM, check out my previous tutorial on the same.

NOTE: You can refer to the Riak Docs for a better understanding of what hardware and OS to use for a Production setup.

Prerequisites

These are optional requirements, but never the less, important ones. Make sure the Riak-CS server has a static set IP and a FQDN hostname as shown below:

NOTE: You can set the FQDN of your server by editing the /etc/hosts file and adding the FQDN there.

# ifconfig

# hostname

Also, make sure you have enough of Hard Drive space available for testing and working. Here, I am using a 50 gig drive and I have set it up as a LVM on my CentOS system

# df -h

Installation
Riak-CS can be installed either by using the source or by the RPMs. Here, I'm using the RPMs cause they are really easy to get going with.

First things first, add the Basho repo using the following command:

# sudo yum install http://yum.basho.com/gpg/basho-release-6-1.noarch.rpm

Once done, install Riak, Riak-CS and Stanchion as shown below:

# sudo yum install riak

# sudo yum install riak-cs

# sudo yum install stanchion

Configurations
Once the packages are installed, there are a few simple steps required to configure Riak-CS out. You can follow the in-depth, step-by-step official guide here as well

Firstly, we need to increase the OS's open files limit. there are two ways to do this:

1) A temporary setting, which is valid ONLY for the current session:
# ulimit -n 65536

2) A permanent setting, done in the limits.conf file:

# vi /etc/security/limits.conf
root soft nofile 65536

root hard nofile 65536

riak soft nofile 65536

riak hard nofile 65536
Save the file and exit the editor.

Next, we need to modify the following config files and add few commands to them:

# vi /etc/riak/app.config

Add this line to the riak_core section, which starts off like this: {riak_core, [

{default_bucket_props, [{allow_mult, true}]},

Next, in the same config file, find the following text and replace that with the contenet as shown below:

Change this --
{storage_backend, riak_kv_bitcask_backend}

To this --
{add_paths, ["/usr/lib/riak-cs/lib/riak_cs-1.5.0/ebin"]},
{storage_backend, riak_cs_kv_multi_backend},
{multi_backend_prefix_list, [{<<"0b:">>, be_blocks}]},
{multi_backend_default, be_default},
{multi_backend, [
{be_default, riak_kv_eleveldb_backend, [
{max_open_files, 50},
{data_root, "/var/lib/riak/leveldb"}
]},
{be_blocks, riak_kv_bitcask_backend, [
{data_root, "/var/lib/riak/bitcask"}
]}

]},

Next, in the same config file, swap the http and https IP addresses with localhost (127.0.0.1) to your Riak Server's IP address (192.168.50.150)

{http, [ {"192.168.50.150", 8098 } ]}
{https, [ {"192.168.50.150", 8098 } ]}

Save the file and exit the editor.

Next, edit the riak-cs config file and change the IP addresses there as well

# vi /etc/riak-cs/app.config

Change the following lines only, replacing the localhost IP with your Riak Server IP:

{cs_ip, "192.168.50.150"}
{riak_ip, "192.168.50.150"}

{stanchion_ip, "192.168.50.150"}

Save and exit the editor.

Follow the same steps for the stanchion config file in /etc/stanchion/app.config as well:

# vi /etc/stanchion/app.config

Change the following lines only, replacing the localhost IP with your Riak Server IP:

{stanchion_ip, "192.168.50.150"}

{riak_ip, "192.168.50.150"}

Save and exit the editor.

Next, edit the following files and change the -name parameter values as shown below:

# vi /etc/riak/vm.args

-name riak@192.168.50.150

# vi /etc/riak-cs/vm.args

-name riak-cs@192.168.50.150

# vi /etc/stanchion/vm.args

-name stanchion@192.168.50.150

All done, we are now ready to start our services.

NOTE: Start them in the correct order as shown below:

# sudo riak start
# sudo stanchion start
# sudo riak-cs start

You can test whether your Riak-CS service is running properly by executing the riak-cs ping command.

# riak-cs ping

You can check riak and stanchion services are functioning correctly or not as well using this same command.

Creating an Admin user
Let's face it, nothing works without an admin user!!

To get started, first edit the /etc/riak-cs/app.config and find the anonymous_user_creation parameter as shown. This is set to false by default. Change it and set it true.

# vi /etc/riak-cs/app.config

{anonymous_user_creation, true},

NOTE: This is a temporary setting that we are changing so that we can create a user. Once the user is created, make sure you set the parameter back to false.

Restart the riak-cs service so that the changes take effect

# riak-cs restart

Next, ruun the following command. This will create a new user named "admin" with an email "admin@cloud.com". it will also generate two VERY IMPORTANT KEYS: an Access Key and a Secret Key

# curl -XPOST http://localhost:8080/riak-cs/user \
-H 'Content-Type: application/json' \

-d '{"email":"admin@cloud.com", "name":"admin"}'

The user's access key and secret key are returned in the key_id and key_secret fields respectively. Take note of these keys as they will be required in the testing step.

In this case, those keys are:
Access key: XNTMDGWOV2VI0EHMMDI6
Secret key: Rvny0twbBMc53pABDvOjY2lxp_Ifv0Yrq4gbTw==

Next, add these keys in BOTH, the riak-cs and stanchion config files as shown:

# vi /etc/riak-cs/app.config

Find the admin_ket and admin_secret parameters and provide the admin_key and admin_secret to them respectively:

{admin_key, "XNTMDGWOV2VI0EHMMDI6"}

{admin_secret, "Rvny0twbBMc53pABDvOjY2lxp_Ifv0Yrq4gbTw=="}

Do the same for the stanchion config file as well located at /etc/stanchion/app.config

Once added, restart both riak-cs and stanchion for the changes to take effect.. and voila.. your Object Store is now ready for use!!!

# sudo stanchion restart

# sudo riak-cs restart

Stay tuned for the next part of this series where we install and configure S3CMD Tools for accessing the Object Store!!

Working with LVMs

2014-09-17T15:15:00.000+05:30

Working with LVMs

LVMs or Logical Volume Management is a really useful alternative towards creating fixed partitions and file systems. Unlike traditional fixed sized partitions, LVM are designed to be added or removed as need be without having to re-organize or spread the data over multiple filesystems.

LVM provides a flexible and high level approach to managing disk space. Instead of each disk drive being split into partitions of fixed sizes onto which fixed size file systems are created, LVM provides a way to group together disk space into logical volumes which can be easily resized and moved. In addition LVM allows administrators to carefully control disk space assigned to different groups of users by allocating distinct volume groups or logical volumes to those users. When the space initially allocated to the volume is exhausted the administrator can simply add more space without having to move the user files to a different file system.

In this tutorial, I'll be showing you simple steps using which you can easily extend your root partition using LVMs. But before we begin, here a look at few LVM related terms and components:

Volume Group (VG)
The Volume Group is the high level container which holds one or more logical volumes and physical volumes.

Physical Volume (PV)
A physical volume represents a storage device such as a disk drive or other storage media.

Logical Volume (LV)
A logical volume is the equivalent to a disk partition and, as with a disk partition, can contain a file system.

Physical Extent (PE)
Each physical volume (PV) is divided into equal size blocks known as physical extents.

Logical Extent (LE)
Each logical volume (LV) is divided into equal size blocks called logical extents.

In the following guide, I'll be demonstrating the steps on a CentOS 6.5 OS. The steps should be the same for most of the popular Linux distributions as well.

First thing to note is that my root partition is already mounted to a logical volume (named /vg_lvm-lv_root). This is essential as the following steps will not work if your root partion is places on a standard partition.

Note down where the root partition is mounted using the following command:

# df -h

you can alternatively check the same using the mount command:

# mount

If you want to see info regarding the existing volume groups(VG), you can use the following command:

# vgdisplay

From the screenshot below, we can see that my Volume Group vg_lvm has a Physical Extent (PE) size of 4MB, and has a total of 7GB available as a total Volume Group (VG) size.

To know more about the individual Logical Volumes(LV) created in a Volume Group, use the following command:

# lvdisplay

As you can see from the screenshot, there are two logical volumes here. Firstly, we have a root logical volume (lv_root) and it is 6.71GB in size. The other is a logical volume for swap (lv_swap) and that is 816MB in size. For the purpose of this tutorial, we will increase the root logical volumes size.

Now that we know what space is being used it is often helpful to understand which devices are providing the space (in other words which devices are being used as physical volumes). To obtain this information we need to run the following command:

# pvdisplay

The following screenshot shows that the space for Volume Group (vg_lvm) is provided by a Physical Volume (PV) located at /dev/sda2

Adding additional space to root partition
I have attached a 32GB HDD to my CentOS machine. I can view my newly added HDD as a disk using the fdisk command:

# fdisk -l

Before I use this new disk as a logical volume, its always good to format it and create a Linux partition on it.

# fdisk /dev/sdb

Make sure the new disk got formatted properly by using the fdisk command as shown below:

# fdisk -l

Now that the new disk is formatted and ready, lets add that o our Volume Group.

First, we convert this disk into a physical volume. This is achieved using the pvcreate command:

# pvcreate /dev/sdb1

With the physical volume created we now need to add it to the volume group (vg_lvm) using the vgextend command:

# vgextend vg_lvm /dev/sdb1

Use the pvdisplay command to view the newly created Physical Volume

# pvdisplay /dev/sdb1

The new physical volume has now been added to the volume group and is ready to be allocated to a logical volume.

We use the lvextend tool to provide the size by which we wish to extend the volume. In this case we want to extend the size of lv_root by approx 31 GB (Since we added a HDD of 32GB, use a disk space of approx 31GB for the logical volume).

NOTE that we need to provide the path to the logical volume which can be obtained from the lvdisplay command

# lvextend -L+31G /dev/vg_lvm/lvm_root

Last step in the process is to simply resize the file system on the new logical volume so that it now starts using the additional 30GB of free space as well.

# resize2fs /dev/vg_lvm/lv_root

Check your root partitions size now, and it should show you ample of new space to work with!!

# df -h

Getting started with KVM- Networking and Storage

2014-08-27T09:50:00.000+05:30

Getting Started with KVM

Part 3: Networking and Storage

In this set of tutorial, I'll be showing some simple steps to creating Virtual Machines easily using KVM (Kernel-based Virtual Machine)

NOTE: before we begin, make sure that your host system has a functional Intel VT or AMD-V flag set. This is essential for creating Virtual Machines on your Host system.

In my PREVIOUS POST, we looked at the two ways in which we can create VMs for KVM. This is a continuation of that post where we look at some of the advanced Networking and Storage options provided for KVM.

1) Virtual Networks
The first thing we are going to do is create Virtual Networks for our VM to use. In the PREVIOUS POST, we created a VM called "My-Debian-VM2", I'll be demonstrating these steps on this VM itself. So without further adieu, lets get started!!

The first thing to do is select the "localhost (QEMU)" option from our VMM console.

This brings up the following "Connection Details" Page. Here you can create Virtual Networks, assign Storage Pools, Add new Network Interfaces etc. You can even monitor the performance of your KVM using the basic performance graphs that are generated here.

For our first exercise, we need to select the "Virtual Networks" Tab. There is already a Virtual Network created here by default, and as you can see, its got few Network configurations done to it, such as DHCP start and end IP addresses, Forwarding enabled and set to NAT etc. What we are going to do is we will "Add" a fresh Virtual Network here. To do that, click on the "+" icon as shown:

This will present a dialog that will help you setup your Virtual Network with ease. To begin, click "Forward"

Provide a suitable name for the Virtual Network. In my case, I'm calling it a "VM_br0_NET"

Next, provide it a suitable "IPv4 Address Space". You need to provide any valid IP range here, provided they belong to the 192.198.xxx.xxx, 172.16.xxx.xxx or 10.xxx.xxx.xxx IP segments. I have provided for a "172.16.32.0/24" network.

The wizard will automatically provide you your Virtual Network details such as Broadcast and Gateway IPs.

In the next section, you can set the DHCP range for your Virtual Network. This is optional, but helps to set it out never the less. You need to enter the "start" and the "end" IP addresses from your Virtual Network IP range as shown.

Next, select how the Virtual Network should connect to a Physical Network on your KVM Host. You can either opt to set the Virtual Network as "Isolated" or have the Virtual Network "Forwarded" to a Physical NIC as we are about to do.

In my case, I have forwarded the Virtual Network on my eth0 device and Ill be using NAT as my forwarding mode.

Review the changes and click "Finish" to create the Virtual Network.

Your new Virtual Network will be created and displayed as below:

Perform the same steps for the second NIC (eth1) as well. Here, I created a similar Virtual Network called "VM_br1_NET" and provided it an IP range of "10.1.0.0/24"

Once the Virtual Networks are created, "Close" the KVM Connections Manager window.

We now have to add the Virtual Network to our VM. To do so, sect the VM (My-Debian-VM2) as shown below:

This pops up a "Virtual Machine Settings" page. Here, you can configure your VMs Virtual Hardware settings, add new Virtual Hardware, etc..

In our case, select the "Network Card" as shown. Next, change the "Source Device" from "Bridge (br0)" to the Virtual Network "VM_br0_NET" that we created earlier.

Do not change the "Device Model" settings.

Perform the same change for the second NIC as well. Change "Bridge (br1)" to "VM_br1_NET" Virtual Network. Once done, hit "Apply" and exit the VM Manager Console.

2) Storage Pools
The second thing that we will configure for our VMs is a Storage Pool. Storage Pools are logical storage groups that can contain one or more volumes or virtual disks in various formats.

Open the Virtual Machine Manager (VMM). Click Edit > Host Details > Storage tab.

This will open the Storage section. In the left column, you will see all of your listed storage pools. By default, there will be only one, default, located under /var (This is a default location). In the screenshot below, 45% of the storage pool is currently in use. As you can see, this storage pool contains the ISO image for our Guest OS, along with the actual guest OS disks (virtual disks) as well.

First, lets add some Storage Pools for our tutorial. Click on the "+" icon as shown.

A dialog box appears which will help you setup your Storage Pools. First thing is to choose a "Name" and "Type". You can use directories (mount points), including NFS, SCSI and iSCSI storage, LVM, and even physical disks.

In my case, I've provided a name "StoragePool1" and am providing a "Directory" as my pool choice.

NOTE: Before starting this activity, I have created and mounted a separate HDD to my KVM Host. This new HDD (/dev/sdb1) is mounted locally to a directory called "StoragePool1" which contains a folder within it called "MyNewVMDirectory".

In the next section, browse to the required folder that you want to use as your Storage Pool destination folder (../StoragePool1/MyNewVMDirectory). Once done, click "Finish"

Your new Storage Pools is now ready for use. The next step is to create a "New Volume" and attach that volume to our VM. to do so, select the "New Volume" option as shown below:

This pops up a dialog that will help you create your Virtual Disks (aka volumes). The disks will be suffixed with ".img", but they can be formatted any which way you choose to. You may want to use qcow2, raw or vmdk, for instance. The format will determine if you can use the virtual machines with other solutions such as VMware Player/ Workstation etc.

Provide a suitable name for your new Volume (Debian-Disk-2). Select "Max Capacity" and "Allocation" units as well.

"Max Capacity" is the max size of the volume. "Allocation" is how much you want to allocate now. Basically, if you choose zero size, you are creating a sparse file with 0 size. You can allocate any value between zero and max. capacity.

Once done, click "Finish" to create your Volume.

Your new Volume is created and ready to be used by your VM.

To add a new Volume to your VM, go to Virtual Machine Manager (VMM) > {VM_NAME} > Virtual Machine Settings page. As you can see from the screenshot below, there is only one Disk allocated to the VM (My-Debian-VM2). Click on "Add Hardware" to begin.

Next, select the "Storage" tab. You can either create a new Disk using this wizard or attach an already existing disk as well. "Browse" for the new Volume that we created in our earlier steps.

As you can see, I have selected my Storage Pool "StoragePool1" and within that, I have my new Volume aka "Debian-Disk-2.img"

Once added, you can choose to change the "Disk Type" however I prefer to keep these values to their defaults. Click "Finish" when done.

You should see your newly added Disk attached to your VM as shown below. You are now ready to Power ON your VM and install the guest OS.

Just a few screenshots I took during the Install process. AS you can see here, I am asked to select the Primary Network Interface for my Debian guest OS. On selecting either of the cards, the network automatically assigns the VM an IP address form the DHCP pool that we created in our earlier steps.

NOTE: Here, eth0 will provide my Debian VM with internet access, while eth1 is routed on my LAN network, so remains isolated from the web.

Secondly, here's a screen grab of the two HDDs that we used for this VM. I created partitions manually on both the disks and formatted both to ext4 file systems.

Thahts all there is to it folks!! Hope this tutorial proves helpful to you in setting up your very own KVM Hosts...

Getting started with KVM- Creating VMs

2014-08-25T12:14:00.000+05:30

Getting Started with KVM

Part 2: Creating Virtual Machines

In this set of tutorial, I'll be showing some simple steps to creating Virtual Machines easily using KVM (Kernel-based Virtual Machine)

NOTE: before we begin, make sure that your host system has a functional Intel VT or AMD-V flag set. This is essential for creating Virtual Machines on your Host system.

For the steps to install and configure KVM, have a look at my EARLIER POST

There are basically two ways to go about creating VMs using KVM. I'll be demonstrating both the steps below:

1) Using Command Line
The first thing to do here is create a separate directory where we will store all our VMs.
Create a new folder at /var/lib/libvirt directory for our VMs.

# cd /var/lib/libvirt

# mkdir VM

Next, download your favorite OSs ISO file. Here, I'm going to be using Debian as my Guest OS.

# wget http://cdimage.debian.org/debian-cd/7.6.0/amd64/iso-cd/debian-7.6.0-amd64-netinst.iso

We will use the /var/lib/libvirt/images directory to store the ISOs that will be used to create our VMs.

# mv debian-7.6.0-amd64-netinst.iso /var/lib/libvirt/images

We are now ready to create our VM. We will be using virt-install for this.

The virt-install utility must be run as root and accepts a wide range of command-line arguments that are used to provide configuration information related to the virtual machine being created. Some of these command-line options are mandatory (specifically name, ram and disk storage must be provided) while others are optional.

In the following example, I will be creating a Debian VM named "My-Debian-VM1" with "1" vCPU and "512MB" RAM with a Disk of "10GB".

# virt-install \
-n My-Debian-VM1 \
--description "My First Debian VM using KVM" \
--ram=512 \
--vcpus=1 \--disk path=/var/lib/libvirt/VM/My-Debian-VM1.img,size=10 \
-c /var/lib/libvirt/images/debian-7.6.0-amd64-netinst.iso \

--graphics vnc \

--network bridge:br0 \

--network bridge:br1 \

--os-type linux \

--os-variant=debiansqueeze \
--hvm \

What do these parameters mean??

-n My-Debian-VM1 : The name of the VM
--description "My First Debian VM using KVM" : The long description of the VM. This is an optional requirement
--ram=512 : Amount of memory allocated to the VM i.e. 512MB

--vcpus=1 : The number of virtual CPU(s) for the VM i.e. 1vCPU

--disk path=/var/lib/libvirt/VM/My-Debian-VM1.img,size=10 : This is is the image file for the VM, the size is specified in GBs
-c /var/lib/libvirt/images/debian-7.6.0-amd64-netinst.iso : Install debian-7.6.0 from an iso the location parameter

--graphics vnc : Use VNC to access installation using vnc server/viewer from your local Linux/OSX/Unix/Windows desktop
--network bridge:br0 : Create a network bridge using br0

--network bridge:br1 : Create a network bridge using br1
--os-type linux : The VM OS type

--os-variant=debiansqueeze : The VM OS variant

--hvm : Request the use of full virtualization

Once your Vm is built, you can access it using the Virtual Machine Manager (VMM) as shown below:

Navigate to Applications > System Tools > Virtual Machine Manager

You should see the newly created VM running. Simply double click on it to view the console.

You can now install your Guest OS just like you would do on a normal server. Just remember that if you want to release the control of the mouse from the Guest OS, use the key combination LEFT CTRL + LEFT ALT

2) Using GUI
This is another method using which we can create VMs. Here I'll be using a tool called Virtual Machine Manager.

Navigate to Applications > System Tools > Virtual Machine Manager

Select the new VM icon to get started

Enter the "Name" of your VM and also select how you would like to install the Guest OS. You can use local storage in the form of ISO images or CD/DVDs, HTTP or FTP servers, NFS area, or PXE. In this example, we will use the local install media itself for our Guest OS.

Next, choose either a physical device or an image. In our case, we will use an ISO. Then, choose the OS type and the version. This does not have to be precise, but an educated guess is good. It allows additional virtual machine optimization.

Browse for the ISO of your Guest OS (In this case, im using Debian v.7)

The final outcome should look something like this:

Allocate the required "RAM" and "CPU"

Next, provide a "Storage" for your VM. You can either create a new disk or use an existing one as well. You can set the size and pre-allocate all space, which is good for performance, plus it reduces fragmentation.

The final step provides a summary of your VMs settings, with some advanced options. You can change your network type, set a fixed Mac address, choose the virtualization type, and the target architecture here.

NOTE: I have purposely selected the check box "Customize configuration before install" as I have to add a secondary NIC card to my VM as well. I can do this step later as well after the VM is created with the Guest OS, but I prefer doing it at the beginning itself.

If you have selected the check box "Customize configuration before install" from the previous step, then this should pop up on your screen now. This are basically the "Settings" of your VM. To add a new NIC, we need to select the "Add hardware" option as shown below:

Next, select the "Network" tab and alongside, select the correct "Host Device" (in this case, its br1). Once done, click Finish.

Verify the new network card was successfully added. Next, click on "Begin Installation" to start the VMs boot up process.

You should see your new VM power ON in the Virtual Machine Manager

The next steps are pretty easy and trivial. Install your Guest OS as you would do it on a normal server. Once again, just remember that if you want to release the control of the mouse from the Guest OS, use the key combination LEFT CTRL + LEFT ALT

Once you power ON the Guest OS, the two NICs will be assigned to it, however, there will be no IP address available as we have not configured the Networking part (DHCP/ NAT/ Routing etc) of KVM yet.

In the next post, I'll show you How to configure the Networking and Storage for KVM.. so stay tuned..!!