Bleed Red

2016-07-07T11:42:00.003+05:30

Logger Module: INFO and ERROR for your BASH scripts

G=$(tput setaf 2) # Green for info

R=$(tput setaf 1) # Red for error

M=$(tput setaf 5) # Magenta for others

N=$(tput sgr0) # Reset the color

LOGDIR="/var/log"

LOG_FILE=mylog.log

log_dir_check()

{

if [ -d $LOGDIR ]

then

echo "Directory already existing"

else

mkdir -p $LOGDIR

}

log_info()

{

local msg="$1"

local tstamp=`date '+%b%e %H:%M:%S'`

echo -e "$G$tstamp:[INFO] $msg$N" 2>&1 | tee -a $LOGDIR/$LOG_FILE

}

log_error()

{

local msg="$1"

local tstamp=`date '+%b%e %H:%M:%S'`

echo -e "$R$tstamp:[ERROR] $msg$N" 2>&1 | tee -a $LOGDIR/$LOG_FILE

}

log_dir_check

log_error "This is an error"

log_info "This is an info"

onscreen and log o/p

# cat /var/log/mylog.log
Jul 8 02:48:49:[ERROR] This is an error
Jul 8 02:48:49:[INFO] This is an info

Hadoop - A Transcript

2015-07-23T11:36:00.002+05:30

Hadoop cluster:

Hadoop clusters integrate MapReduce and HDFS. This is core Hadoop. Each portion of a Hadoop cluster has a specific job and runs on a specific machine. A Hadoop cluster runs on a typical master/slave architecture with the master controlling all the data management, communication, and processing. In Hadoop, the master is called the name node. The slave is called the data node and does all the work. A cluster usually has one name node. However, in newer versions of Hadoop, there can be a secondary name node used for backup and for failover. The name node controls the whole cluster. It contains the master schema and knows the name and location in the cluster of every file on the HDFS system. The name node does not contain any data or files itself. Any client application coordinates communication to the cluster through the name node when locating, moving, copying, or deleting a file. The name node will respond by returning a list of data nodes where the data can be found. For processing MapReduce jobs, the name node contains the job tracker. The job tracker assigns the processing for MapReduce jobs to nodes across the cluster. Slave or data nodes hold all the data from the HDFS cluster and perform all the computation. Each data node has a task tracker. The task tracker monitors all the MapReduce processing within its data node and communicates results back to its name node. Data nodes do not communicate with each other. Only the name node knows how many data nodes are in a Hadoop cluster.
Hadoop clusters integrate MapReduce and HDFS. A Hadoop cluster runs on a typical master/slave architecture. The master controls all the data management. The name node controls the whole cluster. The name node does not contain any files itself. The name node contains the job tracker. The job tracker assigns the processing for MapReduce jobs to data nodes across the cluster. The slave (data) nodes hold all the data from the HDFS. Each data node has a task tracker. The task tracker monitors all MapReduce processing. Data nodes do not communicate with each other.

Hadoop, by default, is configured to run in local standalone mode or as a single node cluster. This is the easiest way to learn Hadoop. For more advanced uses, Hadoop can be configured in pseudo-distributed mode or fully-distributed mode. As you may guess, both these modes require more complex configuration than when running as a single node cluster. Also, when running in fully-distributed mode, the data nodes need to be configured to communicate with the name node. When running in pseudo-distributed mode, Hadoop runs each of its daemons in its own Java thread. This simulates running a true cluster, but on a small scale. Data nodes and their associated task trackers are virtual. They exist on the same physical machine as the name node. Pseudo-distributed mode also differs from local standalone mode as in pseudo-distributed mode, the HDFS is used rather than the native file system in standalone mode. For pseudo-distributed mode, there will be single master node with a single job tracker, which is responsible for scheduling tasks of that job to the data nodes. The job tracker also monitors the progress of the slave nodes and assigns the re-execution of unsuccessful tasks of slave nodes. For a single slave node, there will be a task tracker to execute the tasks as per the master node's directions. When setting up Hadoop in pseudo-distributed mode, three configuration changes need to be made. One to each of the core-site, hdfs-site, and the mapred-site configuration files.

Hadoop, by default, is configured to run in the local standalone mode. Hadoop may be configured in the pseudo-distributed mode. Hadoop may be configured in the fully distributed mode. When running in the fully distributed mode, data nodes need to be configured to communicate with the name node. In the pseudo-distributed mode, Hadoop runs each of its daemons in its own Java thread. Data nodes and their associated task trackers are virtual. The pseudo-distributed mode also differs from the local standalone mode. HDFS uses the pseudo-distributed mode rather than the native file system. In the pseudo-distributed mode, there will be a single master node. The job tracker also monitors the progress of slave nodes. For a single slave node, there will be a single task tracker.

Hadoop creation and evolution:

Since core Hadoop is so sparse, there are a myriad of tools and technologies that are used to augment and in some cases improve Hadoop's core technologies. There are so many of these tools, it is impossible to count them all. In this area of Hadoop development, the virtual space where tools are developed for Hadoop has a name – it's called the Hadoop ecosystem. In the Hadoop ecosystem, there are commercial vendors creating proprietary tools. There are also developers and researchers creating small niche applications. When you start to develop in Hadoop, and perhaps start to customize your own Hadoop solution, you become part of this Hadoop ecosystem as well. The HDFS has alternatives and supplements that work within the Hadoop ecosystem. For example, Red Hat offers a product for large-scale networking attached system and storing files named GlusterFS. It was originally developed by a company named Gluster Incorporated. Red Hat acquired GlusterFS after their purchase of Gluster Incorporated in 2011. The next year, Red Hat Storage server was released as an enterprise supported version of the GlusterFS integrated with Red Hat Enterprise Linux. In addition, Ceph is a storage platform architected to provide file, block, and object storage – all from a single cluster of servers. The main objective of Ceph is to be completely distributed and eliminate a single point of failure. Also Ceph attempts to be exponentially scalable into the Exabyte level. Like many tools in the ecosystem, Ceph is free. Ceph replicates data making it fault tolerant. A drawback of Ceph is it does not run well on certain versions of Hadoop.
In the Hadoop ecosystem, there are tools that augment or replace portions of Hadoop. These tools exist in a virtual space called the Hadoop ecosystem. In the Hadoop ecosystem, there are large commercial vendors creating proprietary tools. There are also developers and researchers creating small niche applications. The HDFS has alternatives and supplements. Red Hat offers a product for large-scale network storage called the GlusterFS. Ceph is a storage platform architecture that provides file, block, and object storage. Ceph attempts to be exponentially scalable. Ceph replicates data, making it fault tolerant.

Apache MapReduce also has many supplementary tools. Apache Pig is an engine used for parallelizing multiple data flows on Hadoop. Pig even has its own language, aptly named Pig Latin. Pig Latin is used to express these data flows. Pig Latin gives developers the ability to develop their own functions for writing, processing, and reading data. Pig Latin also includes familiar operators for legacy data operations such as sort, filter, and join. Pig Latin runs over Hadoop and by default uses the Hadoop Distributed File System and Hadoop MapReduce. There are also tools that could be used for developers who like to use SQL. For example, Apache Hive is a data warehousing infrastructure developed by Facebook. Hive is an OLAP tool used for data aggregation, deep queries, and data analysis. It also has a SQL like language called HiveQL. Facebook has recently open sourced Presto. Presto is a SQL engine that is up to five times faster than Hive for querying very large files in the Hadoop Distributed File System and other file systems in the Hadoop ecosystem. Scheduling software can be found in the Hadoop ecosystem. Apache Oozie is a workflow scheduling system for MapReduce jobs. Oozie uses direct acyclical graphs. Oozie Coordinator can launch MapReduce jobs one at a time or by data availability. LinkedIn also has a workflow manager called Azkaban. Azkaban is a batch scheduler that is a confluence of cron and make utilities and runs under a graphical user interface.
MapReduce also has many supplementary tools. Apache Pig is an engine used for analyzing multiple data flows on Hadoop. Pig Latin gives developers the ability to develop their own functions. Pig Latin also includes familiar operators for legacy data operations. Pig Latin runs over Hadoop. Apache Hive is a data warehousing infrastructure developed by Facebook. Apache Hive is an OLAP tool used for data aggregation. It has a SQL-like language called HiveQL. Facebook has recently open sourced Presto. Presto is a SQL engine that is up to five times faster than Apache Hive. Scheduling software can also be found in the Hadoop ecosystem. Apache Oozie is a workflow scheduling system for MapReduce jobs. LinkedIn also has a workflow scheduling system called Azkaban. Azkaban is a batch scheduler. Azkaban has a user-friendly graphical user interface.

Use of YARN:

Hadoop YARN stands for Yet Another Resource Negotiator. To understand YARN, you first need to understand the evolution of Hadoop. Apache Hadoop has become the de facto open source platform since 2005. Large companies, such as Yahoo!, Facebook, and EBay, have used Hadoop at the core of their Big Data operations. The widespread use of Hadoop has led to the identification of the limitations within the core framework. Many of these limitations were solved in the next large Hadoop release – commonly called Hadoop 2. Hadoop 2 contains significant improvements in the HDFS distributed store layer, especially in the area of snapshots, NFS, and high availability. YARN is the next generation of the MapReduce computing framework built from the ground up, based on the lessons learned from the first version of Hadoop. YARN has been running in Hadoop installations for about two years. Yahoo! and Facebook are two notable large clusters that use YARN. The first version of Hadoop was built for large Big Data batch applications. A single application, no matter if it was batch, interactive, or online, was forced to use the same infrastructure. Basically, all usage patterns must use the same infrastructure. This forced the creation of silos to manage the mixed workload. In this scenario, the jobtracker manages cluster resources and job scheduling, and the per-node tasktracker manages the tasks of its node. This design has its limitations; everything is forced to look like MapReduce. There is a theoretical limit to the size of the cluster and the tasks it can process. Cluster failure can kill running or queued jobs, and resource utilization is not optimal.
Hadoop YARN stands for "Yet Another Resource Negotiator." The widespread use of Hadoop led to the identification of limitations within the core framework. Many of these limitations were solved in the next large Hadoop release. YARN is the next generation of the MapReduce computing framework. Yahoo! and Facebook are two notable large clusters that use YARN. The first version of Hadoop was built for large Big Data batch applications. All usage patterns must use the same infrastructure. This forced the creation of silos to manage the mixed workload. The job tracker manager clusters resources and job scheduling.

Hadoop 2 solved many problems of the original Hadoop release. Hadoop 2 is a multipurpose platform consisting of multiple different kinds of jobs such as batch, interactive, streaming, online, et cetera. In Hadoop 1, MapReduce was responsible for data processing as well as cluster resource management. In Hadoop 2, MapReduce is only responsible for data processing. YARN has been added to Hadoop 2 to handle cluster resource management. The YARN architecture includes: a resource manager, RM – that acts as a central agent, managing and allocating cluster resources; a node manager, NM – that manages and enforces node resources; and an application master, or an AM that manages each application life cycle and task scheduling. YARN takes Hadoop beyond batch and allows all of the data to be stored in one place. All types of applications – batch, applications, online, streaming, et cetera – can now all be run natively within Hadoop. YARN is the layer that sits between the running applications and a HDFS2's redundant, reliable storage. The benefits of YARN are many. With YARN, new applications and services are possible. Cluster utilization is also greatly improved since Hadoop 1. Since cluster utilization is optimized, there is no theoretical limit to the size a cluster can grow or how many tasks can run on it. YARN also makes the cluster more agile and able to share services. Finally, since YARN separates MapReduce from the HDFS, the Hadoop environment is much more suitable for applications that can't wait for other batch jobs to finish.
Hadoop 2 solved many of the problems of the original Hadoop release. MapReduce was responsible for data processing as well as cluster resource management. Now MapReduce is only responsible for data processing. The YARN architecture includes a resource manager. YARN takes Hadoop beyond batch processing and allows all of the data to be stored in one place. All types of applications from batch, application, online, and streaming can now run natively in Hadoop. YARN is the layer that sits between the running applications and the HDFS2's redundant and reliable storage. The benefits of YARN are many. Cluster utilization is also greatly improved from Hadoop 1. There is no theoretical limit to the size a cluster can grow. YARN also makes the cluster more agile. The Hadoop environment is much more suitable for applications that cannot wait for other batch jobs to finish.

Components of Hadoop:

You may be surprised how sparse it might seem at the core of Hadoop, there are only a few core components. One of the most alluring features of Hadoop is how extensible it is. Very few organizations run only core Hadoop. In this topic, I illustrate and describe the components of core Hadoop. By the end of this topic, you will know the different parts of Hadoop and what purposes they serve. Core Hadoop keeps its data in what is called the Hadoop Distributed File System also known as the HDFS. In the HDFS, data is stored as large blocks inside the Hadoop cluster across different physical systems or nodes. Data is also replicated in the HDFS meaning if a node containing a block of data goes down or becomes otherwise unreachable, that block of data can be obtained on a different data node because data is redundantly distributed across several nodes. The HDFS ensures data protection, performance, and reliability. The HDFS can be standalone meaning it has other uses outside of Hadoop. MapReduce is a software framework and programming model first developed by Google in 2004. The model is inspired by a mapping and reducing algorithm that is common in functional programming. MapReduce is used to simplify the processing of very large datasets over large clusters – thousands of nodes of inexpensive off-the-shelf computers in a fault-tolerant, reliable way. Computational processing takes place on all types of data – structured, semi-structured, and unstructured. Hadoop MapReduce has been a highly popular and accepted parallel computing algorithm. Because of this, MapReduce is one of the core components of Hadoop and is generally accepted to be a large reason for the success of the Hadoop framework. Hadoop MapReduce is at the center of many third-party commercial distributions.
Core Hadoop keeps its data in what is called the Hadoop Distributed File System. Data is stored as large blocks. Data is also replicated in the HDFS. Data is redundantly distributed across several nodes. The HDFS can be standalone. MapReduce is a programming model and software framework. It is inspired by the map and reduce functions commonly used in functional programming. Computational processing occurs on both structured and unstructured data. It is a core component of the Hadoop platform.

Hive – an Apache project – is an OLAP framework built over the Hadoop HDFS and is designed for scalable data analysis, querying, and managing large amounts of data. Hive is not part of core Hadoop. Its data structure is similar to traditional database management systems with tables, which supports queries like unions, joins, select queries and subqueries. HiveQL – a query language similar to SQL – is used to query the Hive database. HiveQL supports DDL operations – creation, insertion of records – and does not allow DML operations such as deletions and update of records. Access methods such as CLI, JDBC, and ODBC, and Thrift are used to access the data in the Apache Hive framework and interface with external systems for business continuality. In Hive, the basic structural unit is a table, which has an associated HDFS directory. Each table has more than one partition and the data is evenly distributed within the HDFS through subdirectories. Each partition is further divided in the buckets that represent the hash of column data and is stored as a file inside a partition. Hive is popularly used by organizations for data mining, ad hoc querying, reporting, and research of latest trends. Mostly used for batch jobs over large sets of append-only data like web logs, Hive is known for scalability, extensibility, fault tolerance, and loose coupling with its input format. Like Hive, Apache Pig is a tool used for data extraction. Pig is not part of core Hadoop and must be installed separately. Pig has its own language called Pig Latin and uses a SQL-like interface to allow the developer access data. Users like Pig as it's simple and has a look and feel of legacy tools for legacy database management systems. Developers can use languages such as Java, JavaScript, and Python to write custom operations called UDFs or user-defined functions. Pig and Hive are the two most popular data management tools in the Apache Hadoop ecosystem.
Apache Hive, an Apache project, is an OLAP framework built over the Hadoop HDFS. Apache Hive is not part of core Hadoop. Its data structure is similar to traditional relational databases with tables. HiveQL is a query language similar to SQL. HiveQL supports DDL operations, such as the creation and insertion of records. In Apache Hive, the basic structural unit is a table, which has an associated HDFS directory. Each partition is further divided into buckets. Apache Hive is popularly used by organizations for data mining. Apache Hive is known for scalability, extensibility, and fault tolerance. Apache Pig is a tool used for data extraction. Apache Pig is not part of core Hadoop and must be installed separately. Multiple versions of Apache Pig exist. Users like Apache Pig as it is simple and has the look and feel of legacy tools. Apache Pig and Apache Hive are the two most popular data management tools in the Apache Hadoop ecosystem.

Different types of data:

In this demo, we are going to talk about the three different types of data – structured data, semi-structured data, and unstructured data – and talk about how understanding the differences between the three is important for your understanding of Big Data and Hadoop in general. Let's start with structured data. I'm going to connect to Oracle and look at an Oracle table. Let's look at a table called ledger. So, if I type in desc ledger;, I get some metadata about the table itself. Now this is structured data. Structured data has a schema and it has a specific order in which it's stored. Now here in a relational database management system, we have structured data and tables where we actually have a table name and here we have names of columns and we have their data types. So up until just very recently, we would just call this data; but in the world of Big Data, we call this structured data because it has a predefined format, it has a predefined place – it has a predefined place where its word is stored, and it has some schema information and some metadata information. And it also has a predetermined way that we can get data out of it such as if I do a select person from ledger; by running the SQL statement, I would get the PERSON column displayed back to me from the ledger table. So in addition to having metadata and having predefined formats, we also have, basically, a predefined way in which we could interact with the data, in this case being SQL. Okay, let's Close this,
The presenter opens the Run SQL Command Line window. In the window, the presenter types the following lines of code:
SQL> connect system/december23
Connected.
SQL> desc ledger;
A table showing information about the names and types of structured data appears.
The presenter then types the following command in the window:
SQL> select person from ledger;
The window now shows information about several people.

and let's talk about semi-structured data. What I have here is I have an XML document – that's a decent example of something that we would call semi-structured. Now we have some sort of information about the data itself. For example, if you look at the root element of this XML document, we could say that we have an XML document of BradyGirls that has specific Girl elements and within the Girl element we got an FName, an LName, an Age, Hobby, and School. So we have some sort of structure about what this data is, but it's not as detailed as what we would expect to see when we are looking at the Oracle table. Again, when we looked at the Oracle table, we had structure. What we had – metadata that described not only what these data elements were, but exactly what data types they are. And also within Oracle, we had a predefined way in which it was actually stored. Here not so much because if you look at each of these Girl elements, it doesn't really define, you know, where they came from. They could come from different sources like, for example, this semi-structured data actually could have come from a more structured data source being a database table or could have come from something a little bit more unstructured such as, maybe, a PDF file or, maybe, even an Excel spreadsheet. So semi-structured has some sort of identification about the different data elements, but it doesn't have anything that shows what data types are or, maybe, the source of the destination. So let me Close this.
The presenter opens the semistructured-Notepad file. The file contains the following xml code:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href=https://yorkville.rice.iit,.edu/fall2014/529/firstxsl.xsl?>
<BradyGirls>
<Girl>
<FName>Marsha</FName>
<LName>Brady</LName>
<Age>17</Age>
<Hobby>Cheerleading</Hobby>
<School>Westdale High</School>
</Girl>
<Girl>
<FName>Jane</FName>
<LName>Brady</LName>
<Age>15</Age>
<Hobby>Sports</Hobby>
<School>Fillmore Jr</School>
</Girl>
<Girl>
<FName>Cindy</FName>
<LName>Brady</LName>
<Age>9<Age>
<Hobby>Doll</Hobby>
<School>Clinton Elementary</School>
</Girl>
</BradyGirls>

The third type of data is called unstructured data. Let me click this icon and a photograph will open. Now by definition, unstructured data is data that's not structured or not semi-structured, which means it's kind of a huge catch all for things that we can't categorize. Now remember in a structured data system, we have an overlying schema or data dictionary that tells us a lot about our data. In the Oracle example we went over, we had a schema that described the table names, the column names, the data types of the columns, and even some areas that described the relationship between the tables. In the XML document, the example we went over of semi-structured data is we had a little bit less. We had only elements that describe what the name of the data was, nothing about the source and the destination of the data. Now here in an unstructured system, we have no overlying schema at all. There is nothing that describes this data or how we can describe this data. There's some kind of application.
The presenter then opens a photo titled unstructured. The photo shows a husband and wife with two kids.

So what we try to do is we try to make unstructured data a little bit more structured like, for example, what we can do, perhaps, is we can take this photograph and insert it in a structured system and a relational database management table. But, if we were to do that, we would lose all meaning of the photo itself. We would just know that we have a photo saved in a table. We would have zero context. So what we can do is we can try to derive some properties about this photo. We can say this photo has four people in it and we could describe...maybe put in some columns that would describe if they are wearing shoes or not or if they are wearing jewelry or not or if they are wearing a hat or some sort of ribbon in their hair. And we would be able to measure those and actually put them in a more structured system. But even if we did that, we still lose a lot of the intangible content of this and the intangible context. For example, what is the weather? Is it raining? Is it snowing? Is the sun shining? What is the mood of the people in this photo? Look at each individual person and determine what their mood is. Also, we would lose things such as if this was taken in a rural setting or if this was taken in an urban setting. What kind of clothes are they wearing? Are they wearing professional clothes? Are they wearing casual clothes? Are they wearing formal clothes? Now these are all the intangibles that Big Data needs because these intangibles drive product offerings and they drive marketing campaigns. Now this is something that Hadoop really helps us with. It's the processing of unstructured data.

Four types of NoSQL databases
NoSQL databases are basically classified into four different types based on the data-storage methodology. First are the key-value store databases – they store data as large hash tables with unique keys and specific values indexed to it. Second are the column family store databases – they store data in blocks from single columns. These columns can be grouped together. They have a unique key identifier. Third are the document databases – they store data in documents composed of tagged elements. Last is the graph family of databases. Graph databases store data in graphical format using nodes and edges to represent the data and the format of a network. Key-value databases use a schema less or no-schema format for storing data. Keys can be synthetic or automatically generated. Values can be any format like strings such as JSON or BLOB. These databases use the hash-table concept where there is a unique key and pointer, which points at the item data in the table. A collection of keys is called a bucket. There may be identical keys in different buckets. Performance is increased due to the cache of the mapping of the data with the pointers and the keys. Since it only involves the keys, the values, and the buckets, scalability in terms of horizontal and vertical dimensions is comparatively easy. Since there is no schema-defined structure, querying of a particular value or updating a specific value will not be as efficient as if there are multiple simultaneous transactions. If the volume of data increases, you may need to generate complex-character strings to maintain the uniqueness of the keys among the extremely large dataset.
The four main types of NoSQL database are key-value stores, column family stores, document databases, and graph databases. Key-value stores implement hash table methodology. They have unique keys and values with index pointers. They are simple and easy to implement. The pros of NoSQL databases are performance, scalability, and flexibility. The con of NoSQL databases is it has no schema. The NoSQL database works best with huge data sets that are less complex.

In the column family of data stores, data is stored and accessed through column cells instead of rows. Columns are clustered in column families. There are no limitations on the number of columns in a column family. They can be created either by runtime or compile time while defining the schema. Unlike in RDBMS, where different rows are stored in different locations on the disk, performance is increased as the columns or the cells related to a specific column family are stored in continuous disk entries. This enables faster searches and access of the data. Portions of the column model are – the data model of a column family data store; the column, which is a list of data with the name and the value; the key, which is a unique identifier comprising of a set of columns; the column family, which is a group of columns; and the super family, which is a group of column families.
The column family store stores data using unique keys and multiple columns. The main concept of the column family store is creating columns using column families and super columns. The column family store works best with complex data sets. The pros of the column family store are scalability and performance. The con of the column family store is its flexibility.

In the document family of data stores, data is stored in the form of documents, which in itself is a compressed form of key-value pairs. This is more similar to key-value pair databases. The important difference is the stored data in documents comprise of complex structures to manage data in the form of XML, JSON, BSON, et cetera. Performance improves since the document database stores all of the metadata of the stored data and attributes. This enables the content-driven querying of data across different documents. This feature is not available in key-value databases. Scalability is also easy since it does not have a defined schema. It is easy to add fields to JSON and BSON without having to actually define them. One of the cons is querying, which tends to be very, very, very slow. The graph family of databases uses graphical representation with nodes and edges to address the flexibility and scalability concerns. Graph properties are used for index-free adjacency to represent the relationship between the nodes via edges. Nodes are defined and labeled with certain properties. Edges are directional and labeled. They represent a relation. Graph databases are best with data with complex but definitive relations. They are flexible. And they scale very, very well.
In document databases, each document can be identified with a unique key. Each document, also known as a JSON document, acts as a value. Document databases are also called semi-structured data. They have interrelated values associated with keys. Document databases work best with file and document-related complex data. The pros of document databases are flexibility, scalability, and performance. The con of document database is querying. Graph theory is implemented in a flexible graph model. Data can be represented with N - number of interrelated relations - as edges between nodes or documents. There are no columns or rows in the graph theory. The pros of the graph theory are flexibility and scalability. The cons of the graph theory are clustering and performance.

HDFS overview:

One of the core pieces of Hadoop is the Hadoop Distributed File System, or the HDFS. The HDFS is architected to hold very large files and data, and offer very fast transaction rates when interacting with the data. File storage in the HDFS is redundant. Each file segment is stored in at least three places. This redundancy leads to high availability of the data. The HFDS is designed to operate even if a data node goes down or becomes, otherwise, unreachable. In this lesson, you will understand the basics of the Hadoop Distributed File System. By the end of this lesson, you should understand the basics of the Hadoop Distributed File System and how it is used. You should also know how the HDFS is architected and structured, and know how the HDFS compares to other file system models. Like most other file systems, the HDFS breaks its files down into blocks. All files are split up into blocks of a fixed size. These blocks are huge – about 64 megabytes. Each of these blocks is stored in multiple places across the Hadoop cluster. Each computer on a Hadoop cluster that stores data is called a data node. A file, of course, is made up of many blocks, and each block is stored on multiple data nodes. This distribution of blocks means that there are multiple ways to reconstruct a file. Since the HDFS has such large block sizes, it is a good fit for Big Data applications that need a massive amount of storage capacity across multiple machines. More machines mean more data, more storage capacity, and more data redundancy.
One of the core pieces of Hadoop is the Hadoop Distributed File System. File storage in HDFS is redundant. Each file segment is stored in at least three places. HDFS is designed to operate even if data nodes go down.

The Google File System – the GFS – was the inspirational design for the HDFS. The Google File System incorporates the concept of having multiple computers serving a single file. But the GFS did not take an account for the loss of one of these computers. If in the case of a failure, a file on the GFS would be useless. The HDFS improves on the GFS design by introducing data replication. The same block for a file may be distributed across multiple data nodes. The default is 3. The HDFS is also scalable. Need more storage capacity? Just add more data nodes. The HDFS is architected for Big Data applications. It has few other obvious uses. Many performance and usability tradeoffs have been made to this effect. The HDFS does not pretend to be everything to everybody. The HDFS contrasts in design to the Network File System, or the NFS. The NFS is the one of the most popular file systems. It has been around for a long time. The NFS is pretty easy to understand, but it also has some limitations that do not lend itself to Big Data applications. In the NFS, remote access to a single logical volume is provided via accessing storage on a single machine. Remote clients can access files as though they were local. This works well for most applications, but not for Big Data applications where the HDFS is a much better fit. All of this data needs to be managed. In the HDFS, data management occurs on a single computer called the name node. The name node only needs to know a few things about the files it controls on the data nodes such as the name of the file, permissions, and the locations of file blocks. If an application needs to open a file, a request is made to the name node. The name node retrieves the file – perhaps in parallel, assembles it, and returns it to the client. Client applications only deal with the name node.
The Google File System was the inspiration for the design of the HDFS. HDFS improves on the GFS design by introducing data replication. HDFS is scalable. HDFS contrasts in design to the Network File System. The Network File System is one of the most popular file systems. The Network File System is pretty easy to understand and works well with most applications. HDFS is a better fit for Big Data applications. The name node manages all the data. The name node only needs to know a few things about the files. The name node controls the data node. If an application needs to open a file, a request is made to the name node.

Basics of HDFS:

In this lesson, you will learn the HDFS and learn basic HDFS navigation operations. You'll learn how to navigate the command line and look inside the HDFS. The first step is to start the HDFS with the following command – user@namenode:hadoop$ bin/start-dfs.sh. There is a lot happening in the above script. First it will start the name node machine. Most likely, this is the machine that you are logged into. Then all the data node instances and each slave node will be started. If this installation is a single machine cluster, the data nodes are virtual and exist on the physical machine – the same physical machine as the name node. On a real Hadoop cluster, this startup script will also SSH into each of the data nodes and start them. Let's start with a few commands – someone@anynode:hadoop$ bin/hadoop dfs -ls. And look to see, if there's any contents in the HDFS, if this is a new installation, there should be nothing there. The -ls command should look familiar to many of you. If you issue it here, it returns nothing as no files are yet present in your Hadoop cluster. The command attempted to list the contents of your home directory in the HDFS. Remember, that your home directory in the HDFS is completely separate from your home directory on the host computer. The Hadoop Distributed File System operates in its known namespace, separate from the local file system. Also note that the HDFS has no concept of a current working directory, so the cd commands don't even exist here. In this example, if you ran the -ls command without an argument, there may be a few initial directory contents. These are system objects created in the installation process. Each of these objects are created by a username. The name Hadoop is the system name in which Hadoop daemons were started – data node, name node, et cetera. The username supergroup is a group in which the members include the username that started the HDFS instance.
The command to start the data node is user@namenode: hadoop$ bin/start-dfs.sh. The command to find an installation is someone@anynode: hadoop$ bin/Hadoop dfs -ls. The name "Hadoop" is the system name in which Hadoop daemons were started. The username "supergroup" is a group in which the members include the username that started the HDFS instance.

If it does not exist, create your home directory –someone@anynode:hadoop$ bin/hadoop dfs -mkdir /user. In the above, the directory user may already exist. If it does, you don't need to issue the command to create. The next step is to create your home directory – someone@anynode:hadoop$ bin/hadoop dfs -mkdir /user/person. Of course, replace the user/person with your username or if you like, you can repeat this process with other user IDs. Of course, there are more commands in the bin/hadoop dfs. We only covered a few of them, just the basics to get you started. If you want to see all the commands that could be executed in FS Shell, just execute bin/hadoop dfs with no parameters. Also, if you forget what a command does, just type the following – bin/hadoop dfs -help commandName. The system will display a short form explanation on what that command does. The commands we used so far are in the DFS module. This module has all common commands for file and directory manipulation. They will also work for all objects in the file system. For administrative control, the dfsadmin module commands are used to query and manipulate the whole system. For system status, a status report for the Hadoop Distributed File System can be obtained by the report command. This system report will retrieve information about the running status of the HDFS cluster as well as other diagnostic metrics. To shut down the Hadoop Distributed File System if not logged in already, log in to the name node and issue the stop-dfs command.
The steps to create a home directory are as follows:
1. someone@anynode: hadoop$ bin/hadoop dfs –mkdir /user
2. someone@anynode: hadoop$ bin/Hadoop dfs –mkdir /user/person
You will execute the "bin/Hadoop dfs" command with no parameters to see all commands. If you forget what a command does, type bin/Hadoop dfs –help commandName.

HDFS file operations:

In this topic, you will learn how to perform file operations within the HDFS. You will understand how to add files, how to list files, and see their properties. We will start by inserting data into the cluster. There are a few differences you might notice. In Linux and UNIX, individual user's files are in the home\user folder. In the Hadoop, HDFS files are in the user\user folder. Also, if a command line command, like ls, requires a directory name and one is not supplied, the HDFS assumes the default directory name. Some commands require you to explicitly specify the full source and destination paths. If you use any relative paths as arguments to the Hadoop Distributed File System or MapReduce, the paths are considered to be relative to the base directory. You will find that moving data around in the HDFS is fairly easy and intuitive. If you have experience using FTP or even DOS, the HDFS will come easy to you. For example, to upload or really actually to insert a new file into the HDFS, you can use the put command. And to list files, you can use the ls command. By using these two commands, you can easily add files into HDFS and list them and list all these associated file properties.
Data management within HDFS is simple and is somewhat similar to DOS of FTP. You use the put command to upload files. The put command inserts files into HDFS. The ls command lists files in HDFS. The put command can also be used to move more than single files. Entire folders can be uploaded with the put command.

The put command can also be used to move more than one single file. It can be used to upload multiple files or even entire folders or directory trees into the HDFS. To do this, use the put command. The listing displays that the file tree was uploaded correctly. Notice that in addition to the file path, the numbers or replicas of the file are listed. This is denoted by the "R" and the number that comes after it. Thus R1 means that there is one replica of the file or R3 meaning that there are three replicas of the file. Also present are the file size, upload time, owner, and the file permission information. In place of the put command, it is copyFromLocal. The functionality of put and copyFromLocal are identical. The put command on a file is either fully successful or fully failed. When the command uploads a file into the HDFS, it first copies the contents of the file onto the data nodes. When the data nodes have indicated that the file content is transferred, the put command completes and the file is made available to the system. There will never be a state where the file is partially uploaded. When the file is available, the data nodes will indicate to this system that the file is ready.
The put command on a file is either fully successful or fully failed. The file is first copied to data nodes and the data nodes monitor the transfer. Files are never partially uploaded. Data nodes indicate when the transfer is successful.

There are a few ways to retrieve data from the Hadoop Distributed File System. An easy way is to use stdout with the cat command to display the contents of a file. The cat command can also be used to pipe data to other destinations or applications. If you have not done so already, upload a few files into the HDFS. This example assumes that you have a file named "foo" in your home directory in the HDFS. Display data with cat. Next use get to copy a file from Hadoop Distributed File System to the local file system. The get command is the opposite of put. The get command will copy a file or directory structure from the HDFS into a folder in a local file system. Another way to do the same thing is by using the copyToLocal command. Like the put command, get will operate on directories in addition to individual files.
Data can also be downloaded from HDFS using the stdout command. The cat command is used to pipe data.

Basic principles of MapReduce:

MapReduce is an algorithm used to divide up Big Data jobs. The MapReduce algorithm has been around for a long time. Although Hadoop did not invent MapReduce, one of the core components in Hadoop is the MapReduce engine and how it distributes work to nodes in a cluster. In this topic, you will learn the basic principles of MapReduce and general mapping issues. You will also be able to describe the two steps in MapReduce using mappers and reducers. In short, you will learn how MapReduce is part of the Hadoop framework and how you use it to solve Big Data applications. In this example, we are going to go over the anatomy of a MapReduce job and some programming considerations and also a bit of history about MapReduce and why it's so important. Now what you see here is you see Eclipse opened up with a MapReduce job within it. Now Apache MapReduce is Java based. So you are going to have to know Java. If you don't know Java, you're going to have to learn Java to work with MapReduce. If you don't know Java, there are some tools that we could use down the road to help, such as Pig and Hive which are nonJava based tools, for writing MapReduce jobs. But here let's look at some project settings. I'm going to go to Project - Properties and show you the two .jar files that you need to have in your path for this to work. So I have the hadoop-common-2.3.0.jar and I have the hadoop-mapreduce-client-core-2.3.0.jar. These are the only two that you need to get started.
The Eclipse SDK window is open and shows a MapReduce example. The Eclipse SDK window has the File, Edit, Source, Refractor, Navigate, Search, Project, Run, Window, and Help menus. Below the menus, the window is divided into three areas. The area to the left shows the Package Explorer directory structure. The area in the middle shows the Dictionary.java tabbed page. The area to the right shows the Outline section. The presenter clicks the Project menu to open the Project shortcut menu options and selects the Properties shortcut menu option to display the "Properties for MapReduce Example" dialog box. The dialog box shows the necessary jar files that are needed to run the MapReduce example.

Now as far as MapReduce goes, Hadoop did not invent MapReduce. MapReduce is a programming algorithm, a distributed programming algorithm, that's been around for at least ten years or so. And let's talk a little bit about what it does. Up until very recently being the year 2000 or so, there is always an exponential increase in processing speed and power of CPUs, whereas from the 80s to the 2000s, there seemed to be always an exponential growth in how fast computers got. So the way we process information really didn't change that much because we would always assume that we would get an exponential leap in processing power with the new batch of processors that came out. Now right around the year 2000 or so, there seemed to be some stagnation in how fast these chips can get. So people started looking at different ways that we could process data. So rather than assuming that CPUs are going to get faster and faster and faster, we started looking at the way we process data. In simple words, maybe we should look more at distributing the process and the computation rather than having a CPU do it. And that's really the genesis of MapReduce. MapReduce has been around since approximately 2004 and as the saying goes – "Hadoop did not invent MapReduce, but it did make it famous."
The presenter discusses various classes in the Dictionary.java tabbed page.

So what MapReduce does is MapReduce...here Apache MapReduce...what it does is, it takes the MapReduce algorithm and distributes it across many different nodes in the cluster. Now when you just turn out, you might just be using a single node cluster. So you might ask yourself the real utility of this, you know, if it's only running on a single node. And you are right that we are not actually distributing this across multiple nodes as you are learning it, but it does at least illustrate the basics in how the whole thing works. So let's go over the code. Again, it's a simple program and there is really not a lot to it. We have our imports and as you can see we have some standard Java packages that we are importing. So we are going to need the java.io.IOException package and the java.util.StringTokenizer package. And these imports here are from those two .jar files that I showed you previously. We are getting our MapReduce classes and we are getting some of our core Hadoop classes. Now as far as the code goes, MapReduce programs are Java applications. So being a Java application, we are going to have a main method.
The presenter discusses the main method in the Dictionary.java tabbed page.

So I'm going to navigate down to the main method and as you can see there are only really a few lines of code. When we're writing MapReduce programs, most of the functionality that is implied – or most of the functionality that is executed – is hidden from us. Sort of like when we are doing anything with XML where an XML parcer might actually hide most of its implementations. MapReduce works in the same way. The only thing that we are really concerned with or the two main things that we're concerned with are the mappers and the reducers. Now here what we are doing is in our main method – again our main method is our method where the job actually is search running is we do a couple of things, really just a couple of things where we set some job parameters where we put here some information that has to do with the system configuration. And then we are setting some job parameters and these lines of code here. And then we are using the FileInputFormat to determine exactly the format of the data that is going into our mapper. And where the opposite is also true where we are setting a FileOutputFormat from our reducer.
The presenter discusses the WordMapper class in the Dictionary.java tabbed page. The WordMapper class reads as follows:
public static class WordMapper extends Mapper <Text, Text, Text, Text>

So as you can see the code and actually getting this to run, as far as we're concerned, is not that large or not that vast. Now going up to the top of the program, we really have two static classes. Where we have our static class that does the mapping as you can see highlighted here. And anytime that we define a mapper, what we want to do is we want to extend the mapper class and you could see we have our code for reading our key value pairs here. And then we have the code for our reducer.
The presenter discusses the Reducer class in the Dictionary.java tabbed page. The Reducer class reads as follows:
public static class AllTranslationsReducer extends Reduce <Text, Text, Text, Text>

And when we build a reducer we always extend the reducer class. That's pretty much it as far as what is exposed to the developer, where we have our main method, we have our map section, and a reduce section. Now again, if you are running this on a single node cluster, this is still a very good illustration on how MapReduce works. Now deploying this and actually building this MapReduce program and jarring it up and installing it is a much more complicated task and it's a lot more difficult than just to run this as a regular Java application. So actually jarring this up and deploying it is a whole different task than actually programming a MapReduce job.

2012-11-23T19:47:00.005+05:30

Encrypting your partitions on RHEL after installation

Release:

RHEL 5

RHEL 6

Problem:
Encrypting your partitions on RHEL after installation

Solution:

LUKS:

With modern versions of cryptsetup (i.e., since ~2006), encrypted block devices can be created in two main formats, plain dm-crypt format or the extended LUKS (Linux Unified Key Setup-on-disk-format) format. LUKS provides a standard on-disk-format for hard disk encryption, which facilitates compatibility among Linux distributions and provides secure management of multiple user passwords. In contrast to previous Linux disk-encryption solutions, LUKS stores all necessary setup information in the partition header, enabling the user to more easily transport or migrate their data.

What LUKS does?

LUKS encrypts entire block devices and is therefore well-suited for protecting the contents of mobile devices such as removable storage media or laptop disk drives.

The underlying contents of the encrypted block device are arbitrary. This makes it useful for encrypting swap devices. This can also be useful with certain databases that use specially formatted block devices for data storage.

ü LUKS uses the existing device mapper kernel subsystem.

ü LUKS provides passphrase strengthening which protects against dictionary attacks.

ü LUKS devices contain multiple key slots, allowing users to add backup keys/passphrases.

What LUKS does not do?

LUKS is not well-suited for applications requiring many (more than eight) users to have distinct access keys to the same device.
LUKS is not suited for applications requiring file-level encryption.

In Red Hat Enterprise Linux, userspace interaction with dm-crypt is managed by a tool called cryptsetup, which uses the device-mapper infrastructure to setup and operate on encrypted block devices.

dm-crypt and cryptsetup:

Device-mapper is a part of the Linux kernel that provides a generic way to create virtual layers of block devices, most commonly LVM logical volumes. The device-mapper crypt target (dm-crypt) provides transparent encryption of block devices using the kernel crypto API (supporting ciphers and digest algorithms via standard kernel modules).

Packages required for LUKS in Red Hat Enterprise Linux 5 & 6 :

The main package is called cryptsetup-luks and it depends on cryptsetup-luks-libs. Note that both of these packages should be part of any installation and therefore available without extra effort. The below prerequisites can be included as per your requirement.

Create the block devices if required:

Use any of the below method to create the block device or partition:

fdisk/parted

pvcreate

lvcreate

mdadm

Initializing the device:

This can be done by filling random data on to your device.It is required for strengthening your encryption. You may use any of the below method to write the random data.

dd if=/dev/urandom of=<device>

The above method is quite lengthy but provides you with high quality random data. If required a faster method you may use the one below.

badblocks -c 10240 -s -w -t random -v <device>

Step by Step demo on dm-crypt:

Create you new partition using fdsik:

[root@lvs1 ~]# fdisk /dev/sdb

Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel

Building a new DOS disklabel with disk identifier 0x0ff9997a.

Changes will remain in memory only, until you decide to write them.

After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to

switch off the mode (command 'c') and change display units to

sectors (command 'u').

Command (m for help): p

Disk /dev/sdb: 21.5 GB, 21474836480 bytes

255 heads, 63 sectors/track, 2610 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x0ff9997a

Device Boot Start End Blocks Id System

Command (m for help): n

Command action

e extended

p primary partition (1-4)

Partition number (1-4): 1

First cylinder (1-2610, default 1):

Using default value 1

Last cylinder, +cylinders or +size{K,M,G} (1-2610, default 2610): +100M

Command (m for help): p

Disk /dev/sdb: 21.5 GB, 21474836480 bytes

255 heads, 63 sectors/track, 2610 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x0ff9997a

Device Boot Start End Blocks Id System

/dev/sdb1 1 14 112423+ 83 Linux

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks

Fill the data using random data:

[root@lvs1 ~]# dd if=/dev/urandom of=/dev/sdb1

dd: writing to `/dev/sdb1': No space left on device

224848+0 records in

224847+0 records out

115121664 bytes (115 MB) copied, 23.083 s, 5.0 MB/s

Partition initialization:

[root@lvs1 ~]# cryptsetup --verbose --verify-passphrase luksFormat /dev/sdb1

WARNING!

========

This will overwrite data on /dev/sdb1 irrevocably.

Are you sure? (Type uppercase yes): YES

Enter LUKS passphrase:

Verify passphrase:

Command successful.

Check the status of the Encryption:

If Encrypted Device:

[root@lvs1 ~]# cryptsetup isLuks /dev/sdb1 && echo Success

Success

If on non-encrypted Device:

[root@lvs1 ~]# cryptsetup isLuks /dev/sda1 && echo Success

Device /dev/sda1 is not a valid LUKS device.

Now open your new encrypted device:

For getting the summary on the encryption performed use the below command:

[root@lvs1 ~]# cryptsetup luksDump /dev/sdb1

LUKS header information for /dev/sdb1

Version: 1

Cipher name: aes

Cipher mode: cbc-essiv:sha256

Hash spec: sha1

Payload offset: 4096

MK bits: 256

MK digest: eb 60 b3 82 4a 41 74 c1 e0 0e e3 54 4d 4b 64 8e ee 31 a8 18

MK salt: 1b 3f a8 04 e1 76 78 de c2 da a8 10 86 5e 1e 80

f1 c2 08 6f 92 99 a3 64 60 ba bb fa 71 3f 6a 19

MK iterations: 44000

UUID: c09e9489-8c34-4694-947d-d2b24147fcac

Key Slot 0: ENABLED

Iterations: 176424

Salt: b3 35 b4 e9 d1 8e 63 f7 10 87 14 cb c7 66 7d b9

b9 b2 52 63 79 84 d4 0b 73 34 64 ef 7e 25 df 37

Key material offset: 8

AF stripes: 4000

Key Slot 1: DISABLED

Key Slot 2: DISABLED

Key Slot 3: DISABLED

Key Slot 4: DISABLED

Key Slot 5: DISABLED

Key Slot 6: DISABLED

Key Slot 7: DISABLED

Create a Mapping for device access:

To access your device's decrypted content you need to use the device mapper(dm). Inorder to create this ie, /dev/mapper/<name> to represent this device use the below command:

[root@lvs1 ~]# cryptsetup luksOpen /dev/sdb1 secure

Enter passphrase for /dev/sdb1:

It will ask for the password phrase which you have already given in previous step. Input the same. Check the device file creation under the /dev/mapper

[root@lvs1 ~]# ls -l /dev/mapper | grep secure

lrwxrwxrwx. 1 root root 7 Sep 19 06:43 secure -> ../dm-2

If need to get more information use the below command:

[root@lvs1 ~]# dmsetup info /dev/mapper/secure

Name: secure

State: ACTIVE

Read Ahead: 256

Tables present: LIVE

Open count: 0

Event number: 0

Major, minor: 253, 2

Number of targets: 1

UUID: CRYPT-LUKS1-c09e94898c344694947dd2b24147fcac-secure

Create filesystem:

For creating the filesystem use the below command:

[root@lvs1 ~]# mkfs.ext3 /dev/mapper/secure

mke2fs 1.41.12 (17-May-2010)

Filesystem label=

OS type: Linux

Block size=1024 (log=0)

Fragment size=1024 (log=0)

Stride=0 blocks, Stripe width=0 blocks

27664 inodes, 110372 blocks

5518 blocks (5.00%) reserved for the super user

First data block=1

Maximum filesystem blocks=67371008

14 block groups

8192 blocks per group, 8192 fragments per group

1976 inodes per group

Superblock backups stored on blocks:

8193, 24577, 40961, 57345, 73729

Writing inode tables: done

Creating journal (4096 blocks): done

Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 37 mounts or

180 days, whichever comes first. Use tune2fs -c or -i to override.

Editing the /etc/crypttab:

The /etc/crypptab file describes encrypted block devices that are set up during system boot.

Empty lines and lines starting with the # character are ignored. Each of the remaining lines describes one encrypted block device, fields on the line are delimited by white space. The first two fields are mandatory, the remaining two are optional.

The first field contains the name of the resulting encrypted block device; the device is set up at /dev/mapper/name. The second field contains a path to the underlying block device. If the block device contains a LUKS signature, it is opened as a LUKS encrypted partition; otherwise it is assumed to be a raw dm-crypt partition.

The third field specifies the encryption password. If the field is not present or the password is set to none, the password has to be manually entered during system boot. Otherwise the field is interpreted as a path to a file containing the encryption password. For swap encryption /dev/urandom can be used as the password file; using /dev/random may prevent boot completion if the system does not have enough entropy to generate a truly random encryption key.

In order for the system to set up a mapping for the device, an entry must be present in the /etc/crypttab file. If the file doesn't exist, create it and change the owner and group to root ( root:root) and change the mode to 0744. Add a line to the file with the following format:

In our case we are editing as below:

[root@lvs1 ~]# vi /etc/crpttab

secure UUID=c09e9489-8c34-4694-947d-d2b24147fcac

Note: You may use the blkid command to get the device ID for the device. Eg: blkid /dev/sdb1

Manually mounting the partition:

[root@lvs1 ~]# mkdir /secure

[root@lvs1 ~]# mount /dev/mapper/secure /secure/

[root@lvs1 ~]# mount

/dev/mapper/VolGroup-lv_root on / type ext4 (rw)

proc on /proc type proc (rw)

sysfs on /sys type sysfs (rw)

devpts on /dev/pts type devpts (rw,gid=5,mode=620)

tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")

/dev/sda1 on /boot type ext4 (rw)

/dev/mapper/secret on /secret type ext4 (rw)

none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

Giving entry into the fstab:

Append the below entry into your existing fstab:

/dev/mapper/secret /secret ext4 defaults 0 0

During booting it may give you the screen for inputting the password.

Provide the password to mount the partition at the boot time. We are done with our encryption.

***********************

Converting the File System from ext3 to ext2 without data Loss

2012-11-23T19:41:00.002+05:30

Release:
RHEL

Problem:
This document guide you on how to convert an ext3 file system to ext2 file system without data loss.

Solution:
In this document we are having a Red hat box with a partition /dev/sdb1 formatted using ext3 file system and mounted under /test directory.

[root@akhisar test]# mount

/dev/mapper/vg_akhisar-lv_root on / type ext4 (rw)

proc on /proc type proc (rw)

sysfs on /sys type sysfs (rw)

devpts on /dev/pts type devpts (rw,gid=5,mode=620)

tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")

/dev/sda1 on /boot type ext4 (rw)

none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

gvfs-fuse-daemon on /root/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev)

/dev/sr0 on /media/RHEL_6.1 x86_64 Disc 1 type iso9660 (ro,nosuid,nodev,uhelper=udisks,uid=0,gid=0,iocharset=utf8,mode=0400,dmode=0500)

/dev/sr0 on /mnt type iso9660 (ro)

/dev/sdb1 on /test type ext3 (rw)

We can see that the partition is containing data and the listed output for the /test directory is as below:

[root@akhisar test]# ls

file1 file16 file22 file29 file35 file41 file48 file54 lost+found

file10 file17 file23 file3 file36 file42 file49 file55

file11 file18 file24 file30 file37 file43 file5 file56

file12 file19 file25 file31 file38 file44 file50 file6

file13 file2 file26 file32 file39 file45 file51 file7

file14 file20 file27 file33 file4 file46 file52 file8

file15 file21 file28 file34 file40 file47 file53 file9

Now we are going to convert the file system for the partition /dev/sdb1 from ext3 to ext2.

Step 1:

Unmount the partition on which the activity needs to be carried out.

[root@akhisar /]# umount /test

[root@akhisar /]# mount

/dev/mapper/vg_akhisar-lv_root on / type ext4 (rw)

proc on /proc type proc (rw)

sysfs on /sys type sysfs (rw)

devpts on /dev/pts type devpts (rw,gid=5,mode=620)

tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")

/dev/sda1 on /boot type ext4 (rw)

none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

gvfs-fuse-daemon on /root/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev)

/dev/sr0 on /media/RHEL_6.1 x86_64 Disc 1 type iso9660 (ro,nosuid,nodev,uhelper=udisks,uid=0,gid=0,iocharset=utf8,mode=0400,dmode=0500)

/dev/sr0 on /mnt type iso9660 (ro)

Step 2:

Clear the has_journal using the tune2fs command along with the “^” string.

[root@akhisar /]# tune2fs -O ^has_journal /dev/sdb1

tune2fs 1.41.12 (17-May-2010)

Step 3:

Check the partition for errors by typing the following command as root:

[root@akhisar /]# e2fsck -y /dev/sdb1

e2fsck 1.41.12 (17-May-2010)

/dev/sdb1: clean, 67/42168 files, 6877/168648 blocks

Step 4:

Then mount the partition again as ext2 file system by typing the below command:

[root@akhisar /]# mount -t ext2 /dev/sdb1 /test/

Step 5:

You can check the same by using the mount option.

[root@akhisar /]# mount

/dev/mapper/vg_akhisar-lv_root on / type ext4 (rw)

proc on /proc type proc (rw)

sysfs on /sys type sysfs (rw)

devpts on /dev/pts type devpts (rw,gid=5,mode=620)

tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")

/dev/sda1 on /boot type ext4 (rw)

none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

gvfs-fuse-daemon on /root/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev)

/dev/sr0 on /media/RHEL_6.1 x86_64 Disc 1 type iso9660 (ro,nosuid,nodev,uhelper=udisks,uid=0,gid=0,iocharset=utf8,mode=0400,dmode=0500)

/dev/sr0 on /mnt type iso9660 (ro)

/dev/sdb1 on /test type ext2 (rw)

Step 6:

Check the data by browsing to the test folder.

[root@akhisar /]# cd test

[root@akhisar test]# ls

file1 file16 file22 file29 file35 file41 file48 file54 lost+found

file10 file17 file23 file3 file36 file42 file49 file55

file11 file18 file24 file30 file37 file43 file5 file56

file12 file19 file25 file31 file38 file44 file50 file6

file13 file2 file26 file32 file39 file45 file51 file7

file14 file20 file27 file33 file4 file46 file52 file8

file15 file21 file28 file34 file40 file47 file53 file9

**************************************

2012-11-23T19:40:00.002+05:30

Converting the File System from ext3 to ext2 without data Loss

Release:

RHEL

Problem:

This document guide you on how to convert an ext3 file system to ext2 file system without data loss.

Solution:

In this document we are having a Red hat box with a partition /dev/sdb1 formatted using ext3 file system and mounted under /test directory.

[root@akhisar test]# mount

/dev/mapper/vg_akhisar-lv_root on / type ext4 (rw)

proc on /proc type proc (rw)

sysfs on /sys type sysfs (rw)

devpts on /dev/pts type devpts (rw,gid=5,mode=620)

tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")

/dev/sda1 on /boot type ext4 (rw)

none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

gvfs-fuse-daemon on /root/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev)

/dev/sr0 on /media/RHEL_6.1 x86_64 Disc 1 type iso9660 (ro,nosuid,nodev,uhelper=udisks,uid=0,gid=0,iocharset=utf8,mode=0400,dmode=0500)

/dev/sr0 on /mnt type iso9660 (ro)

/dev/sdb1 on /test type ext3 (rw)

We can see that the partition is containing data and the listed output for the /test directory is as below:

[root@akhisar test]# ls

file1 file16 file22 file29 file35 file41 file48 file54 lost+found

file10 file17 file23 file3 file36 file42 file49 file55

file11 file18 file24 file30 file37 file43 file5 file56

file12 file19 file25 file31 file38 file44 file50 file6

file13 file2 file26 file32 file39 file45 file51 file7

file14 file20 file27 file33 file4 file46 file52 file8

file15 file21 file28 file34 file40 file47 file53 file9

Now we are going to convert the file system for the partition /dev/sdb1 from ext3 to ext2.

Step 1:

Unmount the partition on which the activity needs to be carried out.

[root@akhisar /]# umount /test

[root@akhisar /]# mount

/dev/mapper/vg_akhisar-lv_root on / type ext4 (rw)

proc on /proc type proc (rw)

sysfs on /sys type sysfs (rw)

devpts on /dev/pts type devpts (rw,gid=5,mode=620)

tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")

/dev/sda1 on /boot type ext4 (rw)

none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

gvfs-fuse-daemon on /root/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev)

/dev/sr0 on /media/RHEL_6.1 x86_64 Disc 1 type iso9660 (ro,nosuid,nodev,uhelper=udisks,uid=0,gid=0,iocharset=utf8,mode=0400,dmode=0500)

/dev/sr0 on /mnt type iso9660 (ro)

Step 2:

Clear the has_journal using the tune2fs command along with the “^” string.

[root@akhisar /]# tune2fs -O ^has_journal /dev/sdb1

tune2fs 1.41.12 (17-May-2010)

Step 3:

Check the partition for errors by typing the following command as root:

[root@akhisar /]# e2fsck -y /dev/sdb1

e2fsck 1.41.12 (17-May-2010)

/dev/sdb1: clean, 67/42168 files, 6877/168648 blocks

Step 4:

Then mount the partition again as ext2 file system by typing the below command:

[root@akhisar /]# mount -t ext2 /dev/sdb1 /test/

Step 5:

You can check the same by using the mount option.

[root@akhisar /]# mount

/dev/mapper/vg_akhisar-lv_root on / type ext4 (rw)

proc on /proc type proc (rw)

sysfs on /sys type sysfs (rw)

devpts on /dev/pts type devpts (rw,gid=5,mode=620)

tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")

/dev/sda1 on /boot type ext4 (rw)

none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

gvfs-fuse-daemon on /root/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev)

/dev/sr0 on /media/RHEL_6.1 x86_64 Disc 1 type iso9660 (ro,nosuid,nodev,uhelper=udisks,uid=0,gid=0,iocharset=utf8,mode=0400,dmode=0500)

/dev/sr0 on /mnt type iso9660 (ro)

/dev/sdb1 on /test type ext2 (rw)

Step 6:

Check the data by browsing to the test folder.

[root@akhisar /]# cd test

[root@akhisar test]# ls

file1 file16 file22 file29 file35 file41 file48 file54 lost+found

file10 file17 file23 file3 file36 file42 file49 file55

file11 file18 file24 file30 file37 file43 file5 file56

file12 file19 file25 file31 file38 file44 file50 file6

file13 file2 file26 file32 file39 file45 file51 file7

file14 file20 file27 file33 file4 file46 file52 file8

file15 file21 file28 file34 file40 file47 file53 file9

**************************************

Identifying the Linux file system Type

2012-11-23T19:40:00.000+05:30

Release:
RHEL

Problem:

This document guides you on how to identify the Linux file system type of a partition.

Solution:

Method 1:

Using the df command you can know the type of the file system using the “T” switch.

[root@akhisar /]# df -T

Filesystem Type 1K-blocks Used Available Use% Mounted on

/dev/mapper/vg_akhisar-lv_root

ext4 6362016 3340220 2698624 56% /

tmpfs tmpfs 345956 260 345696 1% /dev/shm

/dev/sda1 ext4 495844 31884 438360 7% /boot

/dev/sr0 iso9660 3516418 3516418 0 100% /media/RHEL_6.1 x86_64 Disc 1

/dev/sr0 iso9660 3516418 3516418 0 100% /mnt

Limitation: Only mounted partitions will be listed by this command.

Method 2:

You can use the mount command as root to know the file system type of your partitions:

[root@akhisar /]# mount | grep "^/dev"

/dev/mapper/vg_akhisar-lv_root on / type ext4 (rw)

/dev/sda1 on /boot type ext4 (rw)

/dev/sr0 on /media/RHEL_6.1 x86_64 Disc 1 type iso9660 (ro,nosuid,nodev,uhelper=udisks,uid=0,gid=0,iocharset=utf8,mode=0400,dmode=0500)

/dev/sr0 on /mnt type iso9660 (ro)

Limitation: Only mounted partitions will be listed by this command.

Method 3:

You can use the file command as root by inputting the individual device name. Illustration of the same is given below. You may get the device name for the partitions from the “fdisk –l” command output.

[root@akhisar /]# file -sL /dev/sdb2

/dev/sdb2: Linux rev 1.0 ext3 filesystem data

Method 4:

You can check for the entry in /etc/fstab file. The third field refers to the file system.

[root@akhisar /]# cat /etc/fstab

# /etc/fstab

# Created by anaconda on Mon Aug 13 17:02:54 2012

# Accessible filesystems, by reference, are maintained under '/dev/disk'

# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info

/dev/mapper/vg_akhisar-lv_root / ext4 defaults 1 1

UUID=eb572a9e-49dc-451a-94e1-6d1de54af5d6 /boot ext4 defaults 1 2

/dev/mapper/vg_akhisar-lv_swap swap swap defaults 0 0

tmpfs /dev/shm tmpfs defaults 0 0

devpts /dev/pts devpts gid=5,mode=620 0 0

sysfs /sys sysfs defaults 0 0

proc /proc proc defaults 0 0

Limitations: It may contain only the default entries created during the installation or the one provided by the administrator. It can only be used as a quick reference on the partitions.

Method 5:

You can execute the fsck command with the “-N” switch followed by the individual partitions to know the partition file system type.

[root@akhisar /]# fsck -N /dev/sdb2

fsck from util-linux-ng 2.17.2

[/sbin/fsck.ext3 (1) -- /dev/sdb2] fsck.ext3 /dev/sdb2

[root@akhisar /]# fsck -N /dev/sdb1

fsck from util-linux-ng 2.17.2

[/sbin/fsck.ext2 (1) -- /dev/sdb1] fsck.ext2 /dev/sdb1

******************************

Using UUIDs as label to identify your filesystems

2012-11-23T19:37:00.000+05:30

Release:

Red Hat Enterprise Linux

Problem:

Using UUIDs as label to identify your filesystems.

Solution:

What is an UUID?

A universally unique identifier (UUID) is an identifier standard used in software construction, standardized by the Open Software Foundation (OSF) as part of the Distributed Computing Environment (DCE). A UUID is a 16-octet (128-bit) number. In its canonical form, a UUID is represented by 32 hexadecimal digits, displayed in five groups separated by hyphens, in the form 8-4-4-4-12 for a total of 36 characters (32 digits and four hyphens). For example:

550e8400-e29b-41d4-a716-446655440000

There are 340,282,366,920,938,463,463,374,607,431,768,211,456 possible UUIDs (16 to the 32nd power or 2 to the 128th power), or ~3.4 × 1038.

UUID in Linux

In Linux, the ext2/ext3/ext4 filesystem also uses UUID to identify partitions. You are not at the safer side if you are using device node names like /dev/sda1 to refer your file system in your /etc/fstab. This is due to the fact that these device node names can change due to switching of cable or upgrading certain packages. So instead of the device node names you may consider using the UUID or LABELs in non-LVM systems.

Determine what is your current entry of the /etc/fstab by using the below command.

# grep /dev/[hsv]d /etc/fstab

/dev/sda1 /boot ext4 defaults 1 2

/dev/sda3 /data ext4 defaults 0 2

Replace the device entries which point to the device node names like /dev/sda with the UUID or the Label. In this session we will be considering the UUID. You may use the blkid command for getting the UUID of the concerned devices.

# blkid | grep sd

/dev/sda1: LABEL="/boot" UUID="c9fdb384-19ed-4b94-b29e-23f0f566e970" TYPE="ext4"

/dev/sda3: UUID="80a27dc2-c309-4cc8-9ceb-3bb1a055cf3d" TYPE="ext4"

Following this example, the above lines from /etc/fstab will look like this after being fixed:

LABEL="/boot" /boot ext4 defaults 1 2

UUID="80a27dc2-c309-4cc8-9ceb-3bb1a055cf3d" /data ext4 defaults 0 2

Check the /etc/grub.conf for the use of device node names for non-LVM device nodes to identify the root file system. If present, change the entry by putting in either the label or the UUID.

Non-recommended Entry:

# grep kernel /etc/grub.conf

kernel /vmlinuz-2.6.18-308.el5 ro root=/dev/sda2

Recommended Entry:

kernel /vmlinuz-2.6.18-308.el5 ro root=LABEL=/

kernel /vmlinuz-2.6.18-308.el5 ro root=UUID=f52529cb-a959-4a11-8d43-0e4fd8fdecd2

******************

Protection of critical files from unauthorized or unknown delete

2012-11-23T19:36:00.004+05:30

Release:

Linux Flavors with ext2 based file system

Problem:

Protecting critical files from unauthorized or unknown delete

Solution:

You can use the Chattr command for protecting your critical files from unauthorized or unknown deletion. Chattr is the acronym for Change file attributes. It is similar to the chmod command in Linux but having a different invocation and syntax. It only works with ext2/ext3/ext4 filesystems.

Once you have given the chattr +i <filename>, the particular file is made as immutable. Then file cannot be modified, linked or deleted even by the root user. One cannot see the overridden rule by using the normal permission check commands. Once set the attribute, it can be removed only by the root user. Demo on it is shown below.

At first we will create a file with the filename as mysafefile which is to be made safe from modification or deletion.

[root@testing Desktop]# touch mysafefile

Now let us view the default permission for the file using the below command.

[root@testing Desktop]# ll mysafefile

-rw-r--r-- 1 root root 0 Aug 27 12:37 mysafefile

So the root is having read write permission, the group root and others are having the read-only permission.

Now let us protect our file from unknown modification or deletion by adding the immutable option using the string “+i”.

[root@testing Desktop]# chattr +i mysafefile

So we will now check whether there was any change in the permission for the created file.

[root@testing Desktop]# ll mysafefile

-rw-r--r-- 1 root root 0 Aug 27 12:37 mysafefile

The permission has no visible change using normal permission check. It remains the same. We will now try editing the file which we have protected using the chattr command.

#vi mysafefile

Once you have opened the file you can see at the bottom left of the file that the file is read-only. Hence you will be unable to edit the file. We will now try now what if we try to delete this file.

[root@testing Desktop]# rm -f mysafefile

rm: cannot remove `mysafefile': Operation not permitted

So the root user is also not able to delete the file. Now let us check on how we can view the overridden permission. You can use the lsattr <filename> command for viewing the attributes currently set for the file.

[root@testing Desktop]# lsattr mysafefile

----i-------- mysafefile

You can now see the “i” attribute set for the file which prevented it from the deletion. Inorder to clear the attribute you can give the below command.

[root@testing Desktop]# chattr -i mysafefile

Confirm the clear using the lsattr command.

[root@testing Desktop]# lsattr mysafefile

------------- mysafefile

Running a physical Inventory in a VERITAS Netbackup

2012-11-23T19:33:00.004+05:30

Release:

Symantec VERITAS Netbackup 6.5 for Windows

Problem:

This document guides you through the scenarios where you can use a physical Inventory and how it can accomplish your requirement.

Solution:

Let us considering the following situations:

Ø For the robots without bar code readers

Ø For the robots that contain media without bar codes

Generally the Netbackup administrative console will report the presence of the media in the above cases whereas more detailed information is required to perform automated media management. In this scenario we need to go for vmphyinv or what we call as a physical inventory utility.

Features of Vmphyinv

Ø Can be run from any master server, media server, or SAN media server.

Ø Can be used with bar coded tape libraries because it verifies the contents of the media.

Ø Recognizes the NetBackup and the Backup Exec tape formats.

Ø Supports the remote administration. You do not need to run vmphyinv from the host to which the drives are attached.

Ø Tries to use multiple drives in a robot even if the drives are attached to different hosts.

Ø Works with shared drives (NetBackup Shared Storage Option).

Ø Supports all supported SCSI-based robot types.

Ø Can be used to inventory a single media in a standalone drive. Use the -u option or the -n option to specify the drive; the drive must contain media and it must be ready.

Applications Of vmphyinv

ü You may use the vmphyinv to update the EMM server in the following cases:

ü If you want to run an inventory on a robot that does not have a bar code reader or that contains nonbar coded media.

ü If you have inserted a new media into a robotic library and no NetBackup volume records correspond to the media. Use the slot range or list option of vmphyinv to perform the inventory operation. You do not need to add volume records to the EMM database.

ü You insert some media that have unknown media IDs or globally unique identifiers (GUIDs) into a robot. For example, you insert 10 media from a different tape library in slots 11 to 20. You do not know the IDs on the tapes. Use the slot range or list option of vmphyinv to perform the inventory operation. The vmphyinv utility mounts the media, reads the tape header, determines the media ID, and adds media records to the EMM database.

ü In cases where some of the media are misplaced and the EMM database does not reflect the correct physical location of these media. Inventory the robot or inventory a subset of media in the robot by using options in vmphyinv.

In all the above scenarios vmphyinv will be executing the below operations:

ü Mounts each tape

ü Reads the tape header

ü Identifies the tape in each slot

ü Updates the NetBackup volume configuration

For initiating a physical inventory you need use the below command syntax in command prompt:

C:\> C:\Program Files\Veritas\Volmgr\bin\vmphyinv -rn <robot_number> -slot_range <from to> -d <density>

Please find an example below. Here we are having 23 slots starting from 1 to 23, there is a single robot with robot number “0” and the density for the tape is hcart in the media server iffco_bkp:

C:\> C:\Program Files\Veritas\Volmgr\bin\vmphyinv -rn 0 -slot_range 1 23 –hcart

----------------------Sample Output----Media Server:iffco_bkp---------------------

Waiting for mount of media ID A00139 on host iffco_bkp

Mounted media ID A00139 on host iffco_bkp

Waiting for mount of media ID A00140 on host iffco_bkp

Mounted media ID A00140 on host iffco_bkp

Waiting for mount of media ID A00141 on host iffco_bkp

Mounted media ID A00141 on host iffco_bkp

Waiting for mount of media ID A00142 on host iffco_bkp

Mounted media ID A00142 on host iffco_bkp

Waiting for mount of media ID A00143 on host iffco_bkp

Mounted media ID A00143 on host iffco_bkp

Waiting for mount of media ID A00144 on host iffco_bkp

Mounted media ID A00144 on host iffco_bkp

Waiting for mount of media ID A00145 on host iffco_bkp

Mounted media ID A00145 on host iffco_bkp

Waiting for mount of media ID A00146 on host iffco_bkp

Mounted media ID A00146 on host iffco_bkp

Waiting for mount of media ID A00147 on host iffco_bkp

Mounted media ID A00147 on host iffco_bkp

Waiting for mount of media ID A00148 on host iffco_bkp

Mounted media ID A00148 on host iffco_bkp

----------------------Sample Output----Media Server:iffco_bkp---------------------

Waiting for mount of media ID A00161 on host iffco_bkp

Mounted media ID A00161 on host iffco_bkp

Waiting for mount of media ID A00150 on host iffco_bkp

Mounted media ID A00150 on host iffco_bkp

Waiting for mount of media ID A00151 on host iffco_bkp

Mounted media ID A00151 on host iffco_bkp

Waiting for mount of media ID A00152 on host iffco_bkp

Mounted media ID A00152 on host iffco_bkp

Waiting for mount of media ID A00153 on host iffco_bkp

Mounted media ID A00153 on host iffco_bkp

Waiting for mount of media ID A00154 on host iffco_bkp

Mounted media ID A00154 on host iffco_bkp

Waiting for mount of media ID A00155 on host iffco_bkp

Mounted media ID A00155 on host iffco_bkp

Waiting for mount of media ID A00156 on host iffco_bkp

Mounted media ID A00156 on host iffco_bkp

Waiting for mount of media ID A00157 on host iffco_bkp

Mounted media ID A00157 on host iffco_bkp

Waiting for mount of media ID A00158 on host iffco_bkp

Mounted media ID A00158 on host iffco_bkp

Waiting for mount of media ID A00159 on host iffco_bkp

Mounted media ID A00159 on host iffco_bkp

Waiting for mount of media ID A00160 on host iffco_bkp

Mounted media ID A00160 on host iffco_bkp

Waiting for mount of media ID A00162 on host iffco_bkp

Mounted media ID A00162 on host iffco_bkp

Proposed Change(s) to Update the Volume Configuration

=====================================================

Logically move media ID A00141 from slot 3 to standalone

Logically add new media A00040 to robot slot 3.

Logically move media ID A00144 from slot 6 to standalone

Logically add new media A00068 to robot slot 6.

Logically move media ID A00146 from slot 8 to standalone

Logically add new media A00041 to robot slot 8.

Logically move media ID A00147 from slot 9 to standalone

Logically add new media A00062 to robot slot 9.

Logically move media ID A00148 from slot 10 to standalone

Logically move media ID A00063 to slot 10.

Logically move media ID A00161 from slot 11 to standalone

Logically add new media A00048 to robot slot 11.

Logically move media ID A00150 from slot 12 to standalone

Logically move media ID A00057 to slot 12.

Logically move media ID A00151 from slot 13 to standalone

Logically move media ID A00050 to slot 13.

Logically move media ID A00152 from slot 14 to standalone

Logically add new media A00064 to robot slot 14.

Logically move media ID A00153 from slot 15 to standalone

Logically move media ID A00060 to slot 15.

Logically move media ID A00154 from slot 16 to standalone

Logically move media ID A00053 to slot 16.

Logically move media ID A00155 from slot 17 to standalone

Logically move media ID A00054 to slot 17.

Logically move media ID A00156 from slot 18 to standalone

Logically move media ID A00055 to slot 18.

Logically move media ID A00157 from slot 19 to standalone

Logically add new media A00056 to robot slot 19.

----------------------Sample Output----Media Server:iffco_bkp---------------------

Logically move media ID A00158 from slot 20 to standalone

Logically add new media A00043 to robot slot 20.

Logically move media ID A00159 from slot 21 to standalone

Logically add new media A00058 to robot slot 21.

Logically move media ID A00160 from slot 22 to standalone

Logically move media ID A00059 to slot 22.

Logically move media ID A00162 from slot 23 to standalone

Logically move media ID A00052 to slot 23.

Update volume configuration? (y/n) n: y

Moved media ID A00141 from slot 3 to standalone residence

Added new media A00040 to slot 3 on host iffco_bkp.

Moved media ID A00144 from slot 6 to standalone residence

Added new media A00068 to slot 6 on host iffco_bkp.

Moved media ID A00146 from slot 8 to standalone residence

Added new media A00041 to slot 8 on host iffco_bkp.

Moved media ID A00147 from slot 9 to standalone residence

Added new media A00062 to slot 9 on host iffco_bkp.

Moved media ID A00148 from slot 10 to standalone residence

Moved media ID A00063 to slot 10.

Moved media ID A00161 from slot 11 to standalone residence

Added new media A00048 to slot 11 on host iffco_bkp.

Moved media ID A00150 from slot 12 to standalone residence

Moved media ID A00057 to slot 12.

Moved media ID A00151 from slot 13 to standalone residence

Moved media ID A00050 to slot 13.

Moved media ID A00152 from slot 14 to standalone residence

Added new media A00064 to slot 14 on host iffco_bkp.

Moved media ID A00153 from slot 15 to standalone residence

Moved media ID A00060 to slot 15.

Moved media ID A00154 from slot 16 to standalone residence

Moved media ID A00053 to slot 16.

Moved media ID A00155 from slot 17 to standalone residence

Moved media ID A00054 to slot 17.

Moved media ID A00156 from slot 18 to standalone residence

Moved media ID A00055 to slot 18.

Moved media ID A00157 from slot 19 to standalone residence

Added new media A00056 to slot 19 on host iffco_bkp.

Moved media ID A00158 from slot 20 to standalone residence

Added new media A00043 to slot 20 on host iffco_bkp.

Moved media ID A00159 from slot 21 to standalone residence

Added new media A00058 to slot 21 on host iffco_bkp.

Moved media ID A00160 from slot 22 to standalone residence

Moved media ID A00059 to slot 22.

Moved media ID A00162 from slot 23 to standalone residence

Moved media ID A00052 to slot 23.

For further customization and running the vmphyinv to suit your scenario and further examples on the same you may use the below link:

http://www.symantec.com/business/support/index?page=content&id=HOWTO72754

****************

Booting your RHEL basic server mode (CLI) to Desktop Mode (Graphical)

2012-11-23T19:20:00.000+05:30

Release:

RHEL 6

Problem:

Booting your RHEL basic server mode (CLI) to Desktop Mode (Graphical).

Solution:

In this document we are concentrating on the migration of the existing basic server installation (CLI Mode) into a Desktop installation (graphical mode). In order to perform such a migration perform the below steps.

Configure Yum using the Installation Media:

Mount the RHEL 6 installation media on to your /mnt directory.

[root@lvs1 ~]# mount /dev/cdrom /mnt/

mount: block device /dev/sr0 is write-protected, mounting read-only

Create the a repo file pointing to the /mnt directory as below:

[root@lvs1 ~]# vi /etc/yum.repos.d/yum.repo

[rhel]

baseurl=file:///mnt/

enabled=1

gpgcheck=0

Check your repository by using the below command:

[root@lvs1 ~]# yum repolist

Loaded plugins: product-id, subscription-manager

Updating Red Hat repositories.

Repository 'rhel' is missing name in configuration, using id

rhel | 4.0 kB 00:00 ...

repo id repo name status

rhel rhel 3,480

repolist: 3,480

Here we are able to view our 3,480 packages from the media.

Check for the Graphical environment packages available in the media:

The following group package needs to be installed to boot your RHEL6 system into a graphical desktop environment.

ü Desktop

ü Desktop Platform

ü X Window System

You may use the below command for listing the packages from your yum repository if present:

[root@lvs1 ~]# yum grouplist "Desktop" "Desktop Platform" "X Window System"

Loaded plugins: product-id, subscription-manager

Updating Red Hat repositories.

Setting up Group Process

Repository 'rhel' is missing name in configuration, using id

Installed Groups:

Desktop Platform

Available Groups:

Desktop

X Window System

Done

Once confirmed that the required group packages are present in your yum repository, you may install the group packages that is not installed in your system. You may use the below command to start the installation of the group packages.

[root@lvs1 ~]# yum groupinstall "Desktop" "Desktop Platform" "X Window System"

Update your inittab file to boot into graphical environment:

You may use the below command to edit your /etc/inittab file to boot into the runlevel 5 by default.

[root@lvs1 ~]# sed -i 's,^id:.:initdefault:,id:5:initdefault:,' /etc/inittab

Now switch to the graphical mode by simply giving the below command.

root@lvs1 ~]# init 5

******************

Features Of Redhat Ver. 6

2012-03-24T20:22:00.001+05:30

Based on 2.6.X kernel
Supports graphical and text-based installations, ie Graphical installer is more feature-rich than text-based installer
Download available for a trial of 30days from the Redhat site.
Installable from:

* Local media: CDs, DVDs (1-DVD ISO image), USB

* Network: HTTP, NFS, FTP

* PXE - Network Installation
KVM & Virtualization support
Default FS for new installations is EXT4 (All new releases of all varient linux comes up with the EXT4)
Disk encryption, including root (/) file system
Supported Platforms:

* x86(32-bit)

* b. x86_64 (AMD64 & Intel64)

* IBM Power

* System z

System Requirements:

1GHz P4 processor - Recommending dual core or higher

1GB RAM - Recommending 4GB or better

3GB Storage for - minimum installation

5GB storage for - full installation

Compatibility support for older programs (compat* RPMS)
Note: These are libraries to ensure the operation of older programs
Note: This ensures that certified applications by redhat will continue to run
Anaconda auto-formats disk with:

'/boot'

'/' - root

'/home' (if >= 50GB of storage are available)

'swap'

Anaconda & running system use: NetworkManager to auto-configure networking

* Uses DHCP by default

* Requires minimum intervention
Anaconda (installer) still supports absolute control over variables

i.e , network settings may be specified during installation

Creating a grub password in RHEL

2012-03-17T12:51:00.001+05:30

1. Login as root user
2. Give the command grub-md5-crypt
3. Now it will ask for the password you wish to give for the grub
4. Enter the password and confirm the password given.
5. now you will get an encrypted password. Just copy the password by selecting the encrypted password
6. Now give the command vim /etc/grub.conf
7. Press I to edit the file.
8. Just before the title paste:
password --md5 <encrypted password>
9. press esc and give ':wq' for write and quit the edit menu.
10.Now reboot the system and try entering into the grub.
11.For entering into the grub it will ask for the password which can be given by just pressing p. This will unlock the grub features.

Which is the optimal strip size in RAID

2012-03-17T12:45:00.003+05:30

Stripe sizes play a very important role in the performance of RAID arrays and thus it is critical to understand the concept of striping before we delve any further into RAID discussion.

As we mentioned before, stripes are blocks of a single file that are broken into smaller pieces. The stripe size, or the size that the data is broken into, is user definable and can range from 1KB to 1024KB or more. The way it works is when data is passed to the RAID controller, it is divided by the stripe size to create 1 or more blocks. These blocks are then distributed among drives in the array, leaving different pieces on different drives.

Like we discussed before, the information can be written faster because it is as if the hard drive is writing a smaller file, although it is really only writing pieces of a large file. At the same time, reading the data is faster because the blocks of data can be read off of all the drives in the array at the same time, so reading back a large file may only require the reading of two smaller files on two different hard drives at the same time.

There is quite a bit of debate surrounding what stripe size is best. Some claim that the smaller the stripe the better, because this ensures that no matter how small the original data is it will be distributed across the drives. Others claim that larger stripes are better since the drive is not always being taxed to write information.

To understand how a RAID card reacts to different stripe sizes, let's use the most drastic cases as examples. We will assume that there are 2 drives setup in a RAID 0 stripe array that has one of two stripe sizes: a 2KB stripe and a 1024KB stripe. To demonstrate how the stripe sizes influence the reading and writing of data, we will use also use two different data sizes to be written and read: a 4KB file and a 8192KB file.

On the first RAID 0 array with a 2KB stripe size, the array is happy to receive the 4KB file. When the RAID controller receives this data, it is divided into two 2KB blocks. Next, one of the 2KB blocks is written to the first disk in the array and the second 2KB blocks is written to the second disk in the array. This, in theory, divides the work that a single hard drive would have to do in half, since the hard drives in the array only have to write a single 2KB file each.

When reading back, the outcome is just as pretty. If the original 4KB file is needed, both hard drives in the array move to and read a single 2KB block to reconstruct the 4KB file. Since each hard drive works independently and simultaneously, the speed of reading the 4KB file back should be the same as reading a single 2KB file back.

This pretty picture changes into a nightmare when we try to write the 8192KB file. In this case, to write the file, the RAID controller must break it into no less than 4096 blocks, each 2KB in size. From here, the RAID card must pass pairs of the blocks to the drives in the array, wait for the drive to write the information, and then send the next 2KB blocks. This process is repeated 4096 times and the extra time required to perform the breakups, send the information in pieces, and move the drive actuator to various places on the disk all add up to an extreme bottleneck.

Reading the information back is just as painful. To recreate the 8192KB file, the RAID controller must gather information from 4096 places on each drive. Once again, moving the hard drive head to the appropriate position 4096 times is quite time consuming.

Now let's move to the same array with a 1024KB stripe size. When writing a 4KB file, the RAID array in this case does essentially nothing. Since 4 is not divisible by 1024 in a whole number, the RAID controller just takes the 4KB file and passes it to one of the drives on the array. The data is not split, or striped, because of the large stripe size and therefore the performance in this instance should be identical to that of a single drive.

Reading back the file results in the same story. Since the data is only stored on one drive in our array, reading back the information from the array is just like reading back the 4KB file from a single disk.

The RAID 0 array with the 1024KB stripe size does better when it comes to the 8192KB file. Here, the 8192KB file is broken into eight blocks of 1024KB in size. When writing the data, both drives in the array receive 4 blocks of the data meaning that each drive only has the task of writing four 1024KB files. This increase the writing performance of the array, since the drives work together to write a small number of blocks. At the same time reading back the file requires four 1024KB files to be read back from each drive. This holds a distinct advantage over reading back a single 8192KB file.

As you can see, the performance of various stripe sizes differ greatly depending on the situation. Just what stripe size should you use?

CONFIGURING SQUID PROXY SERVER IN RED HAT LINUX

2012-03-17T12:45:00.000+05:30

Squid is the most popular proxy server in use today. Some of the reasons why squid is so popular are:

1. Squid is not an ordinary proxy server, it is a proxy cum caching server. This means that when a web page is accessed through squid, it caches the web page on the local hard disk. When a request is made for the same web page either by the same client host computer or by any other computer, squid serves the web page from its local hard disk cache. This eliminates repetitive internet access and speeds up client web access.

2. Squid has excellent access control restrictions. For eg, you can restrict only certain computers to access the internet, restrict certain computers to access only certain web sites, restrict certain computers to access the internet only on certain days between certain time periods, restrict access to pornographic web pages etc. Squid can also be integrated with third party content filtering software to provide comprehensive and easily manageable web access restrictions.

3. Squid has extremely good logging capabilities, for eg, you can monitor which IP address has accessed which web site, how well your cache memory is being utilized, etc.

To install and configure squid proxy on our Linux system we need the following rpm:

squid-2.5.STABLE1-2

Edit the file squid.conf

[root@akhisar root]# vi /etc/squid/squid.conf

#Line No. 53

http_port 3128 # uncomment this line

#Line No.468

cache_mem 8 MB # uncomment this line

# Line No. 666

cache_dir ufs /var/spool/squid 100 16 256 # uncomment this line

# Line No.673

cache_access_log /var/log/squid/access.log # uncomment this line

# Line No.681

cache_log /var/log/squid/cache.log # uncomment this line

# ACCESS CONTROLS

#Recommended minimum configuration:

# Line Nos. 1678-1693 # These are the default ACLs configured for our use

acl all src 0.0.0.0/0.0.0.0

acl manager proto cache_object

acl localhost src 127.0.0.1/255.255.255.255

acl to_localhost dst 127.0.0.0/8

acl SSL_ports port 443 563

acl Safe_ports port 80 # http

acl Safe_ports port 21 # ftp

acl Safe_ports port 443 563 # https, snews

acl Safe_ports port 70 # gopher

acl Safe_ports port 210 # wais

acl Safe_ports port 1025-65535 # unregistered ports

acl Safe_ports port 280 # http-mgmt

acl Safe_ports port 488 # gss-http

acl Safe_ports port 591 # filemaker

acl Safe_ports port 777 # multiling http

acl CONNECT method CONNECT

# Line No.1694 onwards ( Write your own ACLs from this line onwards)

acl ltc1 src "/etc/allowedips"

acl ltc2 src "/etc/domainrestrictedips"

acl ltc3 dstdomain "/etc/domains"

acl ltc4 src "/etc/timerestrictedips"

acl ltc5 time SMT 12:00-14:00

acl badurls url_regex sex hardcore playboy.com

# Line No.1720

#http_access deny all # Do not uncomment this line if using ACLs

# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS

http_access deny badurls

http_access deny ltc4 ltc5

http_access allow ltc2 ltc3

http_access allow ltc1

# And finally deny all other access to this proxy

http_access allow localhost

http_access deny all

# TAG: visible_hostname

# If you want to present a special hostname in error messages, etc,

# then define this. Otherwise, the return value of gethostname()

# will be used. If you have multiple caches in a cluster and

# get errors about IP-forwarding you must set them to have individual

# names with this setting.

visible_hostname akhisar

`Save the file and Quit'.

Explanation of Configuration directives :

Line No.53

Squid listens for requests on tcp port 3128 by default, you can change this.

Line No.468

'cache_mem' specifies the ideal amount of memory to be used

for:

* In-Transit objects

* Hot Objects

* Negative-Cached objects

Data for these objects are stored in 4 KB blocks. This

parameter specifies the ideal upper limit on the total size of

4 KB blocks allocated. In-Transit objects take the highest

priority.

In-transit objects have priority over the others. When

additional space is needed for incoming data, negative-cached

and hot objects will be released. In other words, the

negative-cached and hot objects will fill up any unused space

not needed for in-transit objects.

If circumstances require, this limit will be exceeded.

Specifically, if your incoming request rate requires more than

'cache_mem' of memory to hold in-transit objects, Squid will

exceed this limit to satisfy the new requests.

NOTE: THIS PARAMETER DOES NOT SPECIFY THE MAXIMUM PROCESS SIZE.

IT ONLY PLACES A LIMIT ON HOW MUCH ADDITIONAL MEMORY SQUID WILL

USE AS A MEMORY CACHE OF OBJECTS. SQUID USES MEMORY FOR OTHER

THINGS AS WELL.

Line No.666

This specifies the top level directory squid will use as its cache folder.

`100' specifies the amount of space allocated in MB for this folder (100MB is the default, you can increase this value depending on your usage).

`16' is the number of upper level folders created under the top level cache folder. These upper level folders store the web pages that are more frequently accessed.

`256' is the number of lower level folders created under each of the 16 upper level folders. These lower level folders store the web pages that are less frequently accessed.

The top level folder `/var/spool/squid' is created automatically, the 16 upper level and 256 lower level folders are created only when the squid server service is started for the first time.

Line No.673

The access log file logs every client web request. It is used to monitor which IP address has accessed which web site.

Line No.681

Cache logging file. This is where general information about your cache's behavior goes.

ACCESS CONTROLS :

This section is used to impose access restrictions to Squid Proxy server.

Restrictions are imposed with the help of access controls lists (ACLs) that need to be created by the squid server administrator. Default access lists have been created but for secure access further customization is required.

The syntax used in writing ACLs is as follows :

acl <label> <keyword> <parameter>

Explanation :

Every access control list starts with the word `acl'. Label is just a descriptive name for that acl,it has no technical significance.

The first acl is named `ltc1',it uses the keyword `src'.

The keyword `src' stands for source IP address, i.e. the IP address of the client host computer making a web request via the squid proxy server. Instead of defining IP addresses individually we can write the IP addresses (one below the other) to a file and mention the path to that file on the `acl line'.

Here we have written a few IP addresses to the file `allowedips' created under the /etc subdirectory.

The second `acl',`ltc2' also refers to IP addresses that have been saved to a file `domainrestrictedips' created under /etc.

The third `acl',`ltc3' contains the keyword `dstdomain'. This keyword refers to domain names for eg,www.yahoo.com,www.rediff.com etc. Instead of specifying domain names individually we can write them to a file (one below the other) and mention the path to that file on the `acl line'. Here we have written a few domain names to the file `domains' created under /etc.

The fourth `acl',`ltc4' refers to IP addresses that have been written to a file `timerestrictedips' created under /etc.

The fifth `acl',`ltc5' specifies the keyword `time'. This keyword refers to days of the week and times. Here we have specified the days of the week as Sunday, Monday and Tuesday and the time as 12:00 p.m. to 2:00 p.m.

The sixth `acl', named `badurl', consists of the keyword `url_regex'. This keyword refers to regular expressions. Here we have mentioned a few expressions or phrases like sex, hardcore, playboy.com.

The six `acls' that we have created above are of no use until and unless they have been enforced.

To enforce these rules go to the following section :

# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS

http_access deny badurls

Explanation :

No url mentioning any of the expressions or phrases mentioned in that acl (sex,hardcore,playboy.com) will be allowed access to. In fact even if a search is done through google or any other search engine on any of these expressions, web access will be denied.

http_access deny ltc4 ltc5

Web access will be denied to all host computers having IP addresses contained in the file `/etc/timerestrictedips' on Sunday.Monday and Tuesday between 12:00 p.m. to 2:00 p.m.

http_access allow ltc2 ltc3

The host computers having the IP addresses defined in the file `/etc/domainrestrictedips' will be allowed access to only the web sites defined in the file /etc/domains.

http_access allow ltc1

All IP addresses mentioned in the file /etc/allowedips will be allowed unrestricted internet access other than access to site urls containing the phrases specified in the `badurls' acl.

# And finally deny all other access to this proxy

Explanation :

http_access allow localhost

Any user who accesses the internet from the squid proxy server itself will not be controlled by the access control lists and will have absolute unrestricted internet access.

This privilege should only be reserved for the squid server administrator.

http_access deny all

Any incoming web request that is not matched by any of the acl rules will be denied.

# TAG : visible_hostname

Specifies the hostname of the squid proxy server as defined in the file `/etc/hosts' or in DNS.

To start the squid proxy server service :

[root@akhisar root]# service squid start

Any change made to the file `/etc/squid/squid.conf' will need the service to be restarted.

[root@akhisar root]# service squid restart

To set the squid service to be permanently on:

[root@akhisar root]# chkconfig squid on

" Do not forget to create your access control files"

1. [root@akhisar root]# vi /etc/allowedips

192.0.2.179

192.0.2.176

2. [root@akhisar root]# vi /etc/domainrestrictedips

192.0.2.167

3. [root@akhisar root]# vi /etc/timerestrictedips

192.0.2.176

4. [root@akhisar root]# vi /etc/domains

www.yahoo.com

www.rediff.com

You can create any files by any name located in any folder, just specify the correct name and path to the files on the `acl' lines.

Note : Make sure that the time restricted ips are have also been defined in the `allowedips' file else those ips will be denied web access.

Any change made in any of these files will require the squid service to be restarted.

Visible_hostname akhisar à This line needs to be defined, else the squid service will fail to start. Set this parameter to the hostname of your computer as defined in the file /etc/hosts.

To configure authentication for Squid Proxy server :

We can also configure then squid server to request for authentication whenever a user tries to access the internet through squid proxy.

Perform the following configuration in the auth_param section of the squid.conf file.
[root@akhisar root]# vi /etc/squid/squid.conf
                               # TAG: auth_param
                                    auth_param basic program /usr/lib/squid/ncsa_auth /etc/squid/passwd
                                   auth_param basic children 5
                                   auth_param basic realm Squid proxy-caching web server
                                   auth_param basic credentials ttl 2 hours

The first line points to the name and location of the external authenticator program used. Here we use the ncsa_auth software located in the /usr/lib/squid folder. We need to input all authorized username and password combinations into the file specified viz. /etc/squid/passwd. This passwd file will not exist by default, it needs to be created. You can create a file by any name under any folder.

[root@akhisar root]# cd /etc/squid
[root@akhisar squid]# touch passwd

Enter the authorized usernames/passwords you require.

[root@akhisar squid]# htpasswd passwd proxyuser
New password:
Re-type new password:
Adding password for user proxyuser

Here we create an authorized user by the name of `proxyuser' and assign him a password. This username/password combination will need to be used by requesting users to authenticate themselves before accessing the internet.

Create an Access Control List for ncsa_auth in the file /etc/squid/squid.conf :

[root@akhisar root]# vi /etc/squid/squid.conf
                               # ACCESS CONTROLS
                                  acl password proxy_auth REQUIRED

                               # INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
                                      http_access allow password

Here the name of the acl we create is `password'. The keyword used is `proxy_auth'. The parameter used is `REQUIRED' which will cause ncsa_auth to accept any valid username/password combination for authentication.

INSTALLING A EXTERNAL MODEM :

To install an external modem :

Connect the modem to a serial port at the back of your computer and power it on.

Execute the following command :

# wvdialconf /etc/wvdial.conf

This starts the modem autodetection procedure,you will see the modem lights blinking and a series of messages will be output on screen stating that the modem is detected and the port on which it has been detected on(if all goes well).

[root@akhisar squid]# wvdialconf /etc/wvdial.conf

Scanning your serial ports for a modem.

Port Scan<*1>: Scanning ttyS0 first, /dev/modem is a link to it.

ttyS0<*1>: ATQ0 V1 E1 -- OK

ttyS0<*1>: ATQ0 V1 E1 Z -- OK

ttyS0<*1>: ATQ0 V1 E1 S0=0 -- OK

ttyS0<*1>: ATQ0 V1 E1 S0=0 &C1 -- OK

ttyS0<*1>: ATQ0 V1 E1 S0=0 &C1 &D2 -- OK

ttyS0<*1>: ATQ0 V1 E1 S0=0 &C1 &D2 +FCLASS=0 -- OK

ttyS0<*1>: Modem Identifier: ATI -- 33600

ttyS0<*1>: Speed 4800: AT -- OK

ttyS0<*1>: Speed 9600: AT -- OK

ttyS0<*1>: Speed 19200: AT -- OK

ttyS0<*1>: Speed 38400: AT -- OK

ttyS0<*1>: Speed 57600: AT -- OK

ttyS0<*1>: Speed 115200: AT -- OK

ttyS0<*1>: Max speed is 115200; that should be safe.

ttyS0<*1>: ATQ0 V1 E1 S0=0 &C1 &D2 +FCLASS=0 -- OK

Found a modem on /dev/ttyS0, using link /dev/modem in config.

Modem configuration written to /etc/wvdial.conf.

ttyS0<Info>: Speed 115200; init "ATQ0 V1 E1 S0=0 &C1 &D2 +FCLASS=0"

Edit the ‘wvdial.conf’ file and make changes to the highlighted sections

[root@akhisar squid]# vi /etc/wvdial.conf

[Dialer Defaults]

Modem = /dev/modem

Baud = 115200

Init1 = ATZ

Init2 = ATQ0 V1 E1 S0=0 &C1 &D2 +FCLASS=0

ISDN = 0

Modem Type = Analog Modem

Phone = "Your ISP's internet acces number"

Username = "Your internet username"

Password = "Your internet access password"

To dialup to the internet, execute the following command :

[root@akhisar root]# wvdial

2012-03-17T12:36:00.001+05:30

An Apache Web Server is a computer that stores information in html format.

An organization would normally like to store company related information that should be made available to its employees in a publicly accessible location. One such method of doing this is to put up this information in the form of html web pages so that employees and other related people can browse through it whenever they need to.

An organization that needs to have a presence on the internet needs to put up information about itself in a location that is accessible to the rest of the world. The company thus needs to register a domain name for itself for eg, abc.com, host this domain on a DNS server and setup a web server to host all information about itself in html format. The DNS server performs the task of redirecting requests for abc.com to the designated web server for the company.

There are a number of web server softwares available with the various operating systems for eg, IIS with Windows, Iplanet from Sun Solaris etc. The most popular web server software is however the Apache web server which accounts for almost 90% of the web server installations worldwide. In fact Apache is so popular that there even exists a Windows version of Apache.

The reasons for Apache being so popular are :

1. Being a Linux product it is free and open source.

2. It supports SSL encryption at no extra cost (SSL is the secure sockets layer implementation that creates an encrypted tunnel between the client computer and the web site it is accessing thereby securing all information flowing through that tunnel).

3. Its virtual hosting capabilities. (Virtual hosting is a concept wherein multiple web sites can be hosted on a single web server. This is however a feature of all web server software packages).

To install and configure Apache on our Linux box we need the following rpm :

httpd-2.0.40-21

To configure apache :

Edit the file httpd.conf

[root@akhisar root]# vi /etc/httpd/conf/httpd.conf

ServerRoot "/etc/httpd" # Line No.56

# StartServers: number of server processes to start # Line No 103

# MinSpareServers: minimum number of server processes which are kept spare

# MaxSpareServers: maximum number of server processes which are kept spare

# MaxClients: maximum number of server processes allowed to start

# MaxRequestsPerChild: maximum number of requests a server process serves

StartServers 8

MinSpareServers 5

MaxSpareServers 20

MaxClients 150

MaxRequestsPerChild 1000

</IfModule> # Line No.114

# Listen: Allows you to bind Apache to specific IP addresses and/or

# ports, in addition to the default. See also the <VirtualHost>

# directive.

# Change this to Listen on specific IP addresses as shown below to

# prevent Apache from glomming onto all bound IP addresses (0.0.0.0)

#Listen 12.34.56.78:80

Listen 80 # Line No.157

# (Apache web server listens on tcp port 80, by default).

# Apache runs as the user apache and group apache for security reasons

User apache # Line No.251

Group apache # Line No.252

# ServerAdmin: Your address, where problems with the server should be

# e-mailed. This address appears on some server-generated pages, such

# as error documents. e.g. admin@your-domain.com

ServerAdmin root@localhost # Line No.259

# ServerName gives the name and port that the server uses to identify itself.

# This can often be determined automatically, but we recommend you specify

# it explicitly to prevent problems during startup.

# If this is not set to valid DNS name for your host, server-generated

# redirections will not work. See also the Use CanonicalName directive.

# If your host doesn't have a registered DNS name, enter its IP address here.

# You will have to access it by its address anyway, and this will make

# redirections work in a sensible way.

ServerName localhost:80 # Line No.273

# (You should enter the IP address of the web server,hostname,FQDN or just localhost)

# Location of the html web pages for the company web site

DocumentRoot "/var/www/html" # Line No.289

# Line Nos.338 to 343

# Controls who can get stuff from this server.

Order allow,deny

Allow from all

</Directory>

# Specifies the names that can be used for the index or home page of the web site

DirectoryIndex index.html index.html.var index.htm index.php # Line No.408

# (By default index.htm and index.php are not included, you will have to add them

# here if you have index pages with those extensions).

# Line Nos.952 to 972

# Allow server status reports, with the URL of http://servername/server-status

# [Change the ".your-domain.com" (for eg,lansmart.com) to match your domain to enable.

# You can also enter the subnet address of the network from which web requests will be

# made, for eg, 192.0.2.0/24]

# To access this html page open your web browser and type in the following URL:

# http://www.lansmart.com/server-status. This page will now be inaccessible to everyone with the

# exception of users connecting from the authorized subnet 192.0.2.0.

SetHandler server-status

Order deny,allow

Deny from all

Allow from .your-domain.com

</Location>

# Allow remote server configuration reports, with the URL of

# http://servername/server-info (requires that mod_info.c be loaded).

# Change the ".your-domain.com" (for eg,lansmart.com) to match your domain to en# able. You can also enter your # subnet address here eg,192.0.2.0/24

# To access this web page open up your web browser and type in the following URL

# http://www.lansmart.com/server-info

SetHandler server-info

Order deny,allow

Deny from all

Allow from your-domain.com

</Location>

# Line No.1009 onwards

### Section 3: Virtual Hosts

NameVirtualHost 192.0.2.172

ServerName www.lansmart.com

DocumentRoot /var/www/html/lansmart

</VirtualHost>

NameVirtualHost 192.0.2.173

ServerName training.lansmart.com

DocumentRoot /var/www/html/training

</VirtualHost>

NameVirtualHost 192.0.2.171

ServerName sales.lansmart.com

DocumentRoot /var/www/html/sales

</VirtualHost>

<VirtualHost 192.0.2.171>

ServerName tech.lansmart.com

DocumentRoot /var/www/html/tech

</VirtualHost>

<Directory "/var/www/html/lansmart">

AuthType Basic

AuthName Lansmart

AuthUserFile /etc/httpd/passwd

require valid-user

</Directory>

# (Virtual Hosting as explained earlier is a means of configuring multiple web sites on a single web server. This saves us the cost of purchasing multiple computers or multiple network cards. Using just one computer and one network card we can host multiple web sites. This is especially made use of by web hosting providers. Virtual Hosting is of two types,

a. Name based hosting

Here, all web sites hosted on the web server use the same IP address.

b. IP based hosting

With IP based hosting, all web sites hosted use a different IP address.

Let us consider a scenario in which our organization `Lansmart Technologies' needs to host two four web sites: www.lansmart.com, training.lansmart.com, sales.lansmart.com and tech.lansmart.com.

The first two sites, www.lansmart.com and training.lansmart.com should be IP based and the other two sites should be name based.

Steps to follow to configure your apache web server for the above scenario:

[ The hostname of our Apache web server is akhisar having an IP address 192.0.2.171 ]

akhisar is also hosting DNS server services for ‘Lansmart’.

Step 1 (For IP based hosting):

Create two virtual cards eth0:0 and eth0:1 having the IP addresses 192.0.2.172 and 192.0.2.173 respectively.

Step 2 : Configure your DNS server to support these hostnames (Forward and reverse lookup)

First configure the index file /etc/named.conf to support the domain lanmsart.com

(Refer to the chapter on DNS Server for this)

Configure the forward lookup file for lansmart.com (Refer to the chapter on DNS)

[root@akhisar root]# vi /var/named/lansmart.forward

akhisar A 192.0.2.171

www A 192.0.2.172

training A 192.0.2.173

(akhisar is the actual hostname of the web server computer having the IP address 192.0.2.171)

Configure the reverse lookup file (Refer to the chapter on DNS)

[root@akhisar root]# vi /var/named/lansmart.rev

171 PTR akhisar.lansmart.com.

172 PTR www.lansmart.com.

173 PTR training.lansmart.com.

Step 3 (For Name based hosting) :

Configure CNAME records for the hostnames sales.lansmart.com and tech.lansmart.com in the forward lookup file).

[root@akhisar root]# vi /var/named/lansmart.forward

akhisar A 192.0.2.171

sales CNAME akhisar.lansmart.com.

tech CNAME akhisar.lansmart.com.

Create the file /etc/resolv.conf with the necessary information on the DNS server and on each Linux DNS client host computer.

Test your DNS configuration with the nslookup and dig commands to make sure that everything is working as it should.

Step 4: Configure Apache Web server

Configure the httpd.conf file as shown above to support the virtual hosts.

Create the necessary folders (in the /var/www/html folder), lansmart, training, sales and tech, to store the html pages for the respective web sites. Create the index.htm and the rest of the html pages in these folders.

A sample index.htm file is shown below:

[root@akhisar root]# vi /var/www/html/lansmart/index.htm

<html>

<h1>

Welcome to Lansmart

</h1>

</html>

In the file httpd.conf, the Virtual host sections define the name of the web site and the path to the folder in which the html pages for that site are stored.

Both IP based sites (www.lansmart.com & training.lansmart.com) will have the line `NameVirtualHost <IP Address>' since they are running on different IP addresses. The Name based site sales.lansmart.com will also have the line `NameVirtualHost <IP Address>' since it is running on a different IP address than the IP basedsites. However the site tech.lansmart.com will not have the `NameVirtualHost' line as it is running on the same IP address as sales.lansmart.com, i.e. 192.0.2.171.

(The Virtual host sections are written in XML).

Testing

Connect to the apache web server from a client computer, open up your web browser and test connectivity to all four web sites :

www.lansmart.com

training.lansmart.com

sales.lansmart.com

tech.lansmart.com

‘Web pages for all the four sites should open without any problem’.

Testing Authentication :

The `Directory' section specifies the authentication to be used before being granted access to the site. Here we configure authentication for the site www.lansmart.com. This would require protecting the html pages for this site (stored under /var/www/html/lansmart). AuthType specifies the type of authentication being used, here the keyword `Basic' specifies that username/password authentication is being used. Whenever a request is made for www.lansmart.com an authentication window will open up requiring the user to enter an authorized username/password combination. `AuthName' is the heading that appears at the top of this authorization window. `AuthUserFile' specifies the name and path to the file in which the authorized usernames and password have been stored. We define this file to be `passwd' stored in /etc/httpd.

The keyword `require valid-user' will prevent non-authorized access to the web site www.lansmart.com.

The file `/etc/httpd/passwd' does not exist by default, it will have to be created.

[root@akhisar root]# cd /etc/httpd

[root@akhisar httpd]# touch passwd

Enter the authorized username and password combinations you require :

[root@akhisar httpd]# htpasswd -m passwd webuser

New password:

Re-type new password:

Adding 'password for user webuser

Here we create an authorized user called webuser and assign him a password.

Create additional authorized usernames/passwords if required.

Note : The authorized usernames that we create need not be those of valid users existing on the Linux web server computer. The names can be fictitious unlike our Samba server configuration which requires valid usernames for authentication.

Whenever a user tries to access the web site www.lansmart.com an authentication windows pops up asking for a valid username and password combination.

Here the user needs to enter the authorized username as `webuser' and the password assigned to that user.

SETTING UP SENDMAIL AND POSTFIX MAIL SERVERS IN LINUX

2012-03-17T12:32:00.002+05:30

Email is critical to the success of an organization in today's times.

Red Hat Linux comes bundled with two email server softwares, Sendmail and Postfix.

An organization can implement mail in one of 2 possible ways:

They can either host their mail server at their own location or they can get it hosted by a web hosting provider. If mail is hosted at the clients' location itself then incoming email will come in directly to the mail server. There will not be a need for a pop download utility to pull mail from the internet mailboxes. This chapter concentrates on a mail scenario in which an organization has hosted its mail server at a web hosting provider's data center.

After choosing a domain name by which it would host its mail and web related services the organisation now has to decide the kind of email access it would be offering its designated email users.

Users can be given access to email in two possible ways, pop access and aliased access.

With pop access each designated user of email will have a mailbox assigned to him on the internet mail server. Separate pop accounts will be created for each user, the user will use the username and password assigned to him to retrieve his email. The advantage of this type of access is that the user will have access to his mail from anywhere in the world. He could be at a cybercafe or elsewhere, he would only need a connection to the internet, all he has to do is to open up his web browser access the webmail service provided by the web hosting provider enter his username and password and he would be able to view all his messages.

Each user mailbox would have to be assigned a certain amount of space hence careful planning will have to done before opting for a mail server space usage plan. Pop access is therefore more suited to roaming users or those users who are constantly on the move.

The other type of email access is aliased access. Here there is only one pop box configured called a catchall account. Each designated user will not be assigned a separate mailbox for his use, instead each user will have an aliased account created. Mail addressed to any of the aliased accounts will be stored in the pop box of the catch all account. Aliased users will be able to access their email only when they are at office, not otherwise. Hence this type of email access is more suited to those users who do not travel.

Irrespective of the type of email access, the organisation needs to setup a mail server within its premises to download all the email that is being sent to the internet mail server. All client host computers that need access to email will configure their email clients (Outlook Express, Outlook, Pegasus Mail, Mozilla Mail etc.) to send and recieve mail via this Linux based mail server. Outgoing external email sent by each users will be stored in the outgoing mail queue.

This office mail server will be configured to connect to the internet at regular intervals in time, download all incoming mail stored on the internet mail server and send out all the queued outgoing email.

This chapter teaches you to install and configure the Sendmail and Postfix mail servers to get email up and running at your organisation.

SENDMAIL

For POP users:

The organization `Lansmart Technologies' has registered a domain

name,`Lansmartindia.net', for its use. POP accounts have been configured

for the users ryan, santosh, shaji, rhea etc.

To install Sendmail we need the following rpms :

sendmail-8.12.10-1.1.1

sendmail-cf-8.12.10-1.1.1

sendmail-devel-8.12.10-1.1.1

To configure sendmail :

Edit the file sendmail.cf

[root@akhisar root]# vi /etc/mail/sendmail.cf

# my official domain name

# ... define this only if sendmail cannot automatically determine your domain

#Dj$w.Foo.COM # Line No.95

# "Smart" relay host (may be null)

DS # Line No.100

# avoid connecting to "expensive" mailers on initial submission?

O HoldExpensive=False # Line No.191

O DaemonPortOptions=Port=smtp,Addr=127.0.0.1, Name=MTA # Line No.264

Make the following changes to the Line Nos. shown above in the sendmail.cf file.

Line No.95 -> Change this to the name of the domain name under which you

receive mail for your organization.

Djlansmartindia.net

Line No.100 -> If you wish to relay all external outgoing mail through a mail

relay host, for eg, your ISP's mail server, then enter the FQDN or the IP address

of that relay host, for eg:

DSsmtp.vsnl.net # To relay all outgoing mail through VSNL's smtp server.

Line No.191 -> If you use a dial up connection to the internet it is

considered to be an expensive connection. If you use broadband or a leased

line it is considered as inexpensive.

Hence if you use a dialup modem to connect to the internet change value to

True.

Line No.264 -> The Port No. and IP address on which your mail server is

listening on for SMTP requests. Enter the IP address of all network cards in your mail server on which the mail server is listening for SMTP requests.

Copy and paste this Line No.264 to Line No.265 and change the value of

Addr=127.0.0.1 to the Ip address of the ethernet card in your sendmail server.

O DaemonPortOptions=Port=smtp,Addr=127.0.0.1, Name=MTA

O DaemonPortOptions=Port=smtp,Addr=192.0.2.171, Name=MTA

# (Where 192.0.2.171 is the IP address of the ethernet card in the sendmail

# server).

Configure the file local-host-names :

[root@akhisar root]# vi /etc/mail/local-host-names

localhost.localdomain

lansmart.com

This file will contain the domain names for which the sendmail server will

receive mail. Mail addressed to any other domain name will be rejected unless

the sender IP address has been allowed to relay mail through the sendmail

server. Relay access restrictions are defined in the file

/etc/mail/relay-domains as shown below:

[root@akhisar root]# vi /etc/mail/relay-domains

192.0.2

Here we allow relay access to our internal network, no other IP address will be

allowed to relay mail through our server. An ISP however cannot deny relay access by IP addressing for the simple reason that the IP addresses of dialup clients will be unknown to the ISP. An ISP will instead make use of SMTP Authentication. This topic is however not in the scope of this book.

Configure POP3 and IMAP access to our mail server :

For POP3 access :

POP3 is controlled by the xinetd super server. Enabling this would access to

mail from POP3 email clients like Outlook Express, Microsoft Outlook, Pegasus

Mail, Netscape Mail, Mozilla Mail etc.

[root@akhisar root]# vi /etc/xinetd.d/ipop3

disable = yes

Change this line to :

disable = no

Restart the xinetd server service :

[root@akhisar root]# service xinetd restart

For IMAP access :

IMAP affords us a way to access our email and at the same time allow our mail

to remain on the server. IMAP too is controlled by the xinetd super service.

[root@akhisar root]# vi /etc/xinetd.d/imap

disable = yes

Change this line to :

disable = no

Restart the xinetd server service :

[root@akhisar root]# service xinetd restart

We have now configured our mail server to send internal as well as outgoing

mail. Our mail server now has to be configured to fetch the incoming mail

accumulating on the internet mail server space and have it redistributed to

the respective users on our office mail server.

To do this we use a utility called fetchmail. Fetchmail needs to be configured

for all users that have pop accounts configured on our internet mail server.

All designated email users can now connect to this mail server to send and

receive email. All mail sent by these users to the addresses lansmart.com and

localhost.localdomain will be routed internally to the recipient user's

mailbox. All mail sent to external email addresses will be spooled in the

outgoing mail queue. This queued email will be flushed out when the mail

server connects to the internet.

To flush outgoing mail (when the mail server connects to the internet) :

[root@akhisar root]# sendmail -q -v

(-q -> flush queued outgoing email

-v -> verbose, displays whatever is happening in the background)

Configuring fetchmail :

We need the following rpm :

fetchmail-6.2.0-3

Change to the home folder of the user having a pop account, for eg, the user

ryan, and create the file .fetchmailrc.

[root@akhisar root]# cd /home/ryan

[root@akhisar ryan]# vi .fetchmailrc

poll mail.lansmartindia.net with protocol POP3

user ryan is * here

password ryan123

Change ownership on the .fetchmailrc file.

[root@akhisar ryan]# chown ryan.ryan .fetchmailrc

Change the permissions on .fetchmailrc

[root@akhisar ryan]# chmod 700 .fetchmailrc

(This allows only the user ryan to execute his .fetchmailrc file).

Here mail.lansmartindia.net is our internet pop mail server. The next two

lines indicate that the user ryan on the internet mail server is the same user

ryan on our office mail server. Fetchmail connects to the user ryan's pop

mailbox on the internet mail server (when the server is connected to the internet), collects all the accumulated incoming email and downloads it to the user ryan's mailbox on the internal office mail server. `ryan123' is the password of the user ryan on the office mail server.

Configure fetchmail in an identical fashion (substitute ryan for the

respective user names of the other pop users) for all other users who have pop accounts configured on

the internet mail server.

To execute fetchmail :

[ryan@akhisar ryan]$ fetchmail -a -v

(-a -> all mail

-v -> verbose)

For aliased users:

There is only one pop catchall account configured on the internet mail server.

Mail addressed to all users is directed to this mailbox. We need to configure

fetchmail for only one user account on the office mail server which will pull

all the email from the catchall account on the internet mail server.

The downloaded email will then need to be distributed to the respective users

on the office mail server. The mail thus needs to be filtered with a mail

filtering utility like procmail.

Procmail filters all the mail downloaded into the pop account mailbox on the

office server with the help of `recipes'. These recipes defines what actions

are taken on each email stored in the pop user's mailbox.

Consider our scenario in which lansmartindia.net is our catchall account

created on our internet mail server. Create a user account `lansmartindia' on

the office mail server. Create a .fetchmailrc file for lansmartindia as shown

above and change the ownership and permissions accordingly.

[root@akhisar root]# cd /home/lansmartindia

[root@akhisar lansmartindia]# vi .fetchmailrc

poll mail.lansmartindia.net with protocol POP3

user lansmartindia is * here

password lansmartindia123

Change ownership on the .fetchmailrc file.

[root@akhisar lansmartindia]# chown lansmartindia.lansmartindia .fetchmailrc

Change the permissions on .fetchmailrc

[root@akhisar lansmartindia]# chmod 700 .fetchmailrc

(This allows only the user lansmartindia to execute his .fetchmailrc file).

Email addressed to all users will now be downloaded to the mailbox of the user

lansmartindia on the office mail server. To filter the emails with procmail

create .procmailrc file in the home folder of lansmartindia.

[root@akhisar lansmartindia]# vi .procmailrc

:0 Wh: msgid.lock

| formail –D 8192 msgid.cache

SHELL=/bin/sh

MAILDIR=/var/spool/mail

LOGFILE=${MAILDIR}/procmail.log

VERBOSE=yes

#Recipes

* ^TO_ryan@lansmartindia.net

! ryan@localhost.localdomain

* ^TO_santosh@lansmartindia.net

! santosh@localhost.localdomain

* ^TO_shaji@lansmartindia.net

! shaji@localhost.localdomain

* ^TO_rhea@lansmartindia.net

! rhea@localhost.localdomain

The first two lines of the .procmailrc file help in eliminating duplicate messages. The id of every incoming email message is stored in an 8KB cache. ID numbers of all incoming messages are compared to the existing ids already present in cache. An incoming email with the same id number is considered to be a duplicate message and is dropped immediately. Thus no duplicate mail can get through the procmail filter.

The first part of the .procmailrc file defines the shell interpreter as the

sh shell, the user mailbox directory as /var/spool/mail, the log file as

procmail.log, to be stored in this dirtectory and a verbose option to be

enabled.

The second part is the `Recipes' which is the heart of procmail. Here, for

eg, in the case of the user ryan, any mail from external internet users

addressed to the mail address ryan@lansmartindia.net will be redirected to the

local mailbox of the user ryan on the office mail server. The same explanation

holds true for the users santosh, shaji and rhea.

Change the ownership and permissions on the file .procmailrc.

[root@akhisar lansmartindia]# chown lansmartindia.lansmartindia .procmailrc

Change the permissions on the file .procmailrc

[root@akhisar lansmartindia]# chmod 664 .procmailrc

Create the procmail.log file in the `/var/spool/mail’ folder. Give read and write access permissions on this file to the user root and group lansmartindia.

[root@akhisar lansmartindia]# cd /var/spool/mail

[root@akhisar mail]# touch procmail.log

[root@akhisar mail]# chown root.lansmartindia procmail.log

[root@akhisar mail]# chmod 664 procmail.log

Sendmail has now been configured to send and receive email for all designated

users but the process is not automated. For outgoing email to be sent the root

user needs to login, connect to the internet and then execute the command

sendmail -q -v. For the users to receive their mail each user has to login and

execute the command fetchmail -a -v.

The above scenario is just not practical in a real world scenario.

To fully automate this process perform the following steps :

Login as root user and create a file for eg,/root/mail.

[root@akhisar root]# vi mail

#!/bin/bash

su ryan -c fetchmail

su santosh -c fetchmail

su shaji -c fetchmail

su rhea -c fetchmail

Change the permissions on the file to executable.

[root@akhisar root]# chmod 755 mail

This file enables the root user to execute fetchmail on behalf of all other

users, the other users hence do not need to login individually to download

their email.

In the case of aliased users the file /root/mail will contain only a single

line for the lansmartindia user.

[root@akhisar root]# vi mail

#!/bin/bash

su lansmartindia -c fetchmail

However, to connect to the internet, the root user has to manually execute

`wvdial'.

To send the queued outgoing mail the root user has to execute

`sendmail -q -v'. To automate this as well,

Create a file ip-up.local in the folder /etc/ppp.

[root@akhisar root]# vi /etc/ppp/ip-up.local

#!/bin/bash

sendmail -q -v

/root/mail

/usr/bin/killall pppd

/usr/bin/killall wvdial

Make the file executable.

[root@akhisar root]# chmod 755 /etc/ppp/ip-up.local

This file will send all queued outgoing email, download all incoming email for

the users specified in the file /root/mail, after this the pppd daemon is killed

and so is the wvdial connection to the internet (The last two commands in this

file assume that you will be using a dialup connection to the internet and

will not want the telephone line to remain connected to the internet

throughout the day for obvious reasons. For a broadband or a leased line

internet connection the last two lines will not be required).

This file will automatically be executed once a connection has been

established to the internet.

To automate the wvdial connection to the internet you could configure crontab

to periodically schedule a wvdial connection to the internet at absolute or

repetitive intervals in time.

Here we setup crontab to schedule wvdial to connect to the internet every 30

mins.

[root@akhisar root]# crontab -e

0-59/30 * * * * wvdial

Sendmail has now been fully automated. It will connect to the internet every

30 minutes, will send queued outgoing mail and download all

incoming mail stored on the internet mail server.

Test :

Designated email users can now test the newly configured server by connecting

to it from their POP3 and IMAP mail clients.

POSTFIX MAIL SERVER

Postfix is a relatively newer MTA than sendmail. It is considered to be faster and more secure than sendmail. Security has always been an issue with Sendmail, the creator of sendmail has done his best to fix the security loopholes in it, but nothing short of a complete redesign will eliminate the existing security flaws.

Postfix is known for its speed and reliability. This too is a free, open source MTA which comes bundled with the Red Hat installation CDs.

To install Postfix we need the following rpms:

postfix-1.1.11-11.i386.rpm

To switch from Sendmail to Postfix use the following command :

[root@akhisar root]# redhat-switch-mail

From the menu that appears choose postfix.

Shutdown the sendmail MTA and disable it from starting at the next boot sequence by executing the following command:

[root@akhisar root]# chkconfig sendmail off

To configure Postfix :

[root@akhisar root]# vi /etc/postfix/main.cf

# INTERNET HOST AND DOMAIN NAMES

# The myhostname parameter specifies the internet hostname of this

# mail system. The default is to use the fully-qualified domain name

# from gethostname(). $myhostname is used as a default value for many

# other configuration parameters.

myhostname = akhisar.lansmart.com # Line No.68

# The mydomain parameter specifies the local internet domain name.

# The default is to use $myhostname minus the first component.

# $mydomain is used as a default value for many other configuration

# parameters.

mydomain = lansmart.com # Line No.76

# SENDING MAIL

# The myorigin parameter specifies the domain that locally-posted

# mail appears to come from. The default is to append $myhostname,

# which is fine for small sites. If you run a domain with multiple

# machines, you should (1) change this to $mydomain and (2) set up

# a domain-wide alias database that aliases each user to

# user@that.users.mailhost.

myorigin = lansmartndia.net # Line No.87

# RECEIVING MAIL

# The inet_interfaces parameter specifies the network interface

# addresses that this mail system receives mail on. By default,

# the software claims all active interfaces on the machine. The

# parameter also controls delivery of mail to user@[ip.address].

#inet_interfaces = localhost # Line No.97

#inet_interfaces = all

#inet_interfaces = $myhostname

#inet_interfaces = $myhostname, localhost

# The mydestination parameter specifies the list of domains that this

# machine considers itself the final destination for. That includes

# Sendmail-style virtual domains hosted on this machine.

# Continue long lines by starting the next line with whitespace.

mydestination = localhost, localhost.localdomain, localhost.lansmart.com, akhisar. lansmart.com, lansmartindia.net # Line No.125

#mydestination = $myhostname, localhost.$mydomain $mydomain

#mydestination = $myhostname, localhost.$mydomain, $mydomain,

#mail.$mydomain, www.$mydomain, ftp.$mydomain

# TRUST AND RELAY CONTROL

# The mynetworks parameter specifies the list of "trusted" SMTP

# clients that have more privileges than "strangers".

# In particular, "trusted" SMTP clients are allowed to relay mail

# through Postfix. See the smtpd_recipient_restrictions parameter

# in file sample-smtpd.cf.

# You can specify the list of "trusted" network addresses by hand

# or you can let Postfix do it for you (which is the default).

mynetworks = 192.0.2.0/24, 127.0.0.0/8 # Line No.171

#mynetworks = $config_directory/mynetworks

#mynetworks = hash:/etc/postfix/network_table

# INTERNET OR INTRANET

# The relayhost parameter specifies the default host to send mail to

# when no entry is matched in the optional transport(5) table. When

# no relayhost is given, mail is routed directly to the destination.

#relayhost = $mydomain # Line No.221

# ALIAS DATABASE

# The alias_maps parameter specifies the list of alias databases used

# by the local delivery agent. The default list is system dependent.

# On systems with NIS, the default is to search the local alias

# database, then the NIS alias database. See aliases(5) for syntax

# details.

# If you change the alias database, run "postalias /etc/aliases" (or

# wherever your system stores the mail alias file), or simply run

# "newaliases" to build the necessary DBM or DB file.

# It will take a minute or so before changes become visible. Use

# "postfix reload" to eliminate the delay.

#alias_maps = dbm:/etc/aliases

alias_maps = hash:/etc/postfix/aliases # Line No.298

# mailbox_command = /some/where/procmail # Line No.359

# mailbox_command = /some/where/procmail –a “$EXTENSION” # Line No.360

defer_transports = smtp # Line No.542

always_bcc = mailadmin # Line No.543

Explanation for the Line Nos. mentioned in the postfix main.cf configuration file

Line No.68

The `myhostname' parameter defines the hostname of your postfix mail server.

Line No.76

The `mydomain' parameter specifies the name of the internal domain you have configured your DNS with. In this case we have chosen our internal domain name to be `lansmart.com'.

Line No.87

The `myorigin' directive specifies the name of your internet mail space. This is the name that we have registered our internet domain with. In this case we have chosen lansmartindia.net to be our internet domain name. Mail for all users in the organization `Lansmart' will now be addressed to <user>@lansmartindia.net.

Line No.97

The lines nos.97-100 define the network interfaces on which the Postfix mail server is going to listen to for smtp requests. By default Line No. 97 is left uncommented, which means that by default the postfix mail server will function only if smtp requests are made from the mail server itself i.e. from 127.0.0.1. We naturally would want to enable the postfix server to listen for smtp requests from all network interfaces present in it. Hence the best thing to do is to either comment out Line No.97 and leave the rest of the lines as they are, or comment out Line No.97 and uncomment Line No.98 (inet_interfaces = all). Leaving all lines commented achieves the same functionality as uncommenting Line No.98,it enables the postfix server to listen for smtp requests on all interfaces.

Line No.125

This line defines the domain names or destinations for which the postfix mail server will receive mail. Mail addressed to any domain that is not in this list will be rejected unless the sender IP address has been allowed to relay mail through the postfix server. Here we have also included the hostname localhost.lansmart.com which at first glance might not seem necessary. However this is needed to distribute incoming email to the respective users' mailboxes.

Line No.171

This line prevents your mail server from being an open relay. An open relay allows any sender of email to relay his mail through your mail server. Open relays are found out and exploited very quickly by spammers to send thousands of unsolicited messages to third parties. This is a very serious offence, a mail server known to be an open relay will soon find itself in the Realtiem Blackhole List (RBL) and will be isolated form other mail servers on the internet.

Here we specify that only users on our internal office network (192.0.2.0/24) will be allowed to relay mail through our server.

Line No.221

If you wish to relay all external outgoing mail through another mail server for eg, your ISP's smtp server, then, mention the IP address or the FQDN of that smtp server in this field.

For eg:

relayhost = mail.lansmartindia.net

(where mail.lansmartindia.net is the FQDN of your webhosting provider's smtp server)

Line No.298

With the help of the aliases directive incoming mail addressed to a particular user can be directed to another user or group of users. Aliases are configured in the file /etc/postfix/aliases. After making a change to this file execute the `newaliases' command:

[root@akhisar root]# vi /etc/postfix/aliases

ryan: ryan,santosh

Here any mail addressed to the user ryan will also be sent to the user santosh.

[root@akhisar root]# cd /etc/postfix

[root@akhisar postfix]# newaliases

The new aliases command converts the aliases file to the hashed database format, i.e. the .db format.

# Line Nos.359 & 360

These lines define the name and location of the mail filtering utility used with postfix. Procmail is the utility used to filter mail and it is made use of by both Sendmail as well as Postfix.

Uncomment these lines and edit them to point to the actual location of the procmail executable i.e. /usr/bin. After editing , the lines should read as follows :

# mailbox_command = /usr/bin/procmail

# mailbox_command = /usr/bin/procmail –a “$EXTENSION”

Line No.542

This line should be used only if you are on a slow dialup connection to the internet that is not permanently on. This directive will cause all outgoing email to be deferred in the mail queue (/var/spool/postfix) until a connection has been established to the internet.

If you are using a broadband or internet leased line connection this line should not be configured.

Line No.543

Postfix offers us the facility to send a copy of every outgoing as well as incoming mail to an administrative mail account. Here we mark a bcc to a user account called mailadmin. A copy of every message sent to an internal user or to an external address as well as every incoming message will now be sent to the mailadmin mail acount.

The rest of the configuration procedure is identical to that for Sendmail.

Follow the steps outlined in the chapter on Sendmail to configure POP3 and IMAP mail access, fetchmail and procmail.

Testing :

Users can now connect to the postfix mail server from their mail clients like Outlook Express, Microsoft Outlook, Mozilla Mail etc. and test their sending and receiving of email.

DISK QUOTAS

2012-03-17T12:24:00.000+05:30

Diskquotas are implemented as a means of restricting users to using only a certain amount of hard disk space.

Since users cannot exceed the space limits imposed upon them there is more effective utilization of storage space.

This concept is largely employed by web hosting providers and security conscious companies.

Diskquotas can be employed for users, groups or both. The can also be implemented to restrict hard disk space in MB or to restrict the number of files created.

However as is obvious, restricting users to creating only a limited number of files will not serve much purpose as there is no limit imposed on the size of the individual files. Hence restricting users to a limited amount of hard disk space is the way to go.

Requirement : Restrict users on hard disk space.

Solution : Implement Disk Quotas.

Since users create and store their data under their respective home folders we should implement disk quotas on the /home partition.

Disk Quota functionality has been built into the kernel ,hence there is no need to recompile the kernel.

Procedure :

Step I : Edit the file /etc/fstab

[root@akhisar root]# vi /etc/fstab

/home /home ext3 defaults,usrquota 1 2 # To implement diskquotas for users

To implement disk quotas for groups :

[root@akhisar root]# vi /etc/fstab

/home /home ext3 defaults,grpquota 1 2

#To implement disk quotas for both,users and groups :

[root@akhisar root]# vi /etc/fstab

/home /home ext3 defaults,usrquota,grpquota 1 2

Step II : Create a blank file, aquota.user or aquota.group or both under /home.

[root@akhisar root]# cd /home

[root@akhisar home]# touch aquota.user

(If implementing disk quotas for users)

To implement disk quotas for groups :

[root@akhisar home]# touch aquota.group

To implement disk quotas for both,users and groups :

[root@akhisar home]# touch aquota.user aquota.group

Change the permissions on the files created :

[root@akhisar home]# chmod 600 aquota.user

# chmod 600 aquota.group ( If present )

Step III : Reboot the server for the changes made to /etc/fstab to take effect.

Step IV : Calculate the space already being used by the users and dump the information to the files aquota.user or aquota.group as applicable.

[root@akhisar home]# quotacheck -u /dev/hdax (for users)

# quotacheck -g /dev/hdax (for groups)

# quotacheck -ug /dev/hdax (for both, users and groups)

(The quotacheck command scans the home folder of every user, calculates the disk space used by each user and dumps the information to the files aquota.user and aquota.group as is applicable. The size of the aquota.user or aquota.group files then increase.

`x' is the partition number of the /home partition, which can be obtained by executing the following command :

[root@akhisar root]# df -h (this command will output information about all the linux partitions on the server hard disk.

Step V : Turn the quota settings on

[root@akhisar home]# quotaon -av (For users)

# quotaon -avug (For users and groups)

Step V1 : Edit the quota settings for individual or multiple users, or groups

[root@akhisar root]# edquota -u ryan (edit quota settings for the user ryan)

edit quota settings for multiple users that need similar quota settings.

# edquota -p santosh,shaji (if the users santosh and shaji need identical quota settings)

# edquota -g sales (edit quota settings for the group sales)

# edquota -u ryan

Disk quotas for user ryan (uid 518):

Filesystem blocks soft hard inodes soft

hard

/dev/hda3 424 0 0 115 0

To restrict the user ryan to a maximum of 5MB of hard disk space :

Disk quotas for user ryan (uid 518):

Filesystem blocks soft hard inodes soft

hard

/dev/hda3 424 2000 5000 115 0

(As soon as the user ryan reaches or crosses his soft limit of 2MB he gets a warning message saying that his softlimit has been reached or exceeded. He will still be able to work as long as he stays under his hard limit of 5MB. The moment he toches his hard limit of 5MB he will be stopped from working.

The moment ryan touches his soft limit of 2MB his grace period counter activates and starts decrementing with each passing day. At the end of 7 days although ryan might still be under his hard limit of 5MB,since his grace period has expired,his soft limit will be converted to his hard limit and he will be stopped from working.

Step VII : Test

Step VIII : Run a check on quota settings applied to all users of the system

[root@akhisar home]# repquota -u /dev/hdax (`x' is partition number of /home)

*** Report for user quotas on device /dev/hda3

Block grace time: 7days; Inode grace time: 7days

Block limits File limits

User used soft hard grace used soft hard grace

----------------------------------------------------------------------

root -- 32 0 0 6 0 0

new -- 76 0 0 18 0 0

ryan -- 424 2000 5000 115 0 0

santosh -- 88 0 0 22 0 0

shyam -- 92 0 0 22 0 0

rhea -- 28 0 0 7 0 0

john -- 52 0 0 18 0 0

nilesh -- 40 0 0 10 0 0

raj -- 40 0 0 10 0 0

boss -- 40 0 0 10 0 0

hacluster -- 40 0 0 10 0 0

qscand -- 56 0 0 14 0 0

[ The above report shows that diskquotas have been enabled only for the user ‘ryan’. Softlimit for ryan =2MB

hardlimit = 5MB ]

To run a check on quota settings for a particular user ryan :

[root@akhisar home]# quota -u ryan

Disk quotas for user ryan (uid 518):

Filesystem blocks quota limit grace files quota limit grace

/dev/hda3 424 2000 5000 115 0 0

To edit the default grace period :

[root@akhisar home]# edquota -t

(This is a global setting,it cannot be set per user)

Grace period before enforcing soft limits for users:

Time units may be: days, hours, minutes, or seconds

Filesystem Block grace period Inode grace period

/dev/hda3 7days 7days

To change the block grace period to 14 days :

Time units may be: days, hours, minutes, or seconds

Filesystem Block grace period Inode grace period

/dev/hda3 14days 7days

Important :

To erase a user from the system, first disable the quota settings for that user.

A user who has his quota settings activated cannot be deleted.

Files and Directory permissions in LINUX

2012-03-17T12:17:00.000+05:30

Every file or folder created on the Linux file system has 3 sets of permissions.

Each set has 3 permissions for a total of 9 permissions.

The 3 permissions are:

r -> read

w -> write

x -> execute

Each of these permissions have been assigned numerical values,

r = 4

w = 2

x = 1

To view the rights assigned to a file or folder :

-rw-r--r-- 1 root root 34 Aug 31 11:49 abc

-rw-r--r-- 1 root root 1254 Aug 29 18:48 anaconda-ks.cfg

-rw------- 1 root root 375 Sep 2 09:50 asiainfotech.vcf

drwxr-xr-x 6 501 501 4096 Sep 6 18:14 DB_File-1.810

The first column indicates the rights assigned to that particular file or folder.

Each set of three permissions (rights) can have a maximum value of rwx i.e.

rwxrwxrwx (777).

The first set of 3 permissions (rwx) is for the user owner, the second set (rwx) is for the group owner and the third set (rwx) is for others.

For eg. take the first file abc. It has the following rights assigned :

-rw-r--r-- root root

The first column indicates that the user owner has been granted the rights read (r) & write (w) but not execute (x). the missing permission execute is replaced with a dash (-).

The group owner has been granted only the read (r) permission but no write (w) ad execute (x). the missing permissions have been replaced with dashes (--).

All others have been granted only read (r) permission.

User Owner : The user that creates a file or folder becomes the user owner of that file or folder.

Group Owner : The primary group of that user becomes the group owner.

Others : All other users who do not belong to the primary group of the user owner fall into the others category.

A file will have a dash (-) as the first character of the `ls -l' output.

A directory or folder will have a `d' as the first character.

[ The third column denotes the name of the user owner, the fourth column indicates the name of the group owner, the rest of the fields indicate the size of the file, date & time of creation & the name of the

file ]

Before modifying the rights assigned to files or folders it is important to know what rights are assigned to a file or folder when it is first created.

The rights assigned to a file or folder when it is first created depend upon which user has created that particular file or folder, the root user or an ordinary user. This is because of a parameter called the umask setting whose value differs for the root user and an ordinary user.

The default umask value for the root user is 022 and that for an ordinary user is 002.

When the root user creates a file the umask setting is subtracted from 666,

i.e. 666-022 = 644.

Hence when a root user creates a file the file gets permissions of 644,i.e.

-rw-r--r--.

When the root user creates a folder the umask setting is subtracted from 777,

i.e. 777-022 = 755.

Hence when a root user creates a folder the folder gets permissions of 755,i.e.

drwx-r-xr-x.

When an ordinary user creates a file the umask setting is subtracted from 666,

i.e. 666-002 = 664.

Hence when an ordinary user creates a file the file gets permissions of 664,i.e.

-rw-rw-r--.

When an ordinary user creates a folder the umask setting is subtracted from 777,

i.e. 777-002 = 775.

Hence when an ordinary user creates a folder the folder gets permissions of 775,i.e.

drwx-rwxr-x.

With this background knowledge it now becomes easier for us to manipulate file and folder permissions and user and group ownership.

Take the example of the previous file `abc'.

# ls -l abc

-rw-r--r-- 1 root root 34 Aug 31 11:49 abc

To give `w' permission to `others' :

# chmod 646 abc

# chmod o+w abc

Granting permissions the alphabetical way is relatively easier than assigning rights the numerical way.

To take away the `w' right from others :

# chmod o-w abc

To grant the `x' permission to the user owner, write permission to the group owner and x permission to others :

# chmod u+x,g+w,o+x abc

To assign permissions on folders follow the same procedure with the addition of the -R (recursive) switch which forces permissions to be inherited by the underlying files and folders in the subdirectory structure.

Take the example of the `/evolution' folder :

drwx------ 8 root root 4096 Sep 7 13:07 evolution

# chmod -R o+w /evolution

drwx---w root root evolution

# chmod -R u-x,g+rx,o+r /evolution

drw-r-xr-- root root evolution

To change user and group ownership :

Consider the file abc

# ls -l abc

-rw-r--r-- root root abc

[ Here the user owner is ‘root’ and the group owner is ‘root’ ]

Changing user and group ownership using chown and chgrp commands :

Requirement :

We want to assign the `x' permission to only three users ryan, santosh and shaji. In this scenario we cannot assign the `x' permission to others because that would give all other users the `x' permission.

Solution :

Add the three users ryan, santosh and shaji into a group, for eg. sales.

Change the group ownership of the file abc to sales.

[root@akhisar /]# gpasswd -M ryan, santosh, shaji sales

[root@akhisar /]# chgrp sales abc

[root@akhisar /]# ls -l abc

-rw-r--r-- 1 root sales 26 Sep 10 09:19 abc

To change the user ownership of the file abc to user rhea :

[root@akhisar /]# ls -l abc

-rw-r--r-- 1 root sales 26 Sep 10 09:19 abc

[root@akhisar /]# chown rhea abc

[root@akhisar /]# ls -l abc

-rw-r--r-- 1 rhea sales 26 Sep 10 09:19 abc

To change the user owner and group owner to user `ram' and group `tech' simultaneously.

[root@akhisar /]# chown ram:tech abc

[root@akhisar /]# ls -l abc

-rw-r--r-- 1 ram tech 26 Sep 10 09:19 abc

Changing user and group ownership on directories :

Follow the same procedure as shown above with the addition of the -R (recursive) switch. The -R switch will cause the user and group ownership permissions to be inherited by the underlying file and folders in the directory tree.

For eg: To change the group ownership on the folder /data

[root@akhisar /]# chgrp -R sales data

[root@akhisar /]# ls -l

drwxr-xrwx 3 root sales 4096 Aug 29 13:49 data

To change the user ownership on /data to user shaji :

[root@akhisar /]# chown -R shaji /data

[root@akhisar /]# ls -l

drwxr-xrwx 3 shaji sales 4096 Aug 29 13:49 data

To change the user owner and group owner on /data to user santosh and group mktg respectively :

[root@akhisar /]# chown -R santosh:mktg /data

[root@akhisar /]# ls -l

drwxr-xrwx 3 santosh mktg 4096 Aug 29 13:49 data

Installing Libre office in Fedora 16

2012-03-17T11:55:00.000+05:30

Requirements:

Linux kernel version 2.6.18 or higher;
glibc2 version 2.5 or higher;
gtk version 2.10.4 or higher;
Pentium-compatible PC (Pentium III, Athlon or more-recent system recommended);
256Mb RAM (512Mb RAM recommended);
Up to 1.55Gb available hard disk space;
X Server with 1024×768 resolution (higher resolution recommended), with at least 256 colors;
Gnome 2.16 or higher, with the gail 1.8.6 and at-spi 1.7 packages (required for support for assistive technology [AT] tools), or another compatible GUI (such as KDE, among others).

For certain features of the software – but not most – Java is required. Java is notably required for Base.

For installing your libre office you just have to give the command:

# sudo yum install yum groupinstall "Office/Productivity"

Webcam in fedora 16 during post installation

2012-03-17T11:50:00.001+05:30

For working with your webcam you can use the cheese package to do your job. Just install cheese by giving the following command in the terminal:

# sudo yum install cheese

Kudos...........

Installing your VLC player on Fedora 16 (For all fedora versions above 11)

2012-03-17T11:44:00.004+05:30

VLC player in my Fedora

Upgrade your rpmfusion-free-release-stable.noarch.rpm in the system. Type the command for root when prompted.

$ su -c 'rpm -Uvh http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-stable.noarch.rpm'

$ sudo yum install vlc

That is it...you have installed your favourite VLC player in your new fedora 16... :)