Do It With Linux

What are Index Nodes?

2014-01-29T16:47:00.000+00:00

I'm writing about this because it came up as an issue while at work one day.

Background info!
Quite a while ago I wrote some software which keeps an eye on a number of servers and sends me an email if anything goes wrong, one of the things being if the servers hard disk becomes full / the system cannot create new files.

So I'm sat doing my daily java programming and I receive and email saying our main server is full, pretty strange because each server should have at least 20GB of space free at all times. So I log on to the system to see whats taking up all this room, the first command I do is df -h and I see the following output:

This is not the output I expected, so I thought maybe there was a problem writing files to the disk so I run a touch text.txt command and this seems to run perfectly fine. After looking thought my email software to see if it had somehow made a mistake I get another email saying the server is having problems writing logs, this is when I start getting confused.

I log back onto the system and check the logs. The logs had indeed stopped, as a test I wrote a small bash script to create a new file, it didn't work, instead it came back saying the hard disk was full.

After some extreme head scratching and chin stoking I found the source of my problem by running the command df -i and seeing that my IUse% was 100%.
The problem was these strange little things called 'i-nodes'.

Index Nodes (i-nodes)
An index node is a data-structure used to represent a file system object, this can be disk block location, creation time and a whole host of other metadata. Each index node is roughly 128 bytes in size and all index nodes should really only ever take up 1% of your hard disk.

So for each file on your hard disk there is a i-node. So the solution presented itself, All I had to do was find the largest directory and backup and remove the files. To find the largest directory on the system I ran this command: du -a | sort -n -r and I instantly found the issue. Another developer had written a program which was producing a hell of a lot of log files, every 2 seconds a new log file (very small in size) was created. While this wasn't taking up much room on the hard drive it was taking up alot of index nodes.

So there's a solution to anyone else having this issue.

What The Hell is A Shell?!

2013-11-15T15:01:00.004+00:00

I've had a lot of people ask this question before, myself included. I thought I would write a small straight to the point post about the differences between the most common.

I'll start by explaining exactly what a shell is and what it is used for. (Although most of you will probably already know)

What the hell is a shell?!
Before graphical user interfaces the command line was the main interface for users, just like now where we have KDE, Gnome etc... the command line had shells. Shells were used to give the user a slightly different environment in terms of functionality and scripting. In technical terms a shell is a command line interpreter developed to make things easier for users. One example of this would be filename/command completion (do this by pressing tab after the first part of the command/filename). These small features that a lot of people see as part of linux is actully part of the shell.

Popular shells
The most popular shell (and the shell most of you will probably be using) is the Bash Shell written by a guy called Brian Fox. This is the most popular shell due to its robustness and tons of features. I use this shell at home all the time for scripting.

The Bourne Shell, AKA 'sh'. This shell was developed by a guy called Stephen Bourne, It quickly became the default shell for Unix due to its support for variables, piping, looping and a verity of other usefull stuff. I use this shell mainly when writing software for embedded systems as it is the default shell.

The C-Shell AKA 'csh' was developed by a guy called Bill Joy. This guy added useful things like aliases (short cuts for long commands) and command history. This shell was modeled on the C programming language, this was a good thing as Unix was written in C...although it did mean users of the bourne shell had to learn C in order to use the C-Shell.

Which is the best shell?
There are a large number of shells to choose from, I've only named three here as these are the most popular. I would say the best shell is the Bash Shell due to how popular it is, this means most scripts written with this shell will most likely work on any machine.

Sed command examples

2012-08-03T16:24:00.000+01:00

A command I use constantly at work is the 'sed' command. This command is used when you want to make changes to a file automatically. Most people who use the 'sed' command use it to replace text (which it does brilliantly) but there is so much more
I'm going to show you a few examples of the awesomeness of this command, if you have never used it before I guaranty you will use it after reading this post.

Lets say we have a file which contains the following text:

"This is my favourite place to learn about Linux. Linux is an opensource OS. Linux has a variety of different flavours."

Search and Replace
Using the 'sed' command we can search and replace. for example if I wanted to replace the word 'Linux' with the word 'Unix' I would do the following:

sed 's/Linux/Unix/' aFile.txt

This would change the first occurrence of "Linux" to "Unix" as shown below:

"This is my favourite place to learn about Unix. Linux is an opensource OS. Linux has a variety of different flavours."

Replacing the nth occurrence of a word is just as easy:

sed 's/Linux/Unix/2' aFile.txt

This will skip the first occurrence of "Linux" and replace the second one with "Unix".
"This is my favourite place to learn about Linux. Unix is an opensource OS. Linux has a variety of different flavours."

If we would like to replace all occurrences of a word then we use the "/g" (Global) option:

sed 's/Linux/Unix/g' aFile.txt

All occurrences of the word "Linux" have now been replaced with "Unix"
"This is my favourite place to learn about Unix. Unix is an opensource OS. Unix has a variety of different flavours."

We can mess about with these options and merge them together to replace all occurrences of a word from a certain point, for example replacing all occurrences of the word "Linux" with the word "Unix" after the first occurrence (the example below will make this a bit clearer):

sed 's/Linux/Unix/2g' aFile.txt

So after the second occurrence the word "Linux" changes to "Unix".

Add a line after or before a match
Here we can add a line after a match is found. For this we use the "a" command, for example:

sed '/flavours/ a "a new line!"' aFile.txt

and the result is as follows:
"This is my favourite place to learn about Linux. Unix is an opensource OS. Linux has a variety of different flavours.
a new line!"

The same thing can work adding a line before a match, we do this using the "i" command.

sed '/flavours/ i "a new line!"' aFile.txt

and the result is as follows:
"This is my favourite place to learn about Linux. Unix is an opensource OS. Linux has a
a new line!
variety of different flavours."

Replace a line
We can search for a match and then replace that whole line using the 'c' command. It pretty much the same format as before:

sed '/flavours/ c "a replaced line!"' aFile.txt

and the result is as follows:
"This is my favourite place to learn about Linux. Unix is an opensource OS. Linux has a
a replaced line!"

The last thing I'm going to show you is the transform option. This is used to transform lower case letters to upper case letters. We use the 'y' command for this. In this example I'm going to change all the small l's and v's to large l's and v's, and vice versa:

sed 'y/lv/LV/ aFile.txt

"This is my faVourite pLace to Learn about linux. Unix is an opensource OS. linux has a Variety of different fLaVours." I hope this helps you guy's with the 'sed' command, it can be very useful once you get to know it. Any questions are welcome.

How To Install Software On Linux (It's not that hard!)

2012-06-20T09:12:00.001+01:00

OK, so I have had enough with people asking me how to install stuff on Linux, it's really not that difficult but it does seem to put a lot of people off the OS. I'm going to go through a few things here such as rpm, tar.gz and tar.bz2.

RPM

rpm stands for "Red hat Package Manager". Files with this extension are installed and managed with the rpm program which I will take you through now.

The rpm program not only installs the software but it checks your system has all the prerequisites. For example if you are trying to install a package called "LinuxWow" but that package requires "LinuxFTW" that you haven't installed rpm will not install "LinuxWow". However if you have "LinuxFTW" then the package will be install as it should. This can save a lot of frustration... believe me.

before installing an rpm file it might be a good idea to check if you already have this installed, to do this run the following command:

rpm -q package_name

-q = query

The commands you would use to install, update or remove a .rpm file are as follows:

Install one or multiple rpm files

rmp -ivh packages1.rpm packages2.rpm

-i = install

-v = verbose information (additional info)
-h = Print 50 hash marks as the package archive is unpacked

Update one or multiple rpm files
rpm -Fvh packages1.rpm packages2.rpm

-F = Freshen

Remove one or multiple rpm files
rmp -e packages1.rpm packages2.rpm

-e = erase.

TAR.GZ & TAR.BZ2

tar files usually contain source code so the idea is that you compile (build) them yourself rather then relying on a program like rpm.

I'll show you how to install these files in 3 steps:
1. First this is to list the packages in the file, this is to make sure we don't over right anything precious on our system. We will use the 'less' command for this. The '|' can be used as an extension of a command, so we could do ' | grep "fileName"' if we wanted to grep the package for a certain file name.

tar tvzf package.tar.gz | less
or
tar tvzf package.tar.bz2 | less
-t = list, this is to list the contents of the package
-v = verbose, you know this by now
-z = this filters the package through gzip, like win-zip if you're from windows
-f = uses the package file

2. If we are happy with everything then we extract the files to a new directory, as shown below:
we make a new directory called newdir:
mkdir newdir
Enter the directory:
cd newdir
and extract the files:
tar xvzf <path>/package.tar.gz
or
tar xvzf <path>/package.tar.bz2

3. once extracted look for a file called 'install' or 'readme' these will give you the rest of the information you need. They usually tell you to run a script called 'configure', after that we run the 'make', (this will ask us to enter our root password) command and then finally the 'make install'.
./configure
make
su
(enter password)
make install

The 'readme' file may contain some other information about the configure script which may require you adding some options but the file should give you all the information you need.
It's really not that difficult to install software on a Linux machine. If you use Ubuntu like me then a simple "apt-get install package" would suffice. I hope this post makes people realise how easy it is.

Slightly more advanced guide to Shell Scripting

2012-05-11T11:15:00.002+01:00

Hey guys,

In this post I hope to go through a few more advanced options when writing scripts. I'm just going to cover things like methods, parameters and a few other tricks which can make your script look pretty cool.

Methods

As a Java developer by trade I use methods quite a lot, programming without them is sort of odd, for those who don't know a 'method' or 'function' in basic terms is a section of code which can be called at a later time, you can probably see why this would be useful.

Say you have a menu system in your script where the user must choose an option for the script to continue, each option would produce a different outcome, this is a good example of when you would want to use methods.

Examples

#!/bin/bash

echo "***************EXAMPLE SCRIPT***************"

echo "Please choose and option:"

echo " 1 = option A"

echo " 2 = option B "

echo " 3 = option C "

read option

if [[ "$option" == "1" ]]

then

OPTIONA

exit 0

if [[ "$option" == "2" ]]

then

OPTIONB

exit 0

if [[ "$option" == "3" ]]

then

OPTIONC

exit 0

OPTIONA () {

echo "Hello world, this is option A"

}

OPTIONB () {

echo "Hello world, this is option B"

}

OPTIONC () {

echo "Hello world, this is option C"

}

Ok so I'm going to dissect this and explain what each part is doing, its quite simple really.

echo "***************EXAMPLE SCRIPT***************"

echo "Please choose and option:"

echo " 1 = option A"

echo " 2 = option B "

echo " 3 = option C "

This part is the actual menu which will be shown to the user, we use the 'read option' to read the input from the user and assign it to the variable 'option', this is so we can use it later in the if statements.

if [ "$option" == "1" ]

then

OPTIONA

exit 0

if [ "$option" == "2" ]

then

OPTIONB

exit 0

if [ "$option" == "3" ]

then

OPTIONC

exit 0

Here we have the if statements, quite straight forward really, what's happening is we are calling the 'option' variable, if this equals '1' for example then we call the OPTIONA method.

OPTIONA () {

echo "Hello world, this is option A"

}

As you can see option A echo's out the line "Hello world, this is option A". After each method the code 'exit 0' is run, this is just to exit the script cleanly.

Parameters

A good example of parameters would be if you wanted to write a script to remove all '.txt' files from a directory. In order to do this you could write a script which takes in the directory path as a parameter. This makes the script more flexible, it gives the user the choice of which directory input. The bash shell supports up to 9 parameters which are represented in the script as '$1' (if you remember awk this might make more sense) 1 being the number of the parameter.

Example

#!/bin/bash

(cd $1 && rm *.txt)

This is a very simple example, to run this script (lets call it remove.sh) you would do the following:

./remove.sh /home/zdrenka/pictures

The script would then assign ' /home/zdrenka/pictures' to '$1' and that's what we call a parameter, there are loads of useful ways in which to use this facility.

Since I started scripting I have picked up bits and bobs which can make your script look pretty cool, one of which is colours. Colours can make a script look that little bit more impressive. When I write a script one of the first things I do is paste this at the top:

################################################

# Define colours

################################################

txtund=$(tput sgr 0 1) # Underline

txtbld=$(tput bold) # Bold

txtred=$(tput setaf 1) # Red

txtgrn=$(tput setaf 2) # Green

txtylw=$(tput setaf 3) # Yellow

txtblu=$(tput setaf 4) # Blue

txtpur=$(tput setaf 5) # Purple

txtcyn=$(tput setaf 6) # Cyan

txtwht=$(tput setaf 7) # White

txtrst=$(tput sgr0) # Text reset

the '#' are obviously comments. I wrote this a while ago and now have it saved as a template, I use the colours to highlight errors, its very useful when you have a lot of text on the screen. to use these colours we just add them to the text as shown below:

echo "${txtbld}${txtred}ERROR File does not exist ${txtrst}";

The 'txtrst' variable will reset the text back to its original colour, you need this otherwise all text below your statement will be red.

Another useful command I have come across is the 'set -e' command. Put this at the top of your script and the script will automatically quit if it comes across and error, this is good practise as if you hit an error its not usually a good idea to carry on with the script. Although if you would like to have this command skip a certain part of the code you can surround that part with 'set +e' and 'set -e'. for example:

set +e

cp *.txt *.jar *.war /home/zdrenka/Java

set -e

Without these my script would stop once it hit this cp command as there may not be any '.war', '.txt' or '.jar' files in my Java directory which would technically throw up an error.

I hope this quenches your thirst for knowledge, I'll put more posts on soon!

2012-04-12T20:13:00.002+01:00

SCP Command Examples
Make a secure copy of this page guys!

2012-03-18T17:12:00.003+00:00

Netcat Command Examples
Netcat is out the bag!

Mplayer Command Examples

2012-03-13T11:37:00.001+00:00

mplayer is a media player, its mainly used to play audio or video.
What I like about it is how you can use it in the command line.

For example to play a audio file.
maplyer mediafile.mp3
Its as simple as that.

For a video its slightly differnt. Mplayer does not support dvd menu's which is a bit of a pain really so you need to do the following
mplayer dvd://titlenumber

obviously titlenumber needs to be replaced with your desired title number. Xine is a lot better for DVD menu's but thats a whole other kettle of fish

If you wish to move forward in a track or film there a a whole load of keyboard short cuts avalible here
I know your thinking "whats the big deal?" but the real reason I use mplayer is to rip audio from video files.
I use this a lot in conjunction with the youtube-dl program.
mplayer -ao pcm -vo null -vc dummy -dumpaudio -dumpfile someVideo nameOfAudioFile.mp3

I've taken the liberty off adding a selection of commands below:

Play all files in directory
$mplayer -playlist <(find "$PWD" -type f)

Stream Youtube directly to mplayer
$yt () mplayer -fs -quiet $(youtube-dl -g "$1")

Create an animated gif of a youtube video
url=http://www.youtube.com/watch?v=V5bYDhZBFLA; youtube-dl -b $url; mplayer $(ls ${url##*=}*| tail -n1) -ss 00:57 -endpos 10 -vo gif89a:fps=5:output=output.gif -vf scale=400:300 -nosound

use mplayer to play just the audio from a video
mplayer -vo null something.mpg

Shuffle songs in mplayer with no output
mplayer -shuffle $HOME/music/* $HOME/Dropbox/Music/*/* $HOME/Dropbox/Music/*/*/* etc. >> /dev/null 2>&1

Mail Command Examples

2012-03-06T20:18:00.001+00:00

mail [-t] [-w] [ -m message_type ] recipient

The Mail Command is a good way (not the only way) of sending/receiving emails via the console.

Root is required for this:

mail someone@somewhere.com

This command will set up the email, it will then ask you for a subject, CC's and message.
once your done You need to type . (dot) to send the email. Then Enter.
It's better if you use the following command:

mail -s "subject" too@somewhere.com -- -r "from@somewhere.com" -f </path/to/a/file>

This command will send an email with a subject to an address from the address specified, the
-f </path/to/a/file> is the file which contains the information,in my case an error log.

this can be shorted to:
mail -s "subject" someguy@someplace.com < atextfile.txt

I often find myself using this command instead, its quicker to type...I should really make a script to do all this for me.

More posts coming, and scripts!

My Top 5 SSH Commands

2012-03-06T10:07:00.001+00:00

SSH(Secure Shell) is a network protocol for secure data communication, unlike Telnet SSH connects via a secure channel over an insecure network.
This means that data sent over SSH is encrypted.
Below I have added some useful commands for using SSH, Have Fun :)

ssh -t reachable_pc ssh unreachable_pc

I've had the problem where I need to connect to my PC at work, unfortunately
the ip address is unreachable, but it is reachable from the server at work.
Operation SSH man-in-the-middle is in effect!

ssh user@host cat /path/to/a/text/file | xclip

This is good for copying a file to your work machine and having it sit
in your X11 buffer (clipboard) so then just right click and paste contents of the file

ssh-copy-id user@host

This is very handy if you need to connect to a large number of hosts
it copys the SSH keys thus giving you password-less logins :D!

dd if=/dev/dsp | ssh -c arcfour -C username@host dd of=/dev/dsp

This is a strange one but kinda fun, I found this through google so its not mine
this will output the sound from you microphone to a remote computers speakers via SSH
funky eh?!

ssh -N -L2005:localhost:80 remotehost

One of the best things about SSH is Tunnelling, with the above command you can
view a remote hosts current website by going to http://localhost:2005

Beginners' Guide to Shell Scripting

2012-03-04T15:29:00.003+00:00

It has come to my attention that a lot of people need a beginners' guide to shell scripting, therefore I am obliged to provide one.

This tutorial will provide a grounding for shell scripting.

Shell scripting in Linux isn't to difficult to master, although it helps if you have some programming background. We'll start by explaining the shell.

The shell is a program which actually processes commands and returns output. Bash is the most popular Linux shell as it contains standard features such as foreground and background processes and command history.

A question that is asked a lot is: "what's the difference between the Console and the Terminal?"

The Terminal is a wrapper program which runs a shell. This was once a physical device which consisted of not much more then a monitor and keyboard. As time moved on this became software and can be used to access the shell and run commands.

The Console is a special kind of Terminal, the Console used to be used for low level communication with the operating system. These days it is used in much of the same way as the Terminal.

In conclusion there isn't really much difference between the two, they both have separate pasts but are now essentially used for the same thing.

Writing Scripts

I recommend using gedit to write your scripts as it has a lot of great tools to make use of, although if you are unfortunate enough to be using a Windows system I recommend Notepad++ (I use this at work to write scripts and it's very similar to gedit).

At the top of each script the we have a Shebang line. It consists of a number sign and an exclamation point character (#!), followed by the full path to the interpreter such as /bin/bash. All scripts under UNIX and Linux execute using the interpreter specified on a first line. Then we would have the command we wish to execute.

#!/bin/bash

echo Hello Linux!

The above script when run will display the words "Hello Linux" in the console/terminal. It uses the echo command which does what it says on the tin, it will echo out the words Hello Linux!.

This is one way of doing it, however the use of variables is more elegant. A variable stores data in memory and can be called at differnt times within the script, it can also be changed and amended throughout the script. Below is an example of Hello Linux with variables.

#!/bin/bash

VAR="Hello Linux!”

echo $VAR

Next we must learn how to use the chmod command, this is use to set the permissions for the script we have just created.

Chmod

the chmod command (change mode) is a command that lets the user control how much access it should permit to a file. In order to run our script we must first use chmod to make it executable to do this I use the command

chmod +x helloLinux.sh

the +x makes the scipt executable for ALL users, if you just want it to be exe for you then u+x is for you.

After this command a simple ./helloLinux.sh will run the script.

Hopefully this should give you a good grounding for writing your own shell scripts.

Any questions are welcome!

Grep Command Examples

2012-03-01T10:03:00.000+00:00

grep [options] PATTERN [FILE...]

The grep command basicaly searches though a file for lines which match the given pattern
, for example:

grep apple fruit.txt

this will output all lines which contain the word apple, its as easy as that!
You could also specify a search in more then one file with the '*' wildcard

grep apple *.txt

like all good commands grep lets you use regular expressions such as the following:

grep ^a fruit.txt

the above line will return all lines which begin with the letter a.

Once you get the hang of grep you can do some pretty cool things, heres a command which I use a lot (provided by Sean, he know's who he is), it looks
complicated but I'll break it down:

grep --color=always -niC number "string" filename | less -R

--color=always
this colours the pattern to make it jump out at you when searching
-niCn would be the line number, this isnt always needed so it can be removed.
'i' is to match both upper and lowercase
'C number' is the number of lines surrounding the matchs you want to show, the alternative is A and B which is to show lines before or after the match

"string" is the pattern, but then you already knew that ;)

filename is the file you are searching, you can use the '*' wildcard as mentioned above or you can type another file name next to it which will also work

| less -R
this is the pipe that lets you scroll down the output of the search, the -R is the necessary command to let the color work with the scrolling
basiccaly this makes it easier to read.

you can also save the ouput as a txt file by using | tee output.txt and appending it to the end of the grep command, the cool thing is that the color stays in the output.txt!

There are loads of useful switches to use with grep try 'man grep' in the console

2012-02-29T11:46:00.002+00:00

Awk Command Examples
awk can be a bit tricky but hopefully this will give you a good idea of its functionality.

2012-02-28T12:51:00.002+00:00

Cron Examples
Cron is somthing I find most useful, its definitely worth learning about.

2012-02-28T10:42:00.003+00:00

The 'at' command basically executes a command at a specified time, surprisingly useful if you need to print something at 3pm and you're useless at remembering stuff..

The Below command will run the jobtodo script at 15:00

at -m 15:00 < jobtodo.sh

at -l - this will list all jobs scheduled

atrm 3 - this will remove job 3 from the list

To be honest if you want something that will run every day your better off looking at crontab. But if you after something quick and simple this will do just nicely. I use this command at work for running scripts at set times of the day, that way I wont forget.

Download BBC shows on Linux with get_Iplayer

2012-02-27T18:59:00.002+00:00

get_Iplayer is an easy way to download tv and radio from the bbc iplayer website.
get_iplayer can be downloaded from linuxcentre.net

the below command can be used to download TV or radio given the correct link.

sudo get_iplayer --get --type <type> --url=<url>

<type> - the type will either be 'tv' or 'radio' depending on what you wish to download.
<url> - to get the url its best if you click 'Link to this' button on Iplayer website.

This has been very useful for me as I'm constantly missing my favorite shows. Below I have added a very simple script:

#!/bin/bash
echo "tv or radio?"
read option
echo "PLease paste in you link for the content you wish to download:"
read url
get_iplayer --get --type $option --url=$url

this can be adapted to make a series link. I suggest looking into 'crontab' or the 'at' command