TechDiary

Extract or Reap or Download Pictures from HTML Web Pages

vikas.nv@gmail.com (Viki) — Thu, 13 Aug 2009 04:08:13 PDT

I was exploring python and the HTML parsers that it has. I hit a page that explained about the SGMLParser and its uses.

Using this package I had written a simple script that could extract pictures from simple HTML pages. While experimenting it with many websites I noticed some websites did not allow my script to reap or extract pictures.

Then I found that it is because of the User-Agent string in the HTTP request. Some websites do not respond properly to HTTP requests that have unknown User-Agent string. I then used the urllib's FancyURLOpener to change the User-Agent to Firefox.

Now that the script work on most of the websites, i would like to post the code here. Feel free to modify to make it more robust and reliable, post a comment or a link to your modified script.

This script search pictures from web pages and download them for you. All you need to do is
get the URL of the web page from which you want the pictures to be downloaded and pass it as
an argument to this script. Here is how to use it:
0. Note that python is installed on your system to run this script (Windows/Linux/Solaris ... OS independent;)
1. Download this script and save it as gal_ext.py
2. Open a command-prompt (if Windows) or Terminal (if you use any *nix OS)
3. Type "python gal_ext.py 'http://URL'
4. Here note that the 'http://' is mandatory
Using this script you can avoid downloading images with a web browser, which takes a long time and is annoying to right click on every image and save them. This way you can download picture from web faster and easier. Since this is a script it is open for modifications and is free to use. But if you modify, make a note to send it to me ;). Thanks

The code is here (or a link for you to download):


from sgmllib import SGMLParser
import sys
import os
import re
import urllib
from urllib import FancyURLopener
from urlparse import urlparse

class URLLister(SGMLParser):
def reset(self):
SGMLParser.reset(self)
self.urls = []

def start_a(self, attrs):
href = [v for k, v in attrs if k=='href']
if href:
self.urls.extend(href)

class MyURLOpener(FancyURLopener):
version = 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.6) Gecko/2009011913 Firefox/3.0.6'

def url_mapper(var):
regexp = re.compile('.+?(http://.+&?)', re.I)
values = urlparse(var)
if len(values) >= 5 and values[4] != '':
obj = regexp.search(values[4])
if obj:
values = urlparse(obj.group(1))
else:
if values[4] != '': print('Could not get real url from:' + values[4])
site = re.sub('[^0-9a-zA-Z_/]', '_', values[1])
location_file = re.findall('(.+)/(.+)', values[2])
try:
if len(location_file[0]) != 2:
return ''
except IndexError:
return ''
location = location_file[0][0]
file = location_file[0][1]
location = re.sub('\.{2,}/', '/', location)
location = re.sub('[^0-9a-zA-Z_/]', '_', location)
location = location + '/' + file
return site + location

if __name__ == '__main__':
url_opener = MyURLOpener()
parser = URLLister()
sys.argv[1] = urllib.unquote(sys.argv[1])
try:
usock = url_opener.open(sys.argv[1])
except IOError:
print 'skipping ' + sys.argv[1]
sys.exit(0)
parser.feed(usock.read())
parser.close()
usock.close()
count = 0
urlfs = urlparse(sys.argv[1])
print "Url fs: ", urlfs
parent = urlfs[1]
try:
parent += urlfs[2]
parent = re.search('(.+)/.+',parent).group(1)
except IndexError:
parent = urlfs[1]
for img_url in parser.urls:
if re.search('\.jpe?g$', img_url):
print "looking at: " + img_url
if not re.match('^http://', img_url):
img_url = 'http://' + parent + '/' + img_url
loc = url_mapper(img_url)
if os.path.exists(loc): continue
url_opener1 = MyURLOpener()
retrieve = url_opener1.retrieve
if loc == '':
print "\turl_mapper returned NULL for " + img_url
continue
loc_dir = re.match('(.+)/.+', loc).group(1)
try:
print('\ttrying to save ' + img_url)
if not os.path.exists(loc_dir):
 os.makedirs(loc_dir)
retrieve(img_url, loc)
except IOError:
print('\tSkipping saving ' + img_url)
continue
count += 1
print "\tImg fetched: ",count

Installing andLinux on Windows Vista

vikas.nv@gmail.com (Viki) — Fri, 29 May 2009 08:28:37 PDT

After downloading the andLinux installer (KDE version), launch the installer as Administrator. And follow the steps with default options set.

Reboot/Restart machine after installation.

Click on Unblock if the Firewall prompts.

After the reboot, you click on Konsole (any) launcher on KDE Menu from system tray, you are likely to get this error "could not launch ... could not connect to 192.168.11.150"

So this guide explains how to fix this issue, for the launchers to work on Windows Vista.

First thing to do is to go to the "Network and Sharing Center" from Control Panel. You can see the "customize" and "view status" links on "Unidentified Network". Click on the "view status", click on properties and double click "Internet Protocol Version 4". That should open up a dialog where you can change the IP and the subnet mask, change the IP to 192.168.10.1 and subnet mask to 255.255.254.0. Click on OK OK ....

Secondly, open the "andlinux Console" get sudo access and edit the file /etc/network/interfaces.
After edit the file should have IP of 192.168.10.150 and the mask 255.255.254.0 under eth1

Thirdly edit file /etc/profile to have IPs set to 192.168.10.150

Fourthly navigate to the andlinux installation folder goto Xming folder edit X0.hosts and add the following IPs 192.168.10.150 and 192.168.10.1

Lastly few registry changes, regedit and navigate to \\HKEY_LOCAL_MACHINE\\SOFTWARE\\andLinux\\Launcher
Here you find to keys "IP" and "Port" set them to 192.168.10.150 and 2081 respectively.
(Remember to click on Decimal while editing Port value) And close the regedit.

Finally kill KDE Menu, Xming. Restart andLinux service, and then start Xming and KDE Menu.
Now the launchers work fine.

If clicking on a launcher does not start, you open the andLinux Console application and execute:
"/etc/init.d/networking restart". After this any launcher should work fine.

!Enjoy!

Parsing and Storing parts or paragraphs of log file

vikas.nv@gmail.com (Viki) — Sat, 09 May 2009 07:12:07 PDT

At many times we need to parse and store parts of a log file. Suppose an error log, you would like to parse all errors. If the errors span only one line, you could use a grep.

But if the errors span multiple lines ... you need to put a little logic into your script. There are many ways to do this, but this script that i post here is more readable and can be modified to other's requirement.

Link to the code
Link to a sample input

In the example script i am parsing an error log (named 'input.txt'). The script stores all lines that start with "SQL0204N" till it finds an empty line. Likewise it extracts all errors, even if the error messages span multiple lines.

Perl script inside a DOS Batch file

vikas.nv@gmail.com (Viki) — Wed, 01 Apr 2009 03:46:03 PDT

Whenever I wanted to invoke a utility passing a file as an argument I used to create a shortcut of the utility in the Send To folder, as described here :

How to Add Items to the "Send To" Menu

But how to have a Perl script in the "Send To" menu??

I got an idea when browsing through this page :

DOS Batch - Interfacing non DOS Software

I created a batch script as described in the page. Then modified the part of the batch file where the Perl script actually starts.

I wanted to have a utility that can encrypt/decrypt a text file. A basic encryption was sufficient, so considered the Caesar cipher.

I completed the script and added it to the Send To menu.
Link to the script

A small modification to the above script to handle binary files. Here is the link.

andLinux --- Run Linux from Windows

vikas.nv@gmail.com (Viki) — Sat, 13 Dec 2008 09:28:55 PST

Have you ever had a feeling of using Linux when you are working on Windows???

Felt like using those very useful Linux commands fromWindows???

This is possible ...

Everybody knows that you can have a Dual-Boot on your PC with Linux & Windows. In which you have to restart the PC to access the other OS.

But there are several other ways to have Other OS run right from Windows, that is for example use Linux when you have logged on to Windows... Yes... Virtual Machines (also popular as VM) is one of them. But running a VM requires that your hardware to be really good.

There is one more option, we now have emulated PC software like andLinux.

There are many other softwares other than andLinux, such as CygwinX, MKS Toolkit, etc

But I found andLinux to be really very handy, straight-forward & useful.

Installing andLinux is as simple as installing other software ... clikcing Next, Next, Next ... & finnaly Finish :) there is a very elaborate tutorial on its web site

However if you are the curious cat... you can also fiddle around & do the installation without the tutorial.

The home page of andLinux says "andLinux is not just for development and runs almost all Linux applications without modification". I have installed a lot many applications to andLinux, and its really true.

After installation you can see the icons of andLinux in system tray. You may see these icons :

KDE menu (andLinux)

Xming (andLinux)

While installing say you choose E: drive as destination. It gets installled in a folder E:\andLinux.

It will create some folders & files in E:\andLinux, I will list some folders & files with thier purposes:

1. settings.txt, a plain-text file that will have some paths necessary for andLinux to work properly

2. srvstart.bat, batch file, that has the command to start the andLinux service

3. srvstop.bat, batch file, has the command to stop andLinux service

4. startup.bat, again a batch file used to start the andLinux as deamon

5. Drives, directory, where the andLinux's filesystem is stored as files (base.drv, mine is 4GB in size and swap.drv)

6. Launcher, directory, where all the executables can be seen. A good place to check out what all linux apps you get after installing andLinux.

7. There are other directories as well, netdriver, pulseaudio & Xming

One application that is very usefull after installing andLinux is the synaptic. It is the application manager of andLinux. You can find this from KDE Menu / open a terminal & type synaptic.

There is a huge collection of linux applications that you can select & install. As i feel the GUI is staright-forward & easy to understand, compared to CygwinX. It is more responsive on my machine with 512 MB of RAM & Dual Core Intel.

It is still in Beta, you can go to the andLinux website to download it. There is a torrent tracker also.

EnJoy ...

Script to comment parts of code in a source file

vikas.nv@gmail.com (Viki) — Sat, 13 Dec 2008 08:41:50 PST

A friend of mine wanted a script to comment parts of code in a source file. We were chatting & i gave him this solution:

me: perl --i.bak -pe 's/^(.*)$/#$1/gi if ($. >=start_line_number && $. <= end_line_number)' filename

Try above command in terminal. Where start_line_number is the line number where u shud start commenting & end_line_number is where u want to stop commenting.

But he wanted that the script should take a function name as input rather than line numbers ... this is what he says: (name not published ;))

myfrend: hey vikas, i need a script to start commenting the codes without using line numbers, but using function name....is it possible?

me: u mean to say u have function names ... & u want to comment the whole functions?

u mean to say u have function names ... & u want to comment the whole functions?

myfrend: yeah

me: it is possible ... if u can get correct start&end boundaries for the functions

myfrend: yeah..how to get it....its difficult

me: how big can the file be .. that is MAX big file size?

myfrend: say around 3000 lines

I was asking a lot of questions about the source files like do they follow coding standards, etc .. & all that. So he suggested what he wanted to do ... i mean the logic ... he said:

myfrend: my logic is : search for function name, then keep a counter to increment/decrement when { or } is found, when counter is zero, exit the prgm

me: this is holds good ... but u have to account for { } occuring on same line ... & account for comments, etc

myfrend: ok

Got some free time to think about it ... & wrote a script for this. This may not be a complete solution, but a good starting point i guess.

Here is the link to the script:

http://docs.google.com/Doc?id=dg2jxmfs_43gbtd2jfq

PS - myfrend :

If you permit I can publish your name ... with a hyperlink possibly to your blog ;)

Get Spiritual ...

vikas.nv@gmail.com (Viki) — Thu, 21 Aug 2008 04:41:45 PDT

Get spiritual ...

Take a look at the photo galleries shared by my good friend in his blog.

http://srivatsashroff.blogspot.com

Notice lot of temples in the galleries, that is all with getting spiritual :)

PS: Obviously, this is post has nothing to do with Perl | ;) |
Viki

Example for use of grep(), map() and readdir()

vikas.nv@gmail.com (Viki) — Tue, 28 Oct 2008 07:49:38 PDT

You should have used readdir() to read contents of a directory. For that you have to open a directory as in:

opendir(DIR, "$directory_path");

After which readdir(DIR) returns an entry or all directory entries depending on the context.

my $file = readdir(DIR); # returns only single entry
my @files = readdir(DIR); # returns all the directory entries

But the values returned will not have the absolute path. For this reason, I use map to prefix the path for each entry ...

my @files = map { "$directory_path/$_" } readdir(DIR);

Now using the code given above all directory entries will have absolute paths.

If you want to grep for a pattern & collect only those entries that match a pattern use grep {}, like this:

my @files = grep { /pattern/ } map { "$directory_path/$_" } readdir(DIR);

In the above example, using grep {} before or after map {} makes a difference.

Sockets ... not the electrical ones ;)

vikas.nv@gmail.com (Viki) — Sat, 16 Aug 2008 07:56:38 PDT

This is my first script that used Sockets. Written around 2 years back, when I was still learning the various uses of Perl. Was very excited to see this work ... Check out the files :

The server script:
http://docs.google.com/Doc?id=dg2jxmfs_42hcxtq3fv
THe client script:
http://docs.google.com/Doc?id=dg2jxmfs_41cvcnk6d5

Comments, questions ... welcome !

When data stored becomes very large ...

vikas.nv@gmail.com (Viki) — Tue, 05 Aug 2008 07:51:11 PDT

There will be situations at which the data you store in memory (variables) is huge, & the script cribbs & might throw "Out of memory" error (happens if the available memory is very low) & exit.

This had happened to me when extracting large amount of data & storing it into a data structure. The actual reason for this was not the data structure itself, but because of an array in which I used to read the entire files' contents (~35MB/file). The issue was solved once I started reading the file line-by-line.

That was just a part of the topic I wanted to write here... Actually, in the process of lowering the memory usage I had used some new (at least, its new for me) techniques of storing numerical data & that's what I want to write here.

This is the data structure that I had used:
%hash = {
"key1" => {
"kkey1" => "vvalue1"
...
},
...
};

So that's the pictorial of my data structure. In this I had to store some numbers that had to be paired (there was a relationship b/w them) & also for these numbers I had to calculate the average & median. The number-pairs found to be larger than 999, so I had to use a glue character b/w then number-pair & store as "kkey1" & store the number of occurrences as the "vvalue1".

Since it was average & median to be calculated... I wrote these two neat subroutines:

sub get_average {
my (%input_cnt) = @_;
my $count = 0;
my $sum = 0;
foreach my $input (keys %input_cnt) {
$sum += ($input*$input_cnt{$input});
}
foreach my $cnt (values %input_cnt) {
$count += $cnt;
}
my $average = $sum/$count;
return $average;
}

sub get_median {
my (%inputs_cnt) = @_;
my @sorted = sort { $a <=> $b } keys %inputs_cnt;
my %sorted_vals_index = (); # sorted_vals_index{delay} => 'low_index:high_index'
my $index = 0;
foreach my $ele (@sorted) {
my $h_i = $index + ($inputs_cnt{$ele}-1);
$sorted_vals_index{$ele} = "$index:$h_i";
$index = $h_i+1;
}
my $mid_index = ($index-1)/2;
foreach my $ele (keys %sorted_vals_index) {
my ($l_i, $h_i) = ($sorted_vals_index{$ele} =~ /(.+?):(.+)/);
if (($mid_index >= $l_i) && ($mid_index <= $h_i)) {
return $ele;
}
}
}

Each subroutine takes a hash, where the key is the number & value is the number of occurrences.

So the memory used became very less, since I avoided storing duplicate number-pairs.

Regular Expressions (more)

vikas.nv@gmail.com (Viki) — Mon, 11 Aug 2008 01:52:03 PDT

There are many more options that can be used with PERL regular expressions.

The different options that I have used are: g,i,s,m,o,x
These options are to be specified like this:
/reg-exp/options
One or more options can be used, but one cant' use s & m simultaneously.

So here are the meanings of these regular expressions ...

g -> Match Globally:
This option can be used when you are trying to match a pattern in the whole string. As in this example :
$you_appeared++ while(/\s+you\s+/g);

i -> Ignore Case:
Use this when case of the pattern does not matter.

s -> Treat Workspace As Single Line:
This option is used when you need to match "\n" with reg-exp special character '.'. Or put simple, you this option when the work space has "\n"'s and you want to match something considering "\n".
my $workspace = "I'am sure you are confused\nYou have to use it to understand it\n";
if ($workspace =~ /use/s) {
print "$&$'\n";
}

m -> Treat workspace as multiple lines. This option differs from above, here '.' does not match a "\n" however, treats workspace as multiple lines.

o -> compile the pattern once. The effect this produces is visible when you use a variable. Since the pattern is compiled only once, the variable will be replaced only once in the pattern - no matter how many times the pattern is used.

x -> add comments inside the regexp, using '#'

Reading ... Processing file paragraph by paragraph

vikas.nv@gmail.com (Viki) — Sat, 09 May 2009 06:27:02 PDT

There are many situations you may want to process a text file paragraph by paragraph ...

One such example was this, I wanted to delete those paragraphs from a text file that had a particular pattern. Like, delete all paragraphs that has text like 'copyright protected by blah blah'.

First thing is to learn how to read a text file paragraph by paragraph, for that we will see how to open a file (named web_extract.txt):
open (FILE, "web_extract.txt") or die "Unable to open web_extract.txt: $!\n";

This is how you can read the opened file line by line:
while(my $line = <FILE> ) {
.... do something ...
}

But to read a file in paragraph mode, you have to reset (zero) the special variable $/, look at the code below:
{
local $/ = '';
@paragraphs = <FILE>;
chomp @paragraphs;
}

So this will read the opened file in paragraph mode.

Don't worry about the block, it is used to localize resetting the $/ variable.

Now the variable @paragraphs has paragraphs as its elements. So you can loop around this variable and push the elements (to @filtered_paragraphs) that do not match your pattern. Then print that new array (@filtered_paragraphs) to the (same/another) file.

Done!!!

To check the lists of file created between X-Y minutes

vikas.nv@gmail.com (Viki) — Mon, 11 Aug 2008 02:10:10 PDT

I have seen many questions on this ->

how to get a list of files that were created/modified between 30 to 60 minutes? or 1 to 2 hours ... etc.

Here is a simple PERL script to do this job:

use strict;
use File::Find;

my $path_to_start = $ARGV[0];
my @files_bet_30_60 = ();
my $thirty_mins_in_secs = 30*60; # change the value of this var to decrease the lower limit
my $sixty_mins_in_secs = 60*60; # change the value if this var to increase the upper limit

finddepth(\&wanted, $path_to_start);
foreach my $file (@files_bet_30_60) {
print "$file\n";
}

sub wanted {
my $file = $File::Find::name;
next unless (-f $file);

my $mtime = (stat($file))[9];
my $time_diff = time - $mtime;
push @files_bet_30_60, $file if (($time_diff > $thirty_mins_in_secs) && ($time_diff < $sixty_mins_in_secs)); } Also following shell command can do this:
find /path/to/files -type f $ -newer min60 -a ! -newer min30 $

Video: PERL tutorial that I found on YouTube

vikas.nv@gmail.com (Viki) — Mon, 11 Aug 2008 01:55:16 PDT

Video Playlist: PERL tutorials that I found on YouTube

vikas.nv@gmail.com (Viki) — Mon, 11 Aug 2008 01:54:43 PDT

PERL script to check if a process is alive

vikas.nv@gmail.com (Viki) — Mon, 11 Aug 2008 02:07:46 PDT

Below i have a perl script that checks if a given process (passed as command-line argument) is alive or not. If the process had died for some reason, it re-invokes it.

Put this in crontab & schedule it to run every 1 hour or so. This will take to keep your ever running processes alive.

The script:

#!/usr/bin/perl

my $process_to_check = $ARGV[0] or die "Usage: $0 \n";

open(PS,"/bin/ps -aef|") || die "Can't Open PS: $!\n";

while() {
chomp;
if (/\Q$process_to_check\E/) { close PS; exit;}
}

close PS;
system("$process_to_check");

Collect files that were created/modified in last 30 to 60 minutes

vikas.nv@gmail.com (Viki) — Mon, 11 Aug 2008 01:57:46 PDT

I actually wrote this script in reply to a post.

This is what it does ...
Given a path (as command-line argument), the script will search through the file system & lists the files that were created/modified in last 30-60 minutes.

The script:

use strict;
use File::Find;

my $path_to_start = $ARGV[0];
my @files_bet_30_60 = ();
my $thirty_mins_in_secs = 30*60;
my $sixty_mins_in_secs = 60*60;

finddepth(\&wanted, $path_to_start);
foreach my $file (@files_bet_30_60) {
print "$file\n";
}

sub wanted {
my $file = $File::Find::name;
next unless (-f $file);

my $mtime = (stat($file))[9];
my $time_diff = time - $mtime;
push @files_bet_30_60, $file if
(($time_diff > $thirty_mins_in_secs) && ($time_diff < $sixty_mins_in_secs));
}

A Little Guide to PERL Regular Expressions

vikas.nv@gmail.com (Viki) — Mon, 11 Aug 2008 01:58:19 PDT

Why RegExp?

Regular Expressions are powerful tools for pattern matching and it is also PERL’s powerful tool.

Starting with Regular Expressions:

Before introducing to Regular Expression, we shall see what is $_ and how it is relevant to regular expressions.
$_ is a PERL special variable which is the default workspace for the regular expression operator.

However, $_ is also the default workspace for many commands in PERL. For example, if you just say print in your script it would print the contents in $_, same with chop, chomp, etc.

Coming back to our topic, we shall start with the pattern match operator of PERL /pattern/

It is as “powerful” as “simple to see”.

/pattern/ can be used if the workspace of regular expression is $_. If you want to match a pattern against a variable then the operator is $variable =~ /pattern/

Let us see some simple regular expressions examples:
1. Match vikas in this line “my name is vikas”:
$variable = “my name is vikas”; # variable to match
$variable =~ /vikas/; # matching the pattern ‘vikas’ in $variable
2. The above example if workspace is $_:
$_ = “my name is vikas”;
/vikas/;

How to use this actually?

These operators can be used with if, while & even foreach constructs. I will explain the usage with an example:

Scenario: [Extracting data by matching a pattern from a log file]

Say you have a log file that saved your chat & it is important to get all your friend’s chat ids. The log is very big … maybe because it logs all sessions in a single file.

With that story told … you decide to use regular expressions to extract that all the chat ids.

Now all that you have to do is to get a pattern from the log, which you will have to match … You open the file & you see this line:

##### Start of chat with xyz@bahoo at 05:00 PM 25-01-08 #####

For every chat above format is repeated. So your pattern can be:
/^\s*#+\s+ Start of chat with\s+(\w+\@\w+)\s+/

This pattern uses many other RegExp anchor characters, repetitions, braces & character classes which I will explain.

So … here are the anchor characters that you can use:
^ Or \A Match beginning of the string/line -- used in my example
$ Or \Z Match end of the string/line
\z End of string in any match mode
\b Match word boundary
\B Match non-word boundary

Here are the repetition characters:
? Zero or one occurrence of the previous item
* Zero or more occurrences of the previous item – used in my example
+ One or more occurrences of the previous item – used in my example
There is more repetition operators, which I do not want to, put up here.

Here are the braces/capturing groups:
(...) Group several characters together for later use or capture as a single unit, and the matched values will be stored in special variables named $1, $2, $3 …

All this said, we will move on to our example & look at the code:

# first open the log file
open (LOG, ”chat. log”) or die “Unable to open chat.log: $!\n”;
my @all_contents = ; # read all contents into an array
close LOG;
foreach my $line (@all_contents) {
if ($line =~ /^\s*#+\s+ Start of chat with\s+(\w+\@\w+)\s+/) {
print ”$1\n”;
}
}

The above script will print all the chat ids that the log contains.
For more about regular expressions visit this link: http://perldoc.perl.org/perlre.html

About foreach

vikas.nv@gmail.com (Viki) — Mon, 11 Aug 2008 01:58:47 PDT

After a long time...
This post is about the foreach loop construct. There is something special about the foreach loop, that I noticed recently. Consider this script:

my @array = (1,2,3,4,5,6,7,8,9,0);
foreach my $ele (@array) {
print "$ele\n";
}

Yes, this simply prints the array.

The specialty of foreach is that it assigns the reference of each element to $ele. This is true even if you use an array reference in foreach. This means that if you modify an element inside foreach the array is affected.

my @array = (1,2,3,4,5,6,7,8,9,0);
foreach my $ele (@array) {
print "$ele\n";
$ele = '' if ($ele == 5);
}

Consider above script & then print all the array elements, you will notice 5 is gone.

I have discussed with the perlmonks, visit this thread for more info:
http://perlmonks.org/?node_id=696953

Always confused with 2>&1

vikas.nv@gmail.com (Viki) — Mon, 11 Aug 2008 01:59:57 PDT

For those confused, I am talking about the stdout & stderr. And redirection operators.

This is more of the commands used in linux terminals.

Consider this command:
tar -cvf filename.tar

If you run this command it prints a lot on stdout (terminal). So if you want to redirect this output to /dev/null you would do that using:
tar -cvf filename.tar 1>/dev/null
(or better)
tar -cvf filename.tar.gz >/dev/null

If you are using tar in a script & want to capture/redirect the stdout of tar in a file, you can use:
tar -cvf filename.tar >tar_stdout.log
(here > symbol is used to redirect)

Now, if there is an error while tarring (say disk full). Then the command will print error on stderr & will exit. So how will you capture the errors, now you want to redirect both stderr & stdout to files.

There are to scenarios here:
1. You want to redirect to two seperate files:
tar -cvf filename.tar 1>tar_stdout.log 2>tar_stderr.log
2. You want to redirect stdout & stderr to a single file:
tar -cvf filename.tar >tar_stds.log 2>&1

Remember always: This applies to all commands:
The arguments are put to a variable (ARGV) of that process, and this variable is accessed as a stack. In the previous example 2>&1 (instruction to the shell to redirect 2 to 1) should be at the end. So that this (2>&1) redirection is arranged first & then 1>tar_stds.log. Good example is:
mkdir dir1/dir2 dir1 --> Valid
mkdir dir1 dir1/dir2 --> Invalid

Redirection operators include:
> (output redirect)
< (input redirect) >> (output redirect but append)

The same redirect operators can be used to redirect I/O to other file numbers OR to files. The distinction is:
tar -cvf file.tar 1>stdout.log -> this acts as redirect to file
tar -cvf file.tar 1>&2 -> this acts as redirect to another file number(notice the &)

PS:-
I use bash

PERL References

vikas.nv@gmail.com (Viki) — Mon, 11 Aug 2008 02:00:33 PDT

Familiar with C pointers?

In C pointers are variables that contains the location of some other piece of data. That is, it can be a machine address.

In similar lines PERL also supports pointer variables called references.

If you do not understand the pointer concept, here is an illustration (I had read this in a book on 8085, long back):
Take a look at the picture above.

The image is self explanatory, cup c1 analogous to a pointer variable has a piece of paper that has the name of another cup (address) that has the juice which is cup c2.

Similarly, in PERL consider c1 is the reference to variable c2. These references can point to a scalar/array/hash/a subroutine.

Syntax:
my $scalar_ref = \$scalar_a;
my $array_ref = \@array_a;
my $hash_ref = \%hash_a;
my $sub_ref = \&find_average;

Here the slash (\) operator is used to create references. References can be anonymous too, that is without any name. Anonymous hashes can be used to create complex data structures such as array of hashes, hashes of hashes, etc

Also references can be created to file handles & type globs.

References are just like other scalar variables. Perl also offers a function ref() to check the type of reference a scalar holds. For example:

$ref_type = ref($scalar_ref);

ref() returns (without double-quotes)
"SCALAR" for scalar refs
"ARRAY" for arrays
"HASH" for hash
"CODE" for subroutine
It returns undef for a scalar variable. For more info on this you can visit: http://perldoc.perl.org/functions/ref.html

And for more info on references, visit: http://perldoc.perl.org/perlreftut.html

About being 'strict' (use strict)

vikas.nv@gmail.com (Viki) — Mon, 11 Aug 2008 02:01:13 PDT

This post is about using the strict pragma.

I always use this pragma in my scripts. It helps you in keeping the script neat & clean.

As you know that by default variables are created & destroyed lexically in PERL, what I mean is by default (if you do not mention "my", "our" or "local") all variables are lexically scoped. This leads actually to errors when you mis-spell the variable names in your script.

For example:

Consider this script, of adding two numbers:
$variable1 = 10;
$variable2 = 20;
$result = $variable1 + $variable2;

This is a small script & may not have mistakes, but look at the bigger picture.

Suppose you made a mistake in the script, only just a typo:
$variable1 = 10;
$variable2 = 20;
$result = $variable1 + $varaible2;

The results are annoying. It seems like there is a logical bug (considering a bigger script), but no there is a typo. This can be caught, using the pragma strict.

Strict is pretty strict, you have to explicitly mentioned the scope of the variables when you declare them. The above script (the one with mistake) will now become:
use strict;
my $variable1 = 10;
my $variable2 = 20;
my $result = $variable1 + $varaible2;

Try running this script, you will get this error:
"Global symbol "$varaible2" requires explicit package name at test_strict.pl line 4.
Execution of test_strict.pl aborted due to compilation errors."

I think you have noticed the error by now.

Using strict pragma affects the subroutines, references & variables. You can even turnoff strict on these, for example:
no strict "vars";

For more details, goto: perldoc.perl.org/strict.html

Bottom line :-
Always 'use strict' and 'no strict' whereever necessary.

This time its a Shell script

vikas.nv@gmail.com (Viki) — Mon, 11 Aug 2008 02:02:03 PDT

This small shell script is to find the sizes of each folder under pwd:

This script will print folder names along with its size. The output is redirected to a temp.txt, you can change the name of the file.

Run this script in bash.

find -type d -maxdepth 1 | while read dir; do
if [ $dir == "." ]; then
continue;
else
du -sh $dir;
fi;
done > temp.txt

Find the host/computer name from its IP & vice versa

vikas.nv@gmail.com (Viki) — Mon, 11 Aug 2008 02:05:22 PDT

Here is a small snippet of PERL code to display a host/computer's name from its IP address.

use Socket;
$iaddr = inet_aton("206.190.60.37");
$name = gethostbyaddr($iaddr, AF_INET);
print "$name\n";

Running this code will give the host name as w2.rc.vip.re4.yahoo.com

Similarly, you can get the IP address of a host/computer whose name is known.
Below is the code for that:

use Socket;
$iaddr = gethostbyname("www.google.co.in");
$addr = inet_ntoa($iaddr);
print "$addr\n";

P S -
Your computer should be connected to the internet, to try these example.

Accessing a file in PERL

vikas.nv@gmail.com (Viki) — Mon, 11 Aug 2008 02:04:17 PDT

This is simplified using open().

Usage: open(FILEHANDLE, accessMode_with_fileName)

FILEHANDLE is like a pointer to the file's contents.
accessMode_with_fileName is actually composed of two i.e.

1. Access Mode: that specifies how you wan to access the file. It can be
a. Read, where you can just read from the file
b. Write, where you can just write to the file
c. Append, where you can write to the end of the file
What you use depends on the requirement. To specify one of these modes you need to use any of these symbols : < (read), > (write) & >> (append). If you open a file without any of these symbols, it will default to read mode.
You can put a '+' in front of the '>' or '<' to indicate that you want both read and write access to the file (I have not used that, pretty much).

2. Filename: The second component is the filename itself. so if you want to open a file (say /home/vik/foo) for reading the open call looks like:

open (FILE, "</home/vik/foo");

or

open (FILE, "/home/vik/foo"); ## NO Access mode symbol is by default read mode

The same example holds good for all other modes, you have to just change the access mode symbol as per requirement.

The return value of open() is undef if it cannot open the file, so this check can be put using 'or', like this:

open (FILE, "</home/vik/foo") or die "Could not open file for reading: $!\n";

Here $! is a special variable that holds the error.

Reading/Writing using the file handle

After opening the file for reading, you can access the files' contents using a scalar variable / an array. Here are the examples:
1. Using a scalar: If you want to read a file line by line, use a scalar variable like this,

my $files_line = <FILE>;

Remember this always, when you read a line from the file the '\n' (newline character) will be at the end of the read contents, so use chomp() to remove any newline characters.
So the example becomes:

my $files_line = <FILE>;
chomp $files_lines;

2. Using an array:

If you want to read all the lines of a file, you use the array. See this example below:

my @files_contents = <FILE>;

With the chomp:

my @files_contents = <FILE>;
chomp(@files_contents);

Writing to a file
Use print to write to a file. See this example:

print FILE "I want to be in a file\n"