TechDiary

Debian/LXDE running on Acer Iconia A500 tab

vikas.nv@gmail.com (Viki) — Wed, 24 Aug 2011 06:22:00 PDT

I got a chance to have this tablet for sometime, and since then I have been hacking it. The first thing to do is most obvious, to root the device. I have reloaded the tablet with Taboonay's ROM (1.1) based on the Android Honeycomb 3.1 stock ROM. It has it's cool features, about which you can read here.

And thanks to lanrat for an article about this.

*** Do this if you fully understand what you are doing***

Now over to getting Debian on to the tablet.

Your device should be rooted. You need some free space on the SD card, about 1 GB is sufficient for this. Secondly you have to prepare a base installer image of Debian, I did this on Ubuntu 11.04. If you do not have a Linux box don't sweat it, get Linux on a pen drive and boot from that. And you need the Android terminal, also set it up such that your $PATH variable has /system/xbin in the beginning. For installing/configuring LXDE you need to have a mouse and keyboard connected to your tablet (I used an old bluetooth keyboard and mouse from Logitech EX100).

Once you are in Ubuntu open a terminal and do the following to create the base image -

sudo -s

apt-get install debootstrap

dd if=/dev/zero of=debian.img seek=1073741824 bs=1 count=1 # I use 1 GB image size **

mke2fs -F debian.img

mkdir debian

mount -o loop debian.img debian/

debootstrap --verbose --arch armel --foreign lenny debian http://ftp.us.debian.org/debian

umount debian/

rm -r debian/

** you can use lesser image size (800 MB) if you do not want to install LXDE.

With that you have a Debian base installer image. Now download this chroot script, which will be used to set up environment to run Debian (save this file as bootdebian). Copy this image and the chroot script to the SD card, such that it lays in

/mnt/external_sd

Now put back the SD card into the tablet and power on. Following commands will make the chroot script

executable in Android terminal and turn it into a command, open Android terminal on the tablet and type -

mount -o remount,rw -t yaffs2 /dev/block/mmcblk0p3 /system

cat /mnt/external_sd/bootdebian > /system/xbin/bootdebian

dos2unix /system/xbin/bootdebian

rm /mnt/external_sd/bootdebian

chmod 777 /system/xbin/bootdebian

With that done, you have a command 'bootdebian' which can be run to set up to run Debian.

After this you need to complete the Debian install, for that run the following commands on the Android terminal -

bootdebian

/debootstrap/debootstrap --second-stage

echo 'deb http://ftp.us.debian.org/debian lenny main' >/etc/apt/sources.list

apt-get autoclean

apt-get update

exit

This completes the Debian install. From now on if you want to work on a real Linux box, you just have to ope n the Android terminal and run 'bootdebian'.

Now, commands for installing LXDE -

bootdebian

apt-get install xdm # this downloads ~30 MB

apt-get install lxde # this downloads ~100 MB, so you can 'take wakelock' the terminal & go for a walk

Once the apt-get completes the installations, it starts configuring the LXDE. This is the time you need your hardware keyboard (if not mouse) when it asks for some settings. Look for the place where it asks for which display manager and select 'xdm'.

Next you need to install the tightvncserver so as to create vnc session. For that, run

apt-get install tightvncserver

Once done with that, you can start the session with these commands -

export USER=root

vncserver -geometry 1280x736 -depth 16

Above command will ask you for setting up passwords. This creates a session on localhost:1, which is a basic desktop. To start LXDE in the session, you have to instruct the vncserver using the xstartup script. For doing this, run the following commands -

vncserver -kill :1 # first kill the running session

vi /root/.vnc/xstartup

This opens the vi editor loaded with the vncserver's start up script. Edit this script such that it looks something like this -

#!/bin/sh

xrdb $HOME/.Xresources

xsetroot -solid grey

#x-terminal-emulator -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" &

#x-window-manager &

#/etc/X11/Xsession

startlxde

Save the file and close. Start the VNC session.

To view the session, you can either use the AndroidVNC application on your tablet or any VNC viewer on your PC.

This ends!

Script to set the compatibility mode of executable files on Windows 7

vikas.nv@gmail.com (Viki) — Thu, 28 Apr 2011 04:21:52 PDT

I wanted to install the portable Ubuntu on my Windows machine, and so I downloaded the executable from the sourceforge project site and installed it.

It did not work, I initially thought I have to change the compatibility mode of the executables and set it to Windows XP SP2. There were too many files for which I had to change, so I came up with a script.

This script takes these options:
-p /--path for which you pass the path (all executable files under this path will be considered)
-m /--mode for which you pass the compatibility mode (list of valid modes given at the end of the post)
-a /--runasadmin which is a toggle option which enables/disables the "Run as Administrator" option (default: disabled)
-v /--verbose which is a toggle option which print more information (default: disabled)

The script did work :) ... but the portable Ubuntu did not :(

You can download the script from here

Don't worry to run this script on the same directory multiple times, the Windows registry takes care of duplicates by adding a file as a single entry.

List of compatibility modes (that go with --mode/ -m option):

"WIN95": "Windows 95 compatibilty mode",
"WIN98": "Windows Millennium or 98 compatibility mode",
"NT4SP5": "NT 4.0 Service Pack 5 compatibility mode",
"WIN2000": "Windows 2K compatibility mode",
"WINXPSP2": "Windows XP Service Pack 2 compatibility mode",
"WINSRV03SP1": "Windows Server 2003 SP 1 compatibility mode"

EDIT:
How to run this script on Windows... ?
Firstly, install python on your Windows machine. For installation you should download an appropriate
setup file for your Windows machine from this link, then double click the setup file and follow the on screen instructions.
Secondly, download the script from here. And here is how to run it:
Open a command terminal in Windows by hitting Windows+R keys and entering "cmd" (no quotes)
Then, navigate to the directory to which you downloaded the script (using 'cd'), when you are there
type these commands:
python compatibility_change.py --path "c:\\path\\to\\executables" --mode WINXPSP2

IMDB API

vikas.nv@gmail.com (Viki) — Wed, 05 Oct 2011 06:03:50 PDT

[UPDATE: links are update]

Here I present the module which provides APIs to get information about movies from IMDB.

This is about a Python implementation of IMDB API.

You don't have to download the IMDB's databases locally to use these APIs.

I have used Peteris Krumins' xgoogle library along with the BeautifulSoup to develop this in Python.

You may have to install the BeautifulSoup module (download from here) and xgoogle library (from here)

And here is the link for IMDB API

Here is a little bit of description about the usage:

    from pIMDB import pIMDB
    if len(sys.argv) < 2:
        print "Usage: %s " % sys.argv[0]
        sys.exit(1)
    movie = re.sub(r'[^a-zA-Z0-9\']+', ' ', sys.argv[1])
    imdb = pIMDB(movie)
    imdb.parse_imdb_page()
    print "imdb link:\n\t", imdb.imdb_link
    print "\nmovie name:\n\t", imdb.movie
    print "\nmovie rating:\n\t%s/%s" % (imdb.rating, imdb.outof)
    print "\nmovie details:\n\t%s" % '\n\t'.join(imdb.details)
    print "\nmovie storyline:\n\t%s" % imdb.storyline

It also has an API to download the poster from the IMDB website, you can use it as here:
    imdb.download_poster('/tmp/destination.jpg')

That's the end of the story :)

PS: Write a comment on how you used this module

Configure Gmail on your N900

vikas.nv@gmail.com (Viki) — Thu, 18 Nov 2010 07:04:14 PST

Here is a little guide to setup Gmail on your N900 using the Email application that comes by default on N900.

I am writing this because I could not find a working solution when I tried Google-ing "how to configure n900 for gmail".

Basically, I just helped my N900 to resolve imap.gmail.com and smtp.gmail.com
So here goes the steps...

1. Connect to the internet on your PC/laptop and ping imap.gmail.com & smtp.gmail.com. Note down the IP addresses of both. Here is a screen shot of the command prompt

2. Now on your N900 launch the Terminal and gain root access (if you don't know what I am telling, read this Rootsh)
3. Edit the file /etc/hosts (I use vi to edit)
4. Add following two lines to the file (do not delete any lines already existing in the file)
72.14.213.109 imap.gmail.com
74.125.53.109 smtp.gmail.com
(Replace the IP addresses in the above lines to what you noted down in the 1st step)
5. Save the file and exit the editor
6. Launch the Email application on N900. Follow the wizard that it shows by selecting Gmail in the list of providers that it shows.
7. Enter your user name and password and go till you click finish.

The account you set up should work fine. If not, tap on Edit settings for the account and make sure the settings are as shown in the screen shot below

Frozen Bubble on N900 :) !!!

vikas.nv@gmail.com (Viki) — Wed, 06 Oct 2010 08:04:42 PDT

Have you played this game, frozen bubble? Its the most addictive game I have ever played ... next to some games like bejeweled. I had this game on my previous phone Nokia 5800. So I was searching for this game ... hoping some one would have ported it to N900 (currently I own it). WOW I found it here http://maemo.org/packages/view/frozen-bubble/

It is still in development repository but I have used it and it is working perfectly. It has great music and great graphics this time, and its called frozen bubble 2.

Enjoy!

how to re-organize menu icons on N900 (with catorise)

vikas.nv@gmail.com (Viki) — Sun, 29 Aug 2010 09:23:06 PDT

You can get the icons in the menu of your N900 organized into folders using an application called catorise.
This app automatically groups the menu icons and puts them into different folders of meaningful names.
But sometimes you feel ... hey this icon should be in the games folder why is it put into the ovi folder??? I felt this when i saw bounce in ovi folder after installing catorise.
And so i launched the xterm and started looking for some kind of a config file or something where i can change the grouping. Went into the /opt folder to check out the installed apps ... there was a folder catorise cd into it and ls ... there was something very obviously named file 'menu'. But i need to be root to edit this, became root and opened this file using vi. It is a simple config file with this syntax ->
app name: folder name
I changed ->
bounce: ovi
to this ->
bounce: games
Saved it and closed it.
Then ran the perl script named catorise in pwd.
This changed the grouping, bounce appears in games than in ovi :)

PCron ---- portable cron :)

vikas.nv@gmail.com (Viki) — Thu, 25 Mar 2010 05:02:17 PDT

PCron

Cron for Windows & ... :)

Cron is basically a job scheduler available in Linux and its variants/flavors.
I have tried to implement the functionality of Cron in Python so that it can be used in Windows and ....

You should have Python 2.5 + installed to use this.

Input for PCron is a plain text file (lets' call it pcrontab) with syntax/format similar to *nix crontab. pcrontab can have formats such as -> */12 and 12-20
Here is an example of a pcrontab:
*/5 * * * * F:\Program Files (x86)\K-Lite Codec Pack\Media Player Classic\mpc-hc.exe
57-59 18 * * * F:\Program Files (x86)\Calc.exe

In above example, first line instructs PCron to start the program mpc-hc.exe every 5 minutes.
And the second line instructs PCron to start the program Calc.exe on 18:57, 18:58, 18:59

To understand more on the syntax of pcrontab ... look at the picture below which explains about crontab ->
Usage:
python pcron.py --taskfile
Examples:
python pcron.py --taskfile c:\\pcrontab.txt
python pcron.py --taskfile /home/vikas/pcrontab
Currently it writes out logs to stdout, so you have to redirect it ... to say /dev/null or a file

Limitations:
Each time you add an entry into the pcrontab, you have to restart the script or wait until the script reloads the pcrontab (it reloads in every 5 minutes ~)
Does not support entire format/syntax combinations as compared to crontab

More things to come:
An editor for pcrontab ... that would detect changes in pcrontab and initiates automatic reload in pcron
Support for a timeout field in pcrontab, which enables us to run a program for a certain period of time

You can download the scripts from here:
TimedPopenNT
Scheduler
pcron

On Windows you can generate a .exe file using py2exe and use that as standalone instead of the script.

Extract or Reap or Download Pictures from HTML Web Pages

vikas.nv@gmail.com (Viki) — Thu, 13 Aug 2009 04:08:13 PDT

I was exploring python and the HTML parsers that it has. I hit a page that explained about the SGMLParser and its uses.

Using this package I had written a simple script that could extract pictures from simple HTML pages. While experimenting it with many websites I noticed some websites did not allow my script to reap or extract pictures.

Then I found that it is because of the User-Agent string in the HTTP request. Some websites do not respond properly to HTTP requests that have unknown User-Agent string. I then used the urllib's FancyURLOpener to change the User-Agent to Firefox.

Now that the script work on most of the websites, i would like to post the code here. Feel free to modify to make it more robust and reliable, post a comment or a link to your modified script.

This script search pictures from web pages and download them for you. All you need to do is
get the URL of the web page from which you want the pictures to be downloaded and pass it as
an argument to this script. Here is how to use it:
0. Note that python is installed on your system to run this script (Windows/Linux/Solaris ... OS independent;)
1. Download this script and save it as gal_ext.py
2. Open a command-prompt (if Windows) or Terminal (if you use any *nix OS)
3. Type "python gal_ext.py 'http://URL'
4. Here note that the 'http://' is mandatory
Using this script you can avoid downloading images with a web browser, which takes a long time and is annoying to right click on every image and save them. This way you can download picture from web faster and easier. Since this is a script it is open for modifications and is free to use. But if you modify, make a note to send it to me ;). Thanks

The code is here (or a link for you to download):


from sgmllib import SGMLParser
import sys
import os
import re
import urllib
from urllib import FancyURLopener
from urlparse import urlparse

class URLLister(SGMLParser):
def reset(self):
SGMLParser.reset(self)
self.urls = []

def start_a(self, attrs):
href = [v for k, v in attrs if k=='href']
if href:
self.urls.extend(href)

class MyURLOpener(FancyURLopener):
version = 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.6) Gecko/2009011913 Firefox/3.0.6'

def url_mapper(var):
regexp = re.compile('.+?(http://.+&?)', re.I)
values = urlparse(var)
if len(values) >= 5 and values[4] != '':
obj = regexp.search(values[4])
if obj:
values = urlparse(obj.group(1))
else:
if values[4] != '': print('Could not get real url from:' + values[4])
site = re.sub('[^0-9a-zA-Z_/]', '_', values[1])
location_file = re.findall('(.+)/(.+)', values[2])
try:
if len(location_file[0]) != 2:
return ''
except IndexError:
return ''
location = location_file[0][0]
file = location_file[0][1]
location = re.sub('\.{2,}/', '/', location)
location = re.sub('[^0-9a-zA-Z_/]', '_', location)
location = location + '/' + file
return site + location

if __name__ == '__main__':
url_opener = MyURLOpener()
parser = URLLister()
sys.argv[1] = urllib.unquote(sys.argv[1])
try:
usock = url_opener.open(sys.argv[1])
except IOError:
print 'skipping ' + sys.argv[1]
sys.exit(0)
parser.feed(usock.read())
parser.close()
usock.close()
count = 0
urlfs = urlparse(sys.argv[1])
print "Url fs: ", urlfs
parent = urlfs[1]
try:
parent += urlfs[2]
parent = re.search('(.+)/.+',parent).group(1)
except IndexError:
parent = urlfs[1]
for img_url in parser.urls:
if re.search('\.jpe?g$', img_url):
print "looking at: " + img_url
if not re.match('^http://', img_url):
img_url = 'http://' + parent + '/' + img_url
loc = url_mapper(img_url)
if os.path.exists(loc): continue
url_opener1 = MyURLOpener()
retrieve = url_opener1.retrieve
if loc == '':
print "\turl_mapper returned NULL for " + img_url
continue
loc_dir = re.match('(.+)/.+', loc).group(1)
try:
print('\ttrying to save ' + img_url)
if not os.path.exists(loc_dir):
 os.makedirs(loc_dir)
retrieve(img_url, loc)
except IOError:
print('\tSkipping saving ' + img_url)
continue
count += 1
print "\tImg fetched: ",count

Installing andLinux on Windows Vista

vikas.nv@gmail.com (Viki) — Fri, 29 May 2009 08:28:37 PDT

After downloading the andLinux installer (KDE version), launch the installer as Administrator. And follow the steps with default options set.

Reboot/Restart machine after installation.

Click on Unblock if the Firewall prompts.

After the reboot, you click on Konsole (any) launcher on KDE Menu from system tray, you are likely to get this error "could not launch ... could not connect to 192.168.11.150"

So this guide explains how to fix this issue, for the launchers to work on Windows Vista.

First thing to do is to go to the "Network and Sharing Center" from Control Panel. You can see the "customize" and "view status" links on "Unidentified Network". Click on the "view status", click on properties and double click "Internet Protocol Version 4". That should open up a dialog where you can change the IP and the subnet mask, change the IP to 192.168.10.1 and subnet mask to 255.255.254.0. Click on OK OK ....

Secondly, open the "andlinux Console" get sudo access and edit the file /etc/network/interfaces.
After edit the file should have IP of 192.168.10.150 and the mask 255.255.254.0 under eth1

Thirdly edit file /etc/profile to have IPs set to 192.168.10.150

Fourthly navigate to the andlinux installation folder goto Xming folder edit X0.hosts and add the following IPs 192.168.10.150 and 192.168.10.1

Lastly few registry changes, regedit and navigate to \\HKEY_LOCAL_MACHINE\\SOFTWARE\\andLinux\\Launcher
Here you find to keys "IP" and "Port" set them to 192.168.10.150 and 2081 respectively.
(Remember to click on Decimal while editing Port value) And close the regedit.

Finally kill KDE Menu, Xming. Restart andLinux service, and then start Xming and KDE Menu.
Now the launchers work fine.

If clicking on a launcher does not start, you open the andLinux Console application and execute:
"/etc/init.d/networking restart". After this any launcher should work fine.

!Enjoy!

Parsing and Storing parts or paragraphs of log file

vikas.nv@gmail.com (Viki) — Sat, 09 May 2009 07:12:07 PDT

At many times we need to parse and store parts of a log file. Suppose an error log, you would like to parse all errors. If the errors span only one line, you could use a grep.

But if the errors span multiple lines ... you need to put a little logic into your script. There are many ways to do this, but this script that i post here is more readable and can be modified to other's requirement.

Link to the code
Link to a sample input

In the example script i am parsing an error log (named 'input.txt'). The script stores all lines that start with "SQL0204N" till it finds an empty line. Likewise it extracts all errors, even if the error messages span multiple lines.

Perl script inside a DOS Batch file

vikas.nv@gmail.com (Viki) — Wed, 01 Apr 2009 03:46:03 PDT

Whenever I wanted to invoke a utility passing a file as an argument I used to create a shortcut of the utility in the Send To folder, as described here :

How to Add Items to the "Send To" Menu

But how to have a Perl script in the "Send To" menu??

I got an idea when browsing through this page :

DOS Batch - Interfacing non DOS Software

I created a batch script as described in the page. Then modified the part of the batch file where the Perl script actually starts.

I wanted to have a utility that can encrypt/decrypt a text file. A basic encryption was sufficient, so considered the Caesar cipher.

I completed the script and added it to the Send To menu.
Link to the script

A small modification to the above script to handle binary files. Here is the link.

andLinux --- Run Linux from Windows

vikas.nv@gmail.com (Viki) — Sat, 13 Dec 2008 09:28:55 PST

Have you ever had a feeling of using Linux when you are working on Windows???

Felt like using those very useful Linux commands fromWindows???

This is possible ...

Everybody knows that you can have a Dual-Boot on your PC with Linux & Windows. In which you have to restart the PC to access the other OS.

But there are several other ways to have Other OS run right from Windows, that is for example use Linux when you have logged on to Windows... Yes... Virtual Machines (also popular as VM) is one of them. But running a VM requires that your hardware to be really good.

There is one more option, we now have emulated PC software like andLinux.

There are many other softwares other than andLinux, such as CygwinX, MKS Toolkit, etc

But I found andLinux to be really very handy, straight-forward & useful.

Installing andLinux is as simple as installing other software ... clikcing Next, Next, Next ... & finnaly Finish :) there is a very elaborate tutorial on its web site

However if you are the curious cat... you can also fiddle around & do the installation without the tutorial.

The home page of andLinux says "andLinux is not just for development and runs almost all Linux applications without modification". I have installed a lot many applications to andLinux, and its really true.

After installation you can see the icons of andLinux in system tray. You may see these icons :

KDE menu (andLinux)

Xming (andLinux)

While installing say you choose E: drive as destination. It gets installled in a folder E:\andLinux.

It will create some folders & files in E:\andLinux, I will list some folders & files with thier purposes:

1. settings.txt, a plain-text file that will have some paths necessary for andLinux to work properly

2. srvstart.bat, batch file, that has the command to start the andLinux service

3. srvstop.bat, batch file, has the command to stop andLinux service

4. startup.bat, again a batch file used to start the andLinux as deamon

5. Drives, directory, where the andLinux's filesystem is stored as files (base.drv, mine is 4GB in size and swap.drv)

6. Launcher, directory, where all the executables can be seen. A good place to check out what all linux apps you get after installing andLinux.

7. There are other directories as well, netdriver, pulseaudio & Xming

One application that is very usefull after installing andLinux is the synaptic. It is the application manager of andLinux. You can find this from KDE Menu / open a terminal & type synaptic.

There is a huge collection of linux applications that you can select & install. As i feel the GUI is staright-forward & easy to understand, compared to CygwinX. It is more responsive on my machine with 512 MB of RAM & Dual Core Intel.

It is still in Beta, you can go to the andLinux website to download it. There is a torrent tracker also.

EnJoy ...

Script to comment parts of code in a source file

vikas.nv@gmail.com (Viki) — Sat, 13 Dec 2008 08:41:50 PST

A friend of mine wanted a script to comment parts of code in a source file. We were chatting & i gave him this solution:

me: perl --i.bak -pe 's/^(.*)$/#$1/gi if ($. >=start_line_number && $. <= end_line_number)' filename

Try above command in terminal. Where start_line_number is the line number where u shud start commenting & end_line_number is where u want to stop commenting.

But he wanted that the script should take a function name as input rather than line numbers ... this is what he says: (name not published ;))

myfrend: hey vikas, i need a script to start commenting the codes without using line numbers, but using function name....is it possible?

me: u mean to say u have function names ... & u want to comment the whole functions?

u mean to say u have function names ... & u want to comment the whole functions?

myfrend: yeah

me: it is possible ... if u can get correct start&end boundaries for the functions

myfrend: yeah..how to get it....its difficult

me: how big can the file be .. that is MAX big file size?

myfrend: say around 3000 lines

I was asking a lot of questions about the source files like do they follow coding standards, etc .. & all that. So he suggested what he wanted to do ... i mean the logic ... he said:

myfrend: my logic is : search for function name, then keep a counter to increment/decrement when { or } is found, when counter is zero, exit the prgm

me: this is holds good ... but u have to account for { } occuring on same line ... & account for comments, etc

myfrend: ok

Got some free time to think about it ... & wrote a script for this. This may not be a complete solution, but a good starting point i guess.

Here is the link to the script:

http://docs.google.com/Doc?id=dg2jxmfs_43gbtd2jfq

PS - myfrend :

If you permit I can publish your name ... with a hyperlink possibly to your blog ;)

Get Spiritual ...

vikas.nv@gmail.com (Viki) — Thu, 21 Aug 2008 04:41:45 PDT

Get spiritual ...

Take a look at the photo galleries shared by my good friend in his blog.

http://srivatsashroff.blogspot.com

Notice lot of temples in the galleries, that is all with getting spiritual :)

PS: Obviously, this is post has nothing to do with Perl | ;) |
Viki

Example for use of grep(), map() and readdir()

vikas.nv@gmail.com (Viki) — Tue, 28 Oct 2008 07:49:38 PDT

You should have used readdir() to read contents of a directory. For that you have to open a directory as in:

opendir(DIR, "$directory_path");

After which readdir(DIR) returns an entry or all directory entries depending on the context.

my $file = readdir(DIR); # returns only single entry
my @files = readdir(DIR); # returns all the directory entries

But the values returned will not have the absolute path. For this reason, I use map to prefix the path for each entry ...

my @files = map { "$directory_path/$_" } readdir(DIR);

Now using the code given above all directory entries will have absolute paths.

If you want to grep for a pattern & collect only those entries that match a pattern use grep {}, like this:

my @files = grep { /pattern/ } map { "$directory_path/$_" } readdir(DIR);

In the above example, using grep {} before or after map {} makes a difference.

Sockets ... not the electrical ones ;)

vikas.nv@gmail.com (Viki) — Sat, 16 Aug 2008 07:56:38 PDT

This is my first script that used Sockets. Written around 2 years back, when I was still learning the various uses of Perl. Was very excited to see this work ... Check out the files :

The server script:
http://docs.google.com/Doc?id=dg2jxmfs_42hcxtq3fv
THe client script:
http://docs.google.com/Doc?id=dg2jxmfs_41cvcnk6d5

Comments, questions ... welcome !

When data stored becomes very large ...

vikas.nv@gmail.com (Viki) — Tue, 05 Aug 2008 07:51:11 PDT

There will be situations at which the data you store in memory (variables) is huge, & the script cribbs & might throw "Out of memory" error (happens if the available memory is very low) & exit.

This had happened to me when extracting large amount of data & storing it into a data structure. The actual reason for this was not the data structure itself, but because of an array in which I used to read the entire files' contents (~35MB/file). The issue was solved once I started reading the file line-by-line.

That was just a part of the topic I wanted to write here... Actually, in the process of lowering the memory usage I had used some new (at least, its new for me) techniques of storing numerical data & that's what I want to write here.

This is the data structure that I had used:
%hash = {
"key1" => {
"kkey1" => "vvalue1"
...
},
...
};

So that's the pictorial of my data structure. In this I had to store some numbers that had to be paired (there was a relationship b/w them) & also for these numbers I had to calculate the average & median. The number-pairs found to be larger than 999, so I had to use a glue character b/w then number-pair & store as "kkey1" & store the number of occurrences as the "vvalue1".

Since it was average & median to be calculated... I wrote these two neat subroutines:

sub get_average {
my (%input_cnt) = @_;
my $count = 0;
my $sum = 0;
foreach my $input (keys %input_cnt) {
$sum += ($input*$input_cnt{$input});
}
foreach my $cnt (values %input_cnt) {
$count += $cnt;
}
my $average = $sum/$count;
return $average;
}

sub get_median {
my (%inputs_cnt) = @_;
my @sorted = sort { $a <=> $b } keys %inputs_cnt;
my %sorted_vals_index = (); # sorted_vals_index{delay} => 'low_index:high_index'
my $index = 0;
foreach my $ele (@sorted) {
my $h_i = $index + ($inputs_cnt{$ele}-1);
$sorted_vals_index{$ele} = "$index:$h_i";
$index = $h_i+1;
}
my $mid_index = ($index-1)/2;
foreach my $ele (keys %sorted_vals_index) {
my ($l_i, $h_i) = ($sorted_vals_index{$ele} =~ /(.+?):(.+)/);
if (($mid_index >= $l_i) && ($mid_index <= $h_i)) {
return $ele;
}
}
}

Each subroutine takes a hash, where the key is the number & value is the number of occurrences.

So the memory used became very less, since I avoided storing duplicate number-pairs.

Regular Expressions (more)

vikas.nv@gmail.com (Viki) — Mon, 11 Aug 2008 01:52:03 PDT

There are many more options that can be used with PERL regular expressions.

The different options that I have used are: g,i,s,m,o,x
These options are to be specified like this:
/reg-exp/options
One or more options can be used, but one cant' use s & m simultaneously.

So here are the meanings of these regular expressions ...

g -> Match Globally:
This option can be used when you are trying to match a pattern in the whole string. As in this example :
$you_appeared++ while(/\s+you\s+/g);

i -> Ignore Case:
Use this when case of the pattern does not matter.

s -> Treat Workspace As Single Line:
This option is used when you need to match "\n" with reg-exp special character '.'. Or put simple, you this option when the work space has "\n"'s and you want to match something considering "\n".
my $workspace = "I'am sure you are confused\nYou have to use it to understand it\n";
if ($workspace =~ /use/s) {
print "$&$'\n";
}

m -> Treat workspace as multiple lines. This option differs from above, here '.' does not match a "\n" however, treats workspace as multiple lines.

o -> compile the pattern once. The effect this produces is visible when you use a variable. Since the pattern is compiled only once, the variable will be replaced only once in the pattern - no matter how many times the pattern is used.

x -> add comments inside the regexp, using '#'

Reading ... Processing file paragraph by paragraph

vikas.nv@gmail.com (Viki) — Sat, 09 May 2009 06:27:02 PDT

There are many situations you may want to process a text file paragraph by paragraph ...

One such example was this, I wanted to delete those paragraphs from a text file that had a particular pattern. Like, delete all paragraphs that has text like 'copyright protected by blah blah'.

First thing is to learn how to read a text file paragraph by paragraph, for that we will see how to open a file (named web_extract.txt):
open (FILE, "web_extract.txt") or die "Unable to open web_extract.txt: $!\n";

This is how you can read the opened file line by line:
while(my $line = <FILE> ) {
.... do something ...
}

But to read a file in paragraph mode, you have to reset (zero) the special variable $/, look at the code below:
{
local $/ = '';
@paragraphs = <FILE>;
chomp @paragraphs;
}

So this will read the opened file in paragraph mode.

Don't worry about the block, it is used to localize resetting the $/ variable.

Now the variable @paragraphs has paragraphs as its elements. So you can loop around this variable and push the elements (to @filtered_paragraphs) that do not match your pattern. Then print that new array (@filtered_paragraphs) to the (same/another) file.

Done!!!

To check the lists of file created between X-Y minutes

vikas.nv@gmail.com (Viki) — Mon, 11 Aug 2008 02:10:10 PDT

I have seen many questions on this ->

how to get a list of files that were created/modified between 30 to 60 minutes? or 1 to 2 hours ... etc.

Here is a simple PERL script to do this job:

use strict;
use File::Find;

my $path_to_start = $ARGV[0];
my @files_bet_30_60 = ();
my $thirty_mins_in_secs = 30*60; # change the value of this var to decrease the lower limit
my $sixty_mins_in_secs = 60*60; # change the value if this var to increase the upper limit

finddepth(\&wanted, $path_to_start);
foreach my $file (@files_bet_30_60) {
print "$file\n";
}

sub wanted {
my $file = $File::Find::name;
next unless (-f $file);

my $mtime = (stat($file))[9];
my $time_diff = time - $mtime;
push @files_bet_30_60, $file if (($time_diff > $thirty_mins_in_secs) && ($time_diff < $sixty_mins_in_secs)); } Also following shell command can do this:
find /path/to/files -type f $ -newer min60 -a ! -newer min30 $

Video: PERL tutorial that I found on YouTube

vikas.nv@gmail.com (Viki) — Mon, 11 Aug 2008 01:55:16 PDT

Video Playlist: PERL tutorials that I found on YouTube

vikas.nv@gmail.com (Viki) — Mon, 11 Aug 2008 01:54:43 PDT

PERL script to check if a process is alive

vikas.nv@gmail.com (Viki) — Mon, 11 Aug 2008 02:07:46 PDT

Below i have a perl script that checks if a given process (passed as command-line argument) is alive or not. If the process had died for some reason, it re-invokes it.

Put this in crontab & schedule it to run every 1 hour or so. This will take to keep your ever running processes alive.

The script:

#!/usr/bin/perl

my $process_to_check = $ARGV[0] or die "Usage: $0 \n";

open(PS,"/bin/ps -aef|") || die "Can't Open PS: $!\n";

while() {
chomp;
if (/\Q$process_to_check\E/) { close PS; exit;}
}

close PS;
system("$process_to_check");

Collect files that were created/modified in last 30 to 60 minutes

vikas.nv@gmail.com (Viki) — Mon, 11 Aug 2008 01:57:46 PDT

I actually wrote this script in reply to a post.

This is what it does ...
Given a path (as command-line argument), the script will search through the file system & lists the files that were created/modified in last 30-60 minutes.

The script:

use strict;
use File::Find;

my $path_to_start = $ARGV[0];
my @files_bet_30_60 = ();
my $thirty_mins_in_secs = 30*60;
my $sixty_mins_in_secs = 60*60;

finddepth(\&wanted, $path_to_start);
foreach my $file (@files_bet_30_60) {
print "$file\n";
}

sub wanted {
my $file = $File::Find::name;
next unless (-f $file);

my $mtime = (stat($file))[9];
my $time_diff = time - $mtime;
push @files_bet_30_60, $file if
(($time_diff > $thirty_mins_in_secs) && ($time_diff < $sixty_mins_in_secs));
}

A Little Guide to PERL Regular Expressions

vikas.nv@gmail.com (Viki) — Mon, 11 Aug 2008 01:58:19 PDT

Why RegExp?

Regular Expressions are powerful tools for pattern matching and it is also PERL’s powerful tool.

Starting with Regular Expressions:

Before introducing to Regular Expression, we shall see what is $_ and how it is relevant to regular expressions.
$_ is a PERL special variable which is the default workspace for the regular expression operator.

However, $_ is also the default workspace for many commands in PERL. For example, if you just say print in your script it would print the contents in $_, same with chop, chomp, etc.

Coming back to our topic, we shall start with the pattern match operator of PERL /pattern/

It is as “powerful” as “simple to see”.

/pattern/ can be used if the workspace of regular expression is $_. If you want to match a pattern against a variable then the operator is $variable =~ /pattern/

Let us see some simple regular expressions examples:
1. Match vikas in this line “my name is vikas”:
$variable = “my name is vikas”; # variable to match
$variable =~ /vikas/; # matching the pattern ‘vikas’ in $variable
2. The above example if workspace is $_:
$_ = “my name is vikas”;
/vikas/;

How to use this actually?

These operators can be used with if, while & even foreach constructs. I will explain the usage with an example:

Scenario: [Extracting data by matching a pattern from a log file]

Say you have a log file that saved your chat & it is important to get all your friend’s chat ids. The log is very big … maybe because it logs all sessions in a single file.

With that story told … you decide to use regular expressions to extract that all the chat ids.

Now all that you have to do is to get a pattern from the log, which you will have to match … You open the file & you see this line:

##### Start of chat with xyz@bahoo at 05:00 PM 25-01-08 #####

For every chat above format is repeated. So your pattern can be:
/^\s*#+\s+ Start of chat with\s+(\w+\@\w+)\s+/

This pattern uses many other RegExp anchor characters, repetitions, braces & character classes which I will explain.

So … here are the anchor characters that you can use:
^ Or \A Match beginning of the string/line -- used in my example
$ Or \Z Match end of the string/line
\z End of string in any match mode
\b Match word boundary
\B Match non-word boundary

Here are the repetition characters:
? Zero or one occurrence of the previous item
* Zero or more occurrences of the previous item – used in my example
+ One or more occurrences of the previous item – used in my example
There is more repetition operators, which I do not want to, put up here.

Here are the braces/capturing groups:
(...) Group several characters together for later use or capture as a single unit, and the matched values will be stored in special variables named $1, $2, $3 …

All this said, we will move on to our example & look at the code:

# first open the log file
open (LOG, ”chat. log”) or die “Unable to open chat.log: $!\n”;
my @all_contents = ; # read all contents into an array
close LOG;
foreach my $line (@all_contents) {
if ($line =~ /^\s*#+\s+ Start of chat with\s+(\w+\@\w+)\s+/) {
print ”$1\n”;
}
}

The above script will print all the chat ids that the log contains.
For more about regular expressions visit this link: http://perldoc.perl.org/perlre.html