TechDiary

Quotes

vikas.nv@gmail.com (Viki) — Mon, 22 Feb 2016 21:08:00 +0530

“You are not your idea, and if you identify too closely with your ideas, you will take offense when they are challenged.”
Catmull, E. and Wallace, A. (2014) Creativity, Inc.: Overcoming the Unseen Forces That Stand in the Way of True Inspiration

“This is one of the most important lessons of the scientific method: if you cannot fail, you cannot learn.”
Ries, E. (2011). The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses.

“The trick to having good ideas is not to sit around in glorious isolation and try to think big thoughts. The trick is to get more parts on the table.”
Johnson, S. (2010). Where Good Ideas Come From: The Natural History of Innovation

“Failure isn’t a necessary evil. In fact, it isn’t evil at all. It is a necessary consequence of doing something new.”
Catmull, E. and Wallace, A. (2014). Creativity, Inc.: Overcoming the Unseen Forces That Stand in the Way of True Inspiration

ElasticSearch -- Why Why its still RED

vikas.nv@gmail.com (Viki) — Mon, 22 Feb 2016 21:07:00 +0530

RED: Damnit. Some or all of (primary) shards are not ready.

YELLOW: Elasticsearch has allocated all of the primary shards, but some/all of the replicas have not been allocated.

GREEN: Great. Your cluster is fully operational. Elasticsearch is able to allocate all shards and replicas to machines within the cluster.

* shards not ready: not enuf file descriptors/ memory/ disk space
* when a node is restarted, the bigdesk shows no of docs, size, etc which keeps increasing & when that gets stable is when the status goes green
* you can keep backup nodes, when one goes down u can up another but should have the same node name

Elastic Search -- Notes on Restart nodes

vikas.nv@gmail.com (Viki) — Mon, 22 Feb 2016 21:05:00 +0530

Steps to shutdown & restart Elastic Search:
1. Disable all crons
2. Shutdown the non-master nodes first one by one
3. Shutdown master
4. Start the master by enabling its cron
5. Wait for the master to come up & show "red" in bigdesk and let it load approx number of docs
6. Start the data node one by one, wait until the bigdesk shows "yellow" & wait until the data nodes have approx number of docs
7. Wait until all shards are "started" in http://xyz:9200/_cat/shards
8. Start the non-data nodes now

Perl dancer app.pl -- auto reload (not using the Dancer's feature auto_reload)

vikas.nv@gmail.com (Viki) — Thu, 29 Oct 2015 14:08:00 +0530

Here is a script that can be added as a cron -
To kill and start the Dancer's app.pl, whenever there are changes to files in lib directory of your app.

The script relies on the modified times of all files residing in the lib folder of your Dancer's app.
If there is a change in the mtime i.e. (stat(file))[9] of any of your files, it kills the Dancer app and starts again.

In the script below change the [perl-path] to your Perl installation location. And "/home/vikas/dancer/esui/bin" to your Dancer's bin location.

----------------------------------------------script--------------------------------------------------

#![perl-path]

use strict;
use Digest::MD5 qw(md5_hex);

my $appdir = "/home/vikas/dancer/esui/bin";

if (-f "$appdir/iam.running") {
my $old_mtime_md5 = '';
open(F, "$appdir/last.fstat.md5") and do {
chomp($old_mtime_md5 = );
close F;
};
my $new_mtime_md5 = getMtimeMD5();
if ($new_mtime_md5 ne $old_mtime_md5) {
# restart the app.pl
system("echo $new_mtime_md5 > $appdir/last.fstat.md5");
system("cat $appdir/iam.running | xargs kill -9 && rm -f $appdir/iam.running");
system("( sh -c 'echo \$\$>$appdir/iam.running && exec $appdir/app.pl' >/tmp/esui.dancer.log 2>&1 ) &");
}
} else {
my $new_mtime_md5 = getMtimeMD5();
system("echo $new_mtime_md5 > $appdir/last.fstat.md5");
system("( sh -c 'echo \$\$>$appdir/iam.running && exec $appdir/app.pl' >/tmp/esui.dancer.log 2>&1 ) &");
}

sub getMtimeMD5 {
my $mtimes = '';
opendir(D, "$appdir/../lib") and do {
while (my $f = readdir(D)) {
next if (!-f "$appdir/../lib/$f");
$mtimes .= (stat("$appdir/../lib/$f"))[9];
}
closedir(D);
};
return md5_hex($mtimes);
}

Elasticsearch DSL to Perl Search::Elasticsearch query/filter body

vikas.nv@gmail.com (Viki) — Wed, 28 Oct 2015 23:06:00 +0530

I use "sense", chrome extension to try out all the DSL queries for Elasticsearch.

When I find a suitable query, I copy that to a notepad & modify the DSL query (JSON format) into a Perl data structure since I primarily use Perl dancer for development.

This manual conversion can become tedious if the queries go lengthy. For this I added a Perl dancer route to handle this conversion.

This Perl dancer route basically converts the JSON formatted DSL query into a Perl data structure so that it can be used to pass into a filter or query of Search::Elasticsearch object.

Here are the snippets of both the dancer app and the HTML page that uses the route. In the HTML page you have to modify [YOURSERVER] to your server where you host the Perl dancer app.

--------------------------------------------------------route--handler------------------------------------------------
package miscroutes;
use Dancer ':syntax';
use Dancer::Plugin::Ajax;
use Data::Dumper;

any ['ajax', 'post'] => '/json2perl' => sub {
my $req_ds = from_json(request->body);
local $Data::Dumper::Indent = 1;
local $Data::Dumper::Quotekeys = 0;
local $Data::Dumper::Sortkeys = 0;
local $Data::Dumper::Deparse = 1;
local $Data::Dumper::Terse = 1;
return sprintf "%s", Dumper($req_ds);
};

---------------------------------------------------standalone--html--page------------------------------------------
<html>
<head>
<title> JSON to Perl DS </title>
<script type="text/javascript" src="http:/[YOURSERVER]/javascripts/jquery.js"> </script>

</head>
<body>
<script type="text/javascript">
function submit_() {
$.ajax({
type: "POST",
cache:false,
data: $("textarea#in").val().replace(/(?:\r\n|\r|\n)/g,''),
url: "http://[YOURSERVER]/json2perl",
success: function(d) {
$("textarea#out").val(d);
},
dataType: "text",
error: function(j, errstr, err) {
$("div#res").html("ERRORED: "+errstr);
}
});
}
$(document).ready(function() {
});
</script>

<div>
<textarea type="text" id="in" style="width:500px;height:600px;"></textarea>
<textarea type="text" id="out" style="width:400px;height:600px;"></textarea>
</div>
<div>
<button onclick="submit_();" >Convert</button>
</div>
<div id=res>
</div>
</body>
</html>

Perl dancer - enable CORS

vikas.nv@gmail.com (Viki) — Wed, 28 Oct 2015 22:52:00 +0530

While developing using Perl dancer, it is common to run the app.pl located in ./bin directory while testing.
And suppose you have web pages that requests data from your dancer app through AJAX.
You will likely get into a situation where the browser in which you are trying to load the web page errors out saying -

No 'Access-Control-Allow-Origin' header is present on the requested resource.
Origin 'null' is therefore not allowed access.

This is because of a security feature of AJAX requests. You can solve this by adding a hook into your dancer app, like this -

hook before => sub {
    my $route = shift;
    header 'Access-Control-Allow-Origin' => '*';
};

Automate download web page (after evaluating Javascript/Ajax on page) for offline viewing

vikas.nv@gmail.com (Viki) — Fri, 19 Jun 2015 17:53:00 +0530

At times there is a need to download a webpage for offline viewing and the issue comes when the page is dynamic.

Dynamic web pages uses Javascript to modify the HTML or add styles dynamically, or even have AJAX calls in the web pages that downloads content to be displayed.

If you download the same using a command line tool like 'wget' or 'curl' you will get the web page in its initial state & you need to be connected to internet/intranet so that the browser does the Javascript/AJAX stuff.

To download a fully evaluated web page you need these things --

1. Perl
a. This module installed - WWW::Mechanize::Firefox (how to install Perl module)
2. Mozilla Firefox with MozRepl plugin added and started
a. Once installed, start the MozRepl in your firefox as below:
Tools->MozRepl->Start

To confirm if the plugin is installed and running try this:

And the code to download --
my $mech = new WWW::Mechanize::Firefox;
$mech->get("http://webURLtoDownload", 'no_cache' => 1);
$mech->save_content("$tmp_file_for_dwnload");

For more details about the WWW::Mechanize::Firefox you can read here

Steps to convert any Perl Dancer route handler to a Perl subroutine

vikas.nv@gmail.com (Viki) — Mon, 15 Jun 2015 15:08:00 +0530

For those who don't know what is Dancer; it is a Perl MVC framework and can be checked out here. The cookbook is very elaborate.

Here is what this post is all about --

#
# Way to convert any dancer route handler to a Perl subroutine
# 1. copy the route handler code, give it a subroutine name
# 2. remove use Dancer related lines from 'use' statements
# 3. include 'use Template;'
# 4. create a Template object with options as in -
### my $template = Template->new({
### INCLUDE_PATH => '/opt/my/webapp/views',
### OUTPUT => $output_file,
### ABSOLUTE => 1,
### });
# 5. in your template change the template variables to look like this -> [% tmpl_var %]
# 6. use the 'process' command of Template to write the final html
#

That's all for now.

Thanks

Configuring Apache -- enable Virtual Hosts -- for PerlDancer -- CGI on Plack -- Linux

vikas.nv@gmail.com (Viki) — Sun, 12 May 2013 08:41:00 +0530

Virtual Hosts in Apache can be used to host different web applications on the same server. Each application listening on different ports.

Assuming your Perl Dancer installation (which can be done using cpan shell/ppm shell/cpanminus) and creation of the web app scaffold, say in, /opt/webapp is already done.

Uncomment the rewrite module

Decide on the port on which to listen to (this example uses, 4129); and add another Listen directive along with the one that is already there.

Listen 4129

There will already be entry for Virtual Host at the end of the httpd.conf; which needs to be uncommented. An example for the VirtualHost directive --

Restart httpd

Javascript: Using ajax & jquery-csv to load a CSV file on client side

vikas.nv@gmail.com (Viki) — Tue, 16 Oct 2012 13:48:00 +0530

Basically, the need here is to store the required data in a CSV on the server so all that the client side Javascript needs to do is read that CSV and fetch the required data.

Here is a small snippet of code that uses ajax in jquery to read the CSV file and jquery-csv to parse the CSV and convert it into an array of arrays.

var csv_as_array = [];
function drawVisualization() {
$.ajax({
    url: "data-store/data1.csv",
    aync: false,
    success: function (csvd) {
        csv_as_array = $.csv2Array(csvd);
    },
    dataType: "text",
    complete: function () {
        // use the array of arrays (variable csv_as_array)
       // for further processing
    }
});
}

This I have used along with google charts to create charts depending on the CSV file passed.

HTML context recognizer, data extration, context driven content extraction

vikas.nv@gmail.com (Viki) — Sun, 2 Sep 2012 22:28:00 +0530

This is just an idea to ease the extraction of data from websites. This works on websites that are well structured and consistent in the structure. Usually, all the well crafted websites create webpages dynamically where the server uses a template at its end, so this idea should work.

The idea is divided into two modules, HtmlContextRecognizer and Gather. The other modules used are BeautifulSoup and argparse.

The idea is basically this -- given a sample (text appearing on a webpage) the HtmlContextRecognizer
recognizes the 'context' in which it appears, that is the tags it is surrounded with and the attributes it has.

And the module Gather, it uses HtmlContextRecognizer and extends its concept to do two things --
1. recognizes the context of a sample data text and uses this context to gather 'like' data
2. recognizes the context of a anchor text and uses this context to gather 'like' urls for navigation

As an example, I used it to extract jokes from a website that has a collection of sms'es.

I have not thought of any application for this, other than the one I mentioned above. Let me know how you used it.

Link to scripts: https://docs.google.com/folder/d/0B0-iayzVpwJ3T0gzSFdkMXJ1Vzg/edit

Debian/LXDE running on Acer Iconia A500 tab

vikas.nv@gmail.com (Viki) — Wed, 24 Aug 2011 18:47:00 +0530

I got a chance to have this tablet for sometime, and since then I have been hacking it. The first thing to do is most obvious, to root the device. I have reloaded the tablet with Taboonay's ROM (1.1) based on the Android Honeycomb 3.1 stock ROM. It has it's cool features, about which you can read here.

And thanks to lanrat for an article about this.

*** Do this if you fully understand what you are doing***

Now over to getting Debian on to the tablet.

Your device should be rooted. You need some free space on the SD card, about 1 GB is sufficient for this. Secondly you have to prepare a base installer image of Debian, I did this on Ubuntu 11.04. If you do not have a Linux box don't sweat it, get Linux on a pen drive and boot from that. And you need the Android terminal, also set it up such that your $PATH variable has /system/xbin in the beginning. For installing/configuring LXDE you need to have a mouse and keyboard connected to your tablet (I used an old bluetooth keyboard and mouse from Logitech EX100).

Once you are in Ubuntu open a terminal and do the following to create the base image -

sudo -s

apt-get install debootstrap

dd if=/dev/zero of=debian.img seek=1073741824 bs=1 count=1 # I use 1 GB image size **

mke2fs -F debian.img

mkdir debian

mount -o loop debian.img debian/

debootstrap --verbose --arch armel --foreign lenny debian http://ftp.us.debian.org/debian

umount debian/

rm -r debian/

** you can use lesser image size (800 MB) if you do not want to install LXDE.

With that you have a Debian base installer image. Now download this chroot script, which will be used to set up environment to run Debian (save this file as bootdebian). Copy this image and the chroot script to the SD card, such that it lays in

/mnt/external_sd

Now put back the SD card into the tablet and power on. Following commands will make the chroot script

executable in Android terminal and turn it into a command, open Android terminal on the tablet and type -

mount -o remount,rw -t yaffs2 /dev/block/mmcblk0p3 /system

cat /mnt/external_sd/bootdebian > /system/xbin/bootdebian

dos2unix /system/xbin/bootdebian

rm /mnt/external_sd/bootdebian

chmod 777 /system/xbin/bootdebian

With that done, you have a command 'bootdebian' which can be run to set up to run Debian.

After this you need to complete the Debian install, for that run the following commands on the Android terminal -

bootdebian

/debootstrap/debootstrap --second-stage

echo 'deb http://ftp.us.debian.org/debian lenny main' >/etc/apt/sources.list

apt-get autoclean

apt-get update

exit

This completes the Debian install. From now on if you want to work on a real Linux box, you just have to ope n the Android terminal and run 'bootdebian'.

Now, commands for installing LXDE -

bootdebian

apt-get install xdm # this downloads ~30 MB

apt-get install lxde # this downloads ~100 MB, so you can 'take wakelock' the terminal & go for a walk

Once the apt-get completes the installations, it starts configuring the LXDE. This is the time you need your hardware keyboard (if not mouse) when it asks for some settings. Look for the place where it asks for which display manager and select 'xdm'.

Next you need to install the tightvncserver so as to create vnc session. For that, run

apt-get install tightvncserver

Once done with that, you can start the session with these commands -

export USER=root

vncserver -geometry 1280x736 -depth 16

Above command will ask you for setting up passwords. This creates a session on localhost:1, which is a basic desktop. To start LXDE in the session, you have to instruct the vncserver using the xstartup script. For doing this, run the following commands -

vncserver -kill :1 # first kill the running session

vi /root/.vnc/xstartup

This opens the vi editor loaded with the vncserver's start up script. Edit this script such that it looks something like this -

#!/bin/sh

xrdb $HOME/.Xresources

xsetroot -solid grey

#x-terminal-emulator -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" &

#x-window-manager &

#/etc/X11/Xsession

startlxde

Save the file and close. Start the VNC session.

To view the session, you can either use the AndroidVNC application on your tablet or any VNC viewer on your PC.

This ends!

Script to set the compatibility mode of executable files on Windows 7

vikas.nv@gmail.com (Viki) — Sun, 20 Mar 2011 22:20:00 +0530

I wanted to install the portable Ubuntu on my Windows machine, and so I downloaded the executable from the sourceforge project site and installed it.

It did not work, I initially thought I have to change the compatibility mode of the executables and set it to Windows XP SP2. There were too many files for which I had to change, so I came up with a script.

This script takes these options:
-p /--path for which you pass the path (all executable files under this path will be considered)
-m /--mode for which you pass the compatibility mode (list of valid modes given at the end of the post)
-a /--runasadmin which is a toggle option which enables/disables the "Run as Administrator" option (default: disabled)
-v /--verbose which is a toggle option which print more information (default: disabled)

The script did work :) ... but the portable Ubuntu did not :(

You can download the script from here

Don't worry to run this script on the same directory multiple times, the Windows registry takes care of duplicates by adding a file as a single entry.

List of compatibility modes (that go with --mode/ -m option):

"WIN95": "Windows 95 compatibilty mode",
"WIN98": "Windows Millennium or 98 compatibility mode",
"NT4SP5": "NT 4.0 Service Pack 5 compatibility mode",
"WIN2000": "Windows 2K compatibility mode",
"WINXPSP2": "Windows XP Service Pack 2 compatibility mode",
"WINSRV03SP1": "Windows Server 2003 SP 1 compatibility mode"

EDIT:
How to run this script on Windows... ?
Firstly, install python on your Windows machine. For installation you should download an appropriate
setup file for your Windows machine from this link, then double click the setup file and follow the on screen instructions.
Secondly, download the script from here. And here is how to run it:
Open a command terminal in Windows by hitting Windows+R keys and entering "cmd" (no quotes)
Then, navigate to the directory to which you downloaded the script (using 'cd'), when you are there
type these commands:
python compatibility_change.py --path "c:\\path\\to\\executables" --mode WINXPSP2

IMDB API

vikas.nv@gmail.com (Viki) — Thu, 3 Mar 2011 21:09:00 +0530

[UPDATE: links are update]

Here I present the module which provides APIs to get information about movies from IMDB.

This is about a Python implementation of IMDB API.

You don't have to download the IMDB's databases locally to use these APIs.

I have used Peteris Krumins' xgoogle library along with the BeautifulSoup to develop this in Python.

You may have to install the BeautifulSoup module (download from here) and xgoogle library (from here)

And here is the link for IMDB API

Here is a little bit of description about the usage:

    from pIMDB import pIMDB
    if len(sys.argv) < 2:
        print "Usage: %s " % sys.argv[0]
        sys.exit(1)
    movie = re.sub(r'[^a-zA-Z0-9\']+', ' ', sys.argv[1])
    imdb = pIMDB(movie)
    imdb.parse_imdb_page()
    print "imdb link:\n\t", imdb.imdb_link
    print "\nmovie name:\n\t", imdb.movie
    print "\nmovie rating:\n\t%s/%s" % (imdb.rating, imdb.outof)
    print "\nmovie details:\n\t%s" % '\n\t'.join(imdb.details)
    print "\nmovie storyline:\n\t%s" % imdb.storyline

It also has an API to download the poster from the IMDB website, you can use it as here:
    imdb.download_poster('/tmp/destination.jpg')

That's the end of the story :)

PS: Write a comment on how you used this module

Configure Gmail on your N900

vikas.nv@gmail.com (Viki) — Thu, 18 Nov 2010 20:34:00 +0530

Here is a little guide to setup Gmail on your N900 using the Email application that comes by default on N900.

I am writing this because I could not find a working solution when I tried Google-ing "how to configure n900 for gmail".

Basically, I just helped my N900 to resolve imap.gmail.com and smtp.gmail.com
So here goes the steps...

1. Connect to the internet on your PC/laptop and ping imap.gmail.com & smtp.gmail.com. Note down the IP addresses of both. Here is a screen shot of the command prompt

2. Now on your N900 launch the Terminal and gain root access (if you don't know what I am telling, read this Rootsh)
3. Edit the file /etc/hosts (I use vi to edit)
4. Add following two lines to the file (do not delete any lines already existing in the file)
72.14.213.109 imap.gmail.com
74.125.53.109 smtp.gmail.com
(Replace the IP addresses in the above lines to what you noted down in the 1st step)
5. Save the file and exit the editor
6. Launch the Email application on N900. Follow the wizard that it shows by selecting Gmail in the list of providers that it shows.
7. Enter your user name and password and go till you click finish.

The account you set up should work fine. If not, tap on Edit settings for the account and make sure the settings are as shown in the screen shot below

Frozen Bubble on N900 :) !!!

vikas.nv@gmail.com (Viki) — Wed, 6 Oct 2010 20:34:00 +0530

Have you played this game, frozen bubble? Its the most addictive game I have ever played ... next to some games like bejeweled. I had this game on my previous phone Nokia 5800. So I was searching for this game ... hoping some one would have ported it to N900 (currently I own it). WOW I found it here http://maemo.org/packages/view/frozen-bubble/

It is still in development repository but I have used it and it is working perfectly. It has great music and great graphics this time, and its called frozen bubble 2.

Enjoy!

how to re-organize menu icons on N900 (with catorise)

vikas.nv@gmail.com (Viki) — Sun, 29 Aug 2010 21:27:00 +0530

You can get the icons in the menu of your N900 organized into folders using an application called catorise.
This app automatically groups the menu icons and puts them into different folders of meaningful names.
But sometimes you feel ... hey this icon should be in the games folder why is it put into the ovi folder??? I felt this when i saw bounce in ovi folder after installing catorise.
And so i launched the xterm and started looking for some kind of a config file or something where i can change the grouping. Went into the /opt folder to check out the installed apps ... there was a folder catorise cd into it and ls ... there was something very obviously named file 'menu'. But i need to be root to edit this, became root and opened this file using vi. It is a simple config file with this syntax ->
app name: folder name
I changed ->
bounce: ovi
to this ->
bounce: games
Saved it and closed it.
Then ran the perl script named catorise in pwd.
This changed the grouping, bounce appears in games than in ovi :)

PCron ---- portable cron :)

vikas.nv@gmail.com (Viki) — Wed, 24 Mar 2010 20:05:00 +0530

PCron

Cron for Windows & ... :)

Cron is basically a job scheduler available in Linux and its variants/flavors.
I have tried to implement the functionality of Cron in Python so that it can be used in Windows and ....

You should have Python 2.5 + installed to use this.

Input for PCron is a plain text file (lets' call it pcrontab) with syntax/format similar to *nix crontab. pcrontab can have formats such as -> */12 and 12-20
Here is an example of a pcrontab:
*/5 * * * * F:\Program Files (x86)\K-Lite Codec Pack\Media Player Classic\mpc-hc.exe
57-59 18 * * * F:\Program Files (x86)\Calc.exe

In above example, first line instructs PCron to start the program mpc-hc.exe every 5 minutes.
And the second line instructs PCron to start the program Calc.exe on 18:57, 18:58, 18:59

To understand more on the syntax of pcrontab ... look at the picture below which explains about crontab ->
Usage:
python pcron.py --taskfile
Examples:
python pcron.py --taskfile c:\\pcrontab.txt
python pcron.py --taskfile /home/vikas/pcrontab
Currently it writes out logs to stdout, so you have to redirect it ... to say /dev/null or a file

Limitations:
Each time you add an entry into the pcrontab, you have to restart the script or wait until the script reloads the pcrontab (it reloads in every 5 minutes ~)
Does not support entire format/syntax combinations as compared to crontab

More things to come:
An editor for pcrontab ... that would detect changes in pcrontab and initiates automatic reload in pcron
Support for a timeout field in pcrontab, which enables us to run a program for a certain period of time

You can download the scripts from here:
TimedPopenNT
Scheduler
pcron

On Windows you can generate a .exe file using py2exe and use that as standalone instead of the script.

Extract or Reap or Download Pictures from HTML Web Pages

vikas.nv@gmail.com (Viki) — Thu, 18 Jun 2009 21:12:00 +0530

I was exploring python and the HTML parsers that it has. I hit a page that explained about the SGMLParser and its uses.

Using this package I had written a simple script that could extract pictures from simple HTML pages. While experimenting it with many websites I noticed some websites did not allow my script to reap or extract pictures.

Then I found that it is because of the User-Agent string in the HTTP request. Some websites do not respond properly to HTTP requests that have unknown User-Agent string. I then used the urllib's FancyURLOpener to change the User-Agent to Firefox.

Now that the script work on most of the websites, i would like to post the code here. Feel free to modify to make it more robust and reliable, post a comment or a link to your modified script.

This script search pictures from web pages and download them for you. All you need to do is
get the URL of the web page from which you want the pictures to be downloaded and pass it as
an argument to this script. Here is how to use it:
0. Note that python is installed on your system to run this script (Windows/Linux/Solaris ... OS independent;)
1. Download this script and save it as gal_ext.py
2. Open a command-prompt (if Windows) or Terminal (if you use any *nix OS)
3. Type "python gal_ext.py 'http://URL'
4. Here note that the 'http://' is mandatory
Using this script you can avoid downloading images with a web browser, which takes a long time and is annoying to right click on every image and save them. This way you can download picture from web faster and easier. Since this is a script it is open for modifications and is free to use. But if you modify, make a note to send it to me ;). Thanks

The code is here (or a link for you to download):


from sgmllib import SGMLParser
import sys
import os
import re
import urllib
from urllib import FancyURLopener
from urlparse import urlparse

class URLLister(SGMLParser):
def reset(self):
SGMLParser.reset(self)
self.urls = []

def start_a(self, attrs):
href = [v for k, v in attrs if k=='href']
if href:
self.urls.extend(href)

class MyURLOpener(FancyURLopener):
version = 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.6) Gecko/2009011913 Firefox/3.0.6'

def url_mapper(var):
regexp = re.compile('.+?(http://.+&?)', re.I)
values = urlparse(var)
if len(values) >= 5 and values[4] != '':
obj = regexp.search(values[4])
if obj:
values = urlparse(obj.group(1))
else:
if values[4] != '': print('Could not get real url from:' + values[4])
site = re.sub('[^0-9a-zA-Z_/]', '_', values[1])
location_file = re.findall('(.+)/(.+)', values[2])
try:
if len(location_file[0]) != 2:
return ''
except IndexError:
return ''
location = location_file[0][0]
file = location_file[0][1]
location = re.sub('\.{2,}/', '/', location)
location = re.sub('[^0-9a-zA-Z_/]', '_', location)
location = location + '/' + file
return site + location

if __name__ == '__main__':
url_opener = MyURLOpener()
parser = URLLister()
sys.argv[1] = urllib.unquote(sys.argv[1])
try:
usock = url_opener.open(sys.argv[1])
except IOError:
print 'skipping ' + sys.argv[1]
sys.exit(0)
parser.feed(usock.read())
parser.close()
usock.close()
count = 0
urlfs = urlparse(sys.argv[1])
print "Url fs: ", urlfs
parent = urlfs[1]
try:
parent += urlfs[2]
parent = re.search('(.+)/.+',parent).group(1)
except IndexError:
parent = urlfs[1]
for img_url in parser.urls:
if re.search('\.jpe?g$', img_url):
print "looking at: " + img_url
if not re.match('^http://', img_url):
img_url = 'http://' + parent + '/' + img_url
loc = url_mapper(img_url)
if os.path.exists(loc): continue
url_opener1 = MyURLOpener()
retrieve = url_opener1.retrieve
if loc == '':
print "\turl_mapper returned NULL for " + img_url
continue
loc_dir = re.match('(.+)/.+', loc).group(1)
try:
print('\ttrying to save ' + img_url)
if not os.path.exists(loc_dir):
 os.makedirs(loc_dir)
retrieve(img_url, loc)
except IOError:
print('\tSkipping saving ' + img_url)
continue
count += 1
print "\tImg fetched: ",count

Installing andLinux on Windows Vista

vikas.nv@gmail.com (Viki) — Thu, 28 May 2009 22:23:00 +0530

After downloading the andLinux installer (KDE version), launch the installer as Administrator. And follow the steps with default options set.

Reboot/Restart machine after installation.

Click on Unblock if the Firewall prompts.

After the reboot, you click on Konsole (any) launcher on KDE Menu from system tray, you are likely to get this error "could not launch ... could not connect to 192.168.11.150"

So this guide explains how to fix this issue, for the launchers to work on Windows Vista.

First thing to do is to go to the "Network and Sharing Center" from Control Panel. You can see the "customize" and "view status" links on "Unidentified Network". Click on the "view status", click on properties and double click "Internet Protocol Version 4". That should open up a dialog where you can change the IP and the subnet mask, change the IP to 192.168.10.1 and subnet mask to 255.255.254.0. Click on OK OK ....

Secondly, open the "andlinux Console" get sudo access and edit the file /etc/network/interfaces.
After edit the file should have IP of 192.168.10.150 and the mask 255.255.254.0 under eth1

Thirdly edit file /etc/profile to have IPs set to 192.168.10.150

Fourthly navigate to the andlinux installation folder goto Xming folder edit X0.hosts and add the following IPs 192.168.10.150 and 192.168.10.1

Lastly few registry changes, regedit and navigate to \\HKEY_LOCAL_MACHINE\\SOFTWARE\\andLinux\\Launcher
Here you find to keys "IP" and "Port" set them to 192.168.10.150 and 2081 respectively.
(Remember to click on Decimal while editing Port value) And close the regedit.

Finally kill KDE Menu, Xming. Restart andLinux service, and then start Xming and KDE Menu.
Now the launchers work fine.

If clicking on a launcher does not start, you open the andLinux Console application and execute:
"/etc/init.d/networking restart". After this any launcher should work fine.

!Enjoy!

Parsing and Storing parts or paragraphs of log file

vikas.nv@gmail.com (Viki) — Sat, 9 May 2009 19:20:00 +0530

At many times we need to parse and store parts of a log file. Suppose an error log, you would like to parse all errors. If the errors span only one line, you could use a grep.

But if the errors span multiple lines ... you need to put a little logic into your script. There are many ways to do this, but this script that i post here is more readable and can be modified to other's requirement.

Link to the code
Link to a sample input

In the example script i am parsing an error log (named 'input.txt'). The script stores all lines that start with "SQL0204N" till it finds an empty line. Likewise it extracts all errors, even if the error messages span multiple lines.

Perl script inside a DOS Batch file

vikas.nv@gmail.com (Viki) — Wed, 1 Apr 2009 15:05:00 +0530

Whenever I wanted to invoke a utility passing a file as an argument I used to create a shortcut of the utility in the Send To folder, as described here :

How to Add Items to the "Send To" Menu

But how to have a Perl script in the "Send To" menu??

I got an idea when browsing through this page :

DOS Batch - Interfacing non DOS Software

I created a batch script as described in the page. Then modified the part of the batch file where the Perl script actually starts.

I wanted to have a utility that can encrypt/decrypt a text file. A basic encryption was sufficient, so considered the Caesar cipher.

I completed the script and added it to the Send To menu.
Link to the script

A small modification to the above script to handle binary files. Here is the link.

andLinux --- Run Linux from Windows

vikas.nv@gmail.com (Viki) — Sat, 13 Dec 2008 22:10:00 +0530

Have you ever had a feeling of using Linux when you are working on Windows???

Felt like using those very useful Linux commands fromWindows???

This is possible ...

Everybody knows that you can have a Dual-Boot on your PC with Linux & Windows. In which you have to restart the PC to access the other OS.

But there are several other ways to have Other OS run right from Windows, that is for example use Linux when you have logged on to Windows... Yes... Virtual Machines (also popular as VM) is one of them. But running a VM requires that your hardware to be really good.

There is one more option, we now have emulated PC software like andLinux.

There are many other softwares other than andLinux, such as CygwinX, MKS Toolkit, etc

But I found andLinux to be really very handy, straight-forward & useful.

Installing andLinux is as simple as installing other software ... clikcing Next, Next, Next ... & finnaly Finish :) there is a very elaborate tutorial on its web site

However if you are the curious cat... you can also fiddle around & do the installation without the tutorial.

The home page of andLinux says "andLinux is not just for development and runs almost all Linux applications without modification". I have installed a lot many applications to andLinux, and its really true.

After installation you can see the icons of andLinux in system tray. You may see these icons :

KDE menu (andLinux)

Xming (andLinux)

While installing say you choose E: drive as destination. It gets installled in a folder E:\andLinux.

It will create some folders & files in E:\andLinux, I will list some folders & files with thier purposes:

1. settings.txt, a plain-text file that will have some paths necessary for andLinux to work properly

2. srvstart.bat, batch file, that has the command to start the andLinux service

3. srvstop.bat, batch file, has the command to stop andLinux service

4. startup.bat, again a batch file used to start the andLinux as deamon

5. Drives, directory, where the andLinux's filesystem is stored as files (base.drv, mine is 4GB in size and swap.drv)

6. Launcher, directory, where all the executables can be seen. A good place to check out what all linux apps you get after installing andLinux.

7. There are other directories as well, netdriver, pulseaudio & Xming

One application that is very usefull after installing andLinux is the synaptic. It is the application manager of andLinux. You can find this from KDE Menu / open a terminal & type synaptic.

There is a huge collection of linux applications that you can select & install. As i feel the GUI is staright-forward & easy to understand, compared to CygwinX. It is more responsive on my machine with 512 MB of RAM & Dual Core Intel.

It is still in Beta, you can go to the andLinux website to download it. There is a torrent tracker also.

EnJoy ...

Script to comment parts of code in a source file

vikas.nv@gmail.com (Viki) — Fri, 12 Dec 2008 17:01:00 +0530

A friend of mine wanted a script to comment parts of code in a source file. We were chatting & i gave him this solution:

me: perl --i.bak -pe 's/^(.*)$/#$1/gi if ($. >=start_line_number && $. <= end_line_number)' filename

Try above command in terminal. Where start_line_number is the line number where u shud start commenting & end_line_number is where u want to stop commenting.

But he wanted that the script should take a function name as input rather than line numbers ... this is what he says: (name not published ;))

myfrend: hey vikas, i need a script to start commenting the codes without using line numbers, but using function name....is it possible?

me: u mean to say u have function names ... & u want to comment the whole functions?

u mean to say u have function names ... & u want to comment the whole functions?

myfrend: yeah

me: it is possible ... if u can get correct start&end boundaries for the functions

myfrend: yeah..how to get it....its difficult

me: how big can the file be .. that is MAX big file size?

myfrend: say around 3000 lines

I was asking a lot of questions about the source files like do they follow coding standards, etc .. & all that. So he suggested what he wanted to do ... i mean the logic ... he said:

myfrend: my logic is : search for function name, then keep a counter to increment/decrement when { or } is found, when counter is zero, exit the prgm

me: this is holds good ... but u have to account for { } occuring on same line ... & account for comments, etc

myfrend: ok

Got some free time to think about it ... & wrote a script for this. This may not be a complete solution, but a good starting point i guess.

Here is the link to the script:

http://docs.google.com/Doc?id=dg2jxmfs_43gbtd2jfq

PS - myfrend :

If you permit I can publish your name ... with a hyperlink possibly to your blog ;)

Get Spiritual ...

vikas.nv@gmail.com (Viki) — Thu, 21 Aug 2008 17:01:00 +0530

Get spiritual ...

Take a look at the photo galleries shared by my good friend in his blog.

http://srivatsashroff.blogspot.com

Notice lot of temples in the galleries, that is all with getting spiritual :)

PS: Obviously, this is post has nothing to do with Perl | ;) |
Viki