The Pensieve

R programming

2019-12-02T22:49:00.000+05:30

Of late, I have been following up with data analytics and learnt R and Pandas (using Python). Somehow, most of the work that I now do is crunching data and representing them in a more fancier way. In a lighter vein, when I am at home, I try to see how my non-work data can be represented as charts and graphs. Here is the first attempt to categorize the list of programmers editors that I had used throughout my life time... This was generated using R and ggplot (ggplot is a great library that allows you to generate different types of charts).

Why not excel? Sure, I can do the same in excel as well. But once the data becomes huge to manage, it becomes difficult to monitor each of the cells to see if the formula is right. Sometimes, they get complicated enough that I have to wait for a few seconds every time I make a change in a cell or consider turning off auto-calculation in excel. It is much more simpler to have the processing logic as a source file that you can modify separated from the data.

The data file in this case is a simple .csv file and the R code to generate the content is given below:

And the output produced is this:

What do you think?

World Tobacco Day - May 31

2010-06-05T19:11:00.006+05:30

On my daily commute to my office, my only companion are the various FM radio channels. Most of them were hyping the World Tobacco Day on the 31st of May which included how much a person would have spent if he smokes a few packets to the various health issues. All this was nice until on that same day, where there were a few ads on the same channels sponsored from a tobacco company enticing listeners on their new offerings! Oh what irony!

2009-10-31T15:08:00.002+05:30

On the way to Sholinganallur via the IT highway, a flock of Pelicans have made the backwaters their 'in-transit stay' much to the delight of the commuters. Most of the commuters slow down and some (including myself) visited over the weekends to take a snap of the feathered friends.

50% off - Bah, humbug!

2009-08-09T22:40:00.006+05:30

I had to do some shopping and was lured into a some of these shops by their big bold 'off' posters.
But alas, once inside, anything that I touch, the salesman utters "Sorry sir, but that does not have the discount". Not to be out-done, I ask "So what do you have on the discount sale?". He points to a table where there are a 'few' set of shirts that no one would pick up, in violet and pink and fluorescent green. But their prices seems to be barely justifiable even 'after' you apply the discount.

Note size of the '50%' and 'Sale' on the poster, where I think the entire wares on discount would comfortably fit in and you will still be able to make out what the poster says. Perhaps there should be some legalization on the size of the '50%' that you can have on the poster to depend upon the number of articles you have on a discount compared to the remaining items.

In the end, you end up buying the product at the same price you would have bought on any other normal day, but with the rush of eager shoppers, few choices and sub-par merchandise.

A+: Recommended visit if you have money to burn and love violets, pinks and fluorescent greens!

Desktop search vs good old grep

2009-07-26T23:15:00.005+05:30

Been installing various desktop search tools like the Google Desktop, Copernic (once free), Microsoft Search etc and after a few weeks / months of usage, eventually felt happier to uninstall them. One of the primary reason I need a desktop search was I needed a tool to quickly search for mails. I would seldom search for files on my machine.

But one assumption that we need to keep in mind is that, these search engines are not exact. That is, the engines fetch you results, but they may not be comprehensive. The web crawling spiders that scour the web for data and index them, it is possible thet the server's index may be out of sync with respect to the actual web site as new data would have been posted after the spider index the content. In the Internet, sure, if you have a few sites missed out, it does not matter much. But desktop search apps seem to work in very similar lines. If at some time, for want of speed or to free up available memory, you stop indexing your mails or documents, chances are, these mails might not be indexed at all.

So, there were various instances where I was searching for that particular mail and it did not show up. Be aware that such search engines are not comprehensive and it is a good idea to resort to the good old linear search just it case. Though they may take time, they give you comprehensive results.

The same would apply when searching for keywords in code that you plan to manipulate. You would want to get all of the matching sources and not want to miss out even one.

AJAX - Is it really worth it?

2009-03-28T21:13:00.003+05:30

With the onset of interactive web pages that update dynamically the moment you move your mouse over a control, development of an AJAX (Asynchronous Javascript And XML) based web site adds to the complexity.

For one, they mess up with your browser history and limit your ability to bookmark a particular page as almost all contents are loaded "in-page". The other disadvantage I see is the in-ability of custom scripts to access the site and extract information.

For example, I can write a perl script that can access the Indian Railways page and query the arrival of a particular train. If the site used AJAX, it's going to be more difficult. Cellphones that bundle with micro-browsers either lack javascripting or access is usually blocked by service providers leaving you grapping with an empty framework with no actual data in it.

Development wise, you introduce a sense of uncertainty into your web-page where when your page looks like it has completed loading, more data is still being AJAX'ed. This may lead to difficulties in maintainability.

Though there are a lot of benefits of using AJAX, the main being better user experience, for scripters, this is a blocker.

A parallel web-service based interface would help those who are armed with Perl or Python to extract useful information at a jiffy
A simplified web alternative will cater to the needs of the cellphone user or the average scripter.

Amazon, Google and a few other providers focus and provide open Web Services and / or a simplified interface, but this is something lacking from mosr of the other providers... With the popularity of consolidation sites (www.google.com/ig anyone?), hopefully this will change.

ssh! Here's the scp

2009-01-26T12:53:00.006+05:30

The other day, I was thinking of consolidating all my music collection from various machines and burn them into a DVD as a backup. There were various ways that I could have done the transfer:

Extract the music back from the disk originals
Copy them using a rewritable media like a pen drive or a writable media like a CD
Connect over the network and copy the files

I can use Samba as some of my machines are Windows based
Setup a FTP server to copy files
I can setup a ssh server and securely copy the files to my Linux box

That was how I discovered ssh, the secure shell. I had originally been looking at setting up Samba on my Linux box, but was frightened with the possibility of it suffering a virus attach as most of the viruses / botnets would be scanning for a windows PC and this would become a backdoor for entry.

The other option was ssh which was simple to setup and quite secure. It basically provides a secure way to connect to a computer that enables the ssh service. You are presented with your standard console where you would be able to perform your usual activities.

Another prominent activity is copying files about, between the host and the client and this is where scp (Secure Copy) comes in. scp allows you to copy files between your client and the ssh server securely.

Now, scp is a command line tool and most get intimidated with copying files from the command-line be it the versatile '$' prompt or the more familiar 'C:\>'. On the windows side, you have winscp which is a GUI application with a Norton commander-like interface which would make GUI aficionados feel at home in copying files safely and securely.

Winscp comes as a portable app, which means, you can actually run this app in your windows machine without fear of cluttering your registry and contributing to registry bloat. Portable apps are targeted to those who would like to keep a handful of portable apps in their pen drive which can be executed without any requirement for installation or setup. Check out portable apps for more such applications.

Linux has better throughput?

2009-01-25T12:49:00.005+05:30

The other day, I saw news item at slashdot where people mentioned that Linux seemed to perform better that Windows on network throughput. Some attributed it to the ISP throttling the bandwidth for the first few minutes where as others mentioned that the Linux network drivers seemed to be configured to maximize network performance where as the Windows settings are sub-par.

An indepth article depicting based on actual data is provided which eventually leads to thinking perhaps, you should tweak your windows network settings...

Though people had complained that both the machines tested had slightly different specs and hence the performance results could be skewed, I have been browsing on both a windows PC and a linux box and it does look like (no systematic tests conducted!) the throughput that I get on a Linux PC is much higher than as experienced on the windows PC so far.

The windows PC or laptop has 512MB RAM and is an Intel Centreno Duo 1.66GHz where as the Linux Box is a older AMD Athlon XP @ 2.66GHz with 384 MB of Memory. Both connect via the wireless. The few parameters which may make a difference are:

The XP box is laptop which may compromize for space rather than for performance
The Desktop has a higher MHz rating
Virus scanner on the laptop may bring down the performance. The Linux box has no such apps hogging memory or CPU bandwidth.
Network adapter differences - The desktop has a netgear wlan card which may offer better performance

So, based on these parameter differences, it looks like we would have to take the observation with a pinch of salt. On the better side, as more and more news and comparisons of different OSes gain popularity, it is the users who would stand to benefit.

Global Meltdown Sale!

2009-01-14T10:55:00.003+05:30

Aadi Sale, New Year Sale, Year End Clearence Sale and now the Global Meltdown sale! Happened to see this posted at a window of a garment shop at Pondy Bazzar, T-Nager. An innovative approach to woo in customers. Apart from the PR, this shop sells really good (limited collection) and cheap (3 times cheaper) clothes for kids compared to the branded cos like the Levi's or Lee.

London Stores, Pondy Bazar, T-Nagar.

All time low for IT

2009-01-08T22:35:00.004+05:30

It's unfortunate to see such a big corporation fall down in light of the scam announced yesterday. Share prices of the listed script dropped down from Rs.170 to Rs.35 in the NSE in a matter of hours and the entire IT industry was watching the news unfurl dumbstruck. The fate of 52,000 employees and their families hang and hopefully, the company must either build back the trust of the share holders and its employees or the employees should be is a position to secure jobs in other companies. Only time will tell...

Happy New Year 2009!

2009-01-01T20:54:00.005+05:30

One more year has passed by and the time has come to wish all of you a happy, healthy, safe, peaceful and prosperous New Year!

This is the time when one looks past on what has happened and makes plans on what to achieve and focus going forward.

And the best of luck in making up your New Year resolutions. As one of my mentor mentioned long back, you would usually be bound and emotionally attached to your resolutions if you commit yourself in paper by signing on the paper what you plan to achieve. Though most of my resolutions are broken in the next few days / weeks, I am planning to be different this time and commit my pain points to paper (maybe I can get a bond paper too, to make it really legal ;-)).

Give it a try. You might be able to hold on to your resolutions a little bit longer until it becomes a habit.

And no... I'm not going to publish my resolutions here ;-)

Happy (Internet free) Christmas!

2008-12-25T20:01:00.005+05:30

Merry Christmas wishes to one and all!

In other news, the partial Internet free days make me ponder on reliant we have become on these series of 'tubes' that send and receive data from various servers. This brings in light of the hassles that you may face when you become fully attached to online applications.

My e-mail provider provides LDAP and is online-only. You have a plethora of online-apps that provide basic word-processing, do your taxes, view pictures and store presentations etc. If you have the cable cut, you loose access to everything.

Telephone and credit card companies are urging their users to move online, to receive bills and statement alerts via e-mail rather than the traditional mail in the pretext of accessibility and saving the environment. I'm not against saving the environment, but most of us are moving towards a system where there is only one way to get informed about the various activities in life and if this one way gets disrupted, I wonder what would happen as it would take days or months to get back to the regular, good old reliable snail mail.

PS: Save your stamp collection. Some day, it would fetch million$!

SMS SPAM filtering?

2008-11-30T22:22:00.003+05:30

Been there, done that. That's what you would hear if you are talking about SPAM filters for your e-mail. The other new area that I am constantly getting irritated about is the mobile SMS SPAM that make their way into my cell phone.

One way or the others, we inadvertently give away our mobile phone nos to the various dealers who pass on the same to bulk SMS generators. They in turn make it a point to enlighten you of the various offers that you should be thankful to spend your money on! Thank God I am not charged for incoming messages!

I know most providers like banks or the telecos provide a Do Not Call registry usually hidden away in the small-print. They abide by your request, but it is the other sources where you are forced to give away our number from where the main problem arises.

One thing that I noticed that most mobile phone lack is an active SPAM filter for your mobile phones that can scan your incoming SMSes and over a course of time, learn from you, on what you regard as SPAM and what are valid message. This algorithm of scanning the frequency of words that makeup an SMS and classifying them based on your inputs as one of 'possible SPAM', neutral or 'likely a valid SMS' is called the Bayesian SPAM filtering.

Looking at the Java MIDP API, I could not find any way by which I can intercept or takeover the built-in SMS application. However, from the forums, the friendly folks at Nokia informed that it might be possible using the C++ SDK.

So, my next possible app would be to try on the feat before which there might be a few simple applications to hone my skills on the S60 API. Check out the source forge project mymobiletools shortly for simple apps that might enhance your experience with your mobile.

Design Patterns: The art of making TV serials or masala movies

2008-11-22T22:12:00.005+05:30

After being tortured to sit through one too many series of never ending serials, here are a few rules or patterns that I had observed:

Design patterns on serials or movies

The-Not-Good-as-it-seems Pattern: If it is too good to be true, it is a dream and in real-world, it will happen exactly the opposite
The-One-is-not-enough Pattern: One of the primary character will have two wives and will struggle between the two (or more?).
Medical-Miracle Pattern: Someone will get amnesia and later will be always be cured.
Double-Trouble Pattern: If twins are born, they will get separated to meet again 20 years later. Of-course, there will be scope for impersonation.
The-Cryptic-Prophet Pattern: Swamijis, gypsies, sadhus will usually be able to give cryptic clues on what is about to happen next
The-Cycle-of-Life Pattern: If a primary actor's family is portrayed as being very rich, chances are they will become poor. Conversely, the family opposing the hero / heroin will have the lifestyle opposite to them.

The art of extending airtime

The-Matrix-Bullet-Time Pattern: When the situation becomes tense, you get into slow-motion mode with background accompaniment *after* every statement / action made by the actors.
Manic-Mondays Pattern: Friday (for serials that are aired on weekdays) sessions will usually be a cliffhanger, ending just before a pretty tense moment which will usually fizzle out on Mondays.
Bored-With-the-Existing-Faces Pattern: When things become too dull, introduce a new set of characters and follow a side storyline
The-Forget-Me-Not Pattern: On long running soaps, introduce a recap session that compresses all major moments that had happened earlier which the actor will ruminate

Now a days, a lot of reality shows have made it into our TV channels. Here are a few patters that I have observed here:

The-Blunt-in-Your-Face Pattern: The judges make it an effort to be rude while pointing out the participants faults, with closeups of both the judges as well as the participants at the recieving end with again, background accompaniments.
The-Let's-Cry-Together Pattern: When a participant or pair is bumped out of the contest, you should watch the emotional drama. The relatives and friends of the bumped out participants hurling verbal assaults towards judges and the judges themselves get into an emotional state deciding to quit and all others wooing them to come back. This is something that you should not miss . OK, OK, I was kidding about the last statement!
You-Have-Yet-Another-Chance Pattern: When the participants are dropping off the chart like flies after you switch on the repellent, the organizer will decide to pardon the non-performers and continue to the next session.

CD-less Car Audio Systems

2008-11-22T21:51:00.003+05:30

The new Honday City comes with an audio player that takes your portable player, collections of memory or USB sticks. Though a good option that gives flexibility allowing you to carry your music and listen to all of them, the minus point is, they now lack a CD player.

I see this as a disadvantage. Guess where I will play a newly purchased CD? In my car, on my way back home! The same goes for CDs purchased during a long trip. I might not have my laptop handy to rip and transfer songs on the go.

The music download scenario currently in India is not good either. Amazon and iTunes do not offer song downloads here. So, purchasing online music which is likely to happen during your home or office are not it atleast in India.

With these audio system, I will have to hold on until I reach home, rip the songs, copy them to my USB stick and only then, I will be able to listen to them in my car. Though most cars offer to add a CD as an option, for the cost, I feel this should be standard.

The Age of the Hybrid?

2008-11-19T22:33:00.003+05:30

Honda recently offered a slash in their new Civic Hybrid model. Though not all are sure if it was a publicity move or an inventory clearing sale, I believe with good mileage figures and a growing concern on how green everyone should be, hybrid cars will see an increase in prominence in the Indian roads.

The Indian market is price sensitive. Be it an industrialist who is buying a Rolls Royce or a car enthusiast pining for a BMW, the inevitable question remains: "What is the mileage?". This has led to a slew of Diesel cars in the market . Though economic to us, it still fails against the 'green' parameter.

The other competitor to hybrids are the all-electric vehicles which may be termed greener (assuming that the cost of production, battery is far less harmful than the actual emissions), they lack in range, ability to quickly recharge (which seems to be a 5 minute affair for it's counterparts) and the lack of accessible recharge locations (do you think at the current electricity cuts).

The hybrid vehicles usually offer dual modes which combines the benefit of being green (to an extent), offer great mileage along with the ability to refuel using popular modes quickly. Maruti and Hyundai have already started offering a less greener dual option: CNG and petrol to the consumer which guarantees fewer paise / km.

So, hybrids, though being more expensive caters to all the other needs. The government can, to encourage this, offer to reduce the import duty for such cars as, in general, they help the environment much better than the other vehicles on the read.

So Toyota... are you listening?

Dynamic Math Question Paper

2008-11-09T12:45:00.004+05:30

If you have kids who are into schooling, now would be the time when you are revising their knowledge for the upcoming mid-term and half-yearly exams.

A good way to check up on your kid's (and your) math skills is to setup up a question paper. Unfortunately, kids have good memory power that you can't reuse the same maths question paper. Otherwise, you would see them breeze through the QP as they tend to 'remember' most of the answers.

The alternative is to have multiple QPs. But that is a pain and I had been thinking of coming up with a simple script or software (TeX) to generate the QP dynamically.

The final solution was much simpler. Use Excel or Open-office Calc (in my case) to dynamically generate the question paper. The key is, for each of the question, have a bunch of random numbers (to your specifications) generated so that you can construct the parts required for your sum and generate multiple versions of the question paper by recalculating the workbook.

A bookshelf in your palm

2008-11-07T23:26:00.004+05:30

With the advent of e-books, software readers, hand-held devices and the Project Gutenberg, you literally have access to more than a book shelf.

A good e-book reader for your mobile when you are standard at some place with nothing to do is Mobipocket reader. It comes along with a good PC based e-book creator that can take simple HTML or .txt files and convert into a format usable from a mobile device.

With the text files combined with Emacs and a few regexps, I was able to convert most of the files to readable html files in a few minutes and feed them back to the e-book format converter for hours of readable entertainment.

So, my current bookshelf contains:

Arabian Nights
Robinson Crusoe by Defoe Daniel
The Devil's Dictionary by Ambrose Bierce
Dracula by Bram Stoker

With these books with me, who would mind waiting on a long queue?

The portable game framework

2008-10-19T23:05:00.005+05:30

Ever since I had seen GCompris, a children activity application, I always wanted to add modules that caters to the need of getting my son to learn while he plays games. GCompris supports a python interface for rapid board development. But one quip that I had was, GCompris is a Linux only application (though they have a commercial win32 port). Now, if I can find a framework that works both on the Linux as well as the windows platform, it would make great sense. Gcompris as of now only will restrict schools to the Linux OS (which is good, but most schools would probably be using Windows and getting them to change would prove difficult) would be good.

An alternative framework would be Java with it's great 2D API (no fancy 3D here to confuse the kids). Others would by SDL with C/C++ or pygame (which I read has performance issues on the windows front. Read childsplay). A new contender that I noticed was the Mono framework which sports a 2D API identical to the DotNet framework.

My immediate course of action would be to write a few python modules based on GCompris and see if I would be migrate it to another framewor later as GCompris currently has the framework where as mine is still vapoware. Plus, it also has good i18n capabilities with a library of digitized voice samples where a build-from-scratch approach would require a lot of work.

WiFi atlast

2008-10-12T14:08:00.006+05:30

After numerous hours of tweaking, I resolved the problem with my WG311v3 NetGear PCI card. The problem? The router was too far away!

Now that I am online using my Linux box, here are a few screen-shots of my desktop!

The next time your supported WiFi card acts up, you know what to do...

For those inquisitive lot, I am currently using the XP drivers with ndiswrapper.

Look around you...

2008-10-11T22:25:00.006+05:30

Most people think that they need to be at the right place and at the right time to get a great picture. I say, all it takes is patience and the ability to see differently. While exploring the place nearby, here is what I managed to click, all within a radius of 10 meters from the front door...

For more, visit my picasa page.

Whois[she|he]

2008-10-03T20:21:00.008+05:30

One of my favorite ploy to those who newly joined my previous company was to give them a command line application and ask them to find out how it worked. This magic command line tool happens to guess your gender if you give your name.

char lastc = argv[1][strlen(argv[1])-1];

printf("%s is a %s\n", argv[1], (lastc == 'a' || lastc == 'i') ? "She" : "He");

Basically, it checks to see if your name ends with an 'a' or an 'i' and if it does, assumes that the person is a lady. Chances are, 95% of most Indian names fall into this category and this simple app seems to do magic.

Most were able to (after a few tries) guess the grand algorithm, but a few poor souls could not, which eventually leads to step number 2.

Step #2 is, as they were unable to decipher the algorithm, I said that I will provide with two separate command line apps called whoisshe and whoishe. I also mentioned that as I had not done the validation part properly, to please provide only men's name to whoishe and a ladies name to whoisshe. If they provide the wrong name, the results could be wrong as well. The code for whoisshe is as follows:

printf("%s is a she\n", argv[1]);

And appropriately modified for whoishe.

The aim is to see how long our new joinee continues to test these tools... Wicked!

Hardy Heron - Linux getting better!

2008-10-03T20:15:00.002+05:30

A few months back, I had requested Canonical to dispatch me (they still do it for free!) a CD of Ubuntu Linux 8.04 LTS (hardy heron). I was still running an older version of the same distro for quite some time and this was an opportunity to change. I had installed the newer OS and these are a few observations:

Display is much better, with my ancient VIA chip-set getting recognized with 3D acceleration too. Now, I can play most of the 3D games (tuxkart, supertux, billards GL) and a few non 3D games (chromium) in full glory.
Wireless is still a problem. My netgear wireless adapter was able to recognize my ADSL router among the other routers in range in my neighborhood, but fails to connect frequently. But once connected, they blaze through at 200kbps. I have not been able to see this kind of speed on a windows machine though...
Audio CDs are mounted properly. This happened to be a problem with 7.04 when only data CDs were recognized.
Video / audio codecs are easier to download from the main application installer.

I happened to find more applications available on the Linux platform, especially for kids compared to windows. TuxPaint, TuxMath, celestica, GCompris are some must haves. And my son's current favorite is Pingus (a lemmings clone where you deal with penguins instead).

Of couse, you have the standard suite of tools for regular work (open office), CD burning, browsing (FireFox), e-mail (Evolution) and all this happened to work well on my 4 year old Athlon with 384 megs of RAM!

Keep watching... I'll post in a few screen-shots.

It's bright and sunny at Chennai, but the fuel queue is long

2008-07-04T22:42:00.003+05:30

All at Tamil Nadu would have faced the spurts of acute fuel shortage and chances are, you too, like me had spent some time on the queue waiting to fill your tank. The last few days had been 'exciting' looking out for every fuel station on the way to see a line of already queued vehicles waiting for the fuel tanker to arrive. The recent fuel hike just added to the woes.

We are accustomed to travel long distances, and imagine if for a week or two, the fuel tanker fails to arrive? A tank full of fuel may sustain you for a few days, but after that, you are left with your own foot (or other animals) for travel.

Public transport (fuel driven) will not function and as traveling by foot would be difficult to span the distance that we once used to travel in mere minutes. Bicycles will be in great demand and possibly, we would start considering bullock-carts and horse riding.

But, somehow, solar energy does not seem to be exploited by us, being in a very sunny place. Perhaps this free / cheap energy source that has been neglected for so long would see a spurt of new advancements to make fast fuel-less (Petrol / Diesel) travel a thing of the past.

20 Years Before - 1988

2008-06-27T23:30:00.010+05:30

When I had attended a course for GW BASIC. I already had a computer at home, and it was called the ZX Spectrum with, wait for it, 48k memory. Before formally learning the BASIC, I had restricted myself to a few prints and beeps and meticulously typing out program listings off computer books which eventually resulted in a star walled winding road with a car in the middle. The beeps always allowed you to specify the frequency and duration to compose a simple tune of your own.

The I/O was innovative with the ability to use standard audio cassette tapes to record your programs and load it back. Hmmm. The good old sequential storage medium. And mostly, I was using it for playing games. There was one called DLAN, a space simulator which I had spent countless hours trying to save the Earth and solar system from menacing alien fighters which featured a 3D environment along with a full solar system map, portholes and docking to a planet to refuel and carry out simple maintenance activities.

Later, I upgraded to a 128k machine with only one game that utilized the full memory which required to tap two keys as soon as possible to increase the power for some athletic activities like a 100 meter sprint or a tug-of-war against the computer counter part.

Bah, now a days, each game occupies more that a GB of disk and memory and provides the same level of experience.

The 128k also gave me an opportunity to learn about machine code language where I had to painstakingly type out hex byte-codes which resulted in a simple editor to allow you to directly program your computer using machine code. That was the time when I learned about machine instructions like move or registers.

We have come a long way until then...