Polo's Blog

Essential PhD skills

2013-04-12T23:02:00.003-07:00

http://www.phds.org/graduate-school-success/essential-skills/

Mentoring
Communication
Negotiation
Computer
Networking
Grant Writing
Professional Ethics
Management
Teaching
Marketing Yourself
Time Management

Must read for PhD students

2013-04-12T22:54:00.001-07:00

Probably the best few minutes you will spend in your PhD life. Life-changing advice!
How to have a bad career in research/academia by Prof. David Patterson

Automatically fix EPS bounding box on Mac, using ps2eps

2013-03-28T23:27:00.004-07:00

Shell script to import MySQL dumpfile into SQLite

2013-01-14T19:57:00.001-08:00

Download shell script: mysql2sqlite.sh

# Compiled by Polo Chau, by adapting Igor's StackOverflow answer on
# http://stackoverflow.com/questions/489277/script-to-convert-mysql-dump-sql-file-into-format-that-can-be-imported-into-sqli

#!/bin/sh
if [ "x$1" == "x" ]; then
echo ""
echo "Usage: $0 "
echo "This is a shell script that imports a MySQL dumpfile into an SQLite database."
echo "Tested on a Mac with OS X 10.8.2 (late 2012)"
echo "Creates 3 output files."
echo "1) out.db -- SQLite database created from the MySQL dumpfile"
echo "2) out.sql -- a cleaned-up version of the MySQL dumpfile, readable by SQLite"
echo "3) out.err -- SQLite's error messages generated during importation"
echo ""
exit
fi

cat $1 |

# delete the lines containing "KEY", "UNIQUE KEY", "PRIMARY KEY"
grep -v ' KEY "' |
grep -v ' UNIQUE KEY "' |
grep -v ' PRIMARY KEY ' |

# use sed to substitute mysql-specific keywords with general SQL keywords that sqlite knows
# tell sed to read the text as unicode text, by specifying LANG=en-US.UTF-8
LANG=en-US.UTF-8 sed '/^SET/d' |
LANG=en-US.UTF-8 sed 's/ unsigned / /g' |
LANG=en-US.UTF-8 sed 's/ auto_increment,/,/g' |
LANG=en-US.UTF-8 sed 's/ auto_increment/ primary key autoincrement/g' |
LANG=en-US.UTF-8 sed 's/ smallint([0-9]*) / integer /g' |
LANG=en-US.UTF-8 sed 's/ tinyint([0-9]*) / integer /g' |
LANG=en-US.UTF-8 sed 's/ int([0-9]*) / integer /g' |
LANG=en-US.UTF-8 sed 's/ character set [^ ]* / /g' |
LANG=en-US.UTF-8 sed 's/ enum([^)]*) / varchar(255) /g' |
LANG=en-US.UTF-8 sed 's/ on update [^,]*//g' |
LANG=en-US.UTF-8 sed 's/UNLOCK TABLES;//g' |
LANG=en-US.UTF-8 sed 's/LOCK TABLES `[a-zA-Z0-9]*` WRITE;//g' |
# tell perl to read the text as unicode text, by specifying -C
perl -C -e 'local $/;$_=<>;s/,\n\)/\n\)/gs;print "begin;\n";print;print "commit;\n"' |
perl -C -pe '
if (/^(INSERT.+?)$/) {
$a=$1;
s/\\'\''/'\'\''/g;
s/\\n/\n/g;
s/$,$/$;\n$a\(/g;
}
' > out.sql

# import the cleaned-up version of mysql dump into out.db sqlite database
# errors saved to out.err
cat out.sql | sqlite3 out.db &> out.err

Must read for (Java) programmers

2013-01-11T18:30:00.002-08:00

If you are a new programmer (e.g., high-school student, undergrad), and want to become a good one, you should read this:

If you're programming in Java, you should also read this:

You will benefit from them for the rest of your (programming) life!

Convert PNG/JPG to EPS on a Mac without creating a huge file

2013-01-11T18:22:00.002-08:00

The command for converting PNG/JPG to EPS are:
convert temp.png -compress lzw eps2:temp.eps
convert temp.jpg -compress jpeg eps2:temp.eps

The convert command comes from Mac's built-in imagemagick package.

If you type only convert temp.png temp.eps, it's gonna create the EPS file without compression, creating a huge file.

Best Cloth Rack for Apartments

2012-07-31T23:21:00.000-07:00

Rarely am I so satisfied with a piece of furniture. For this this cloth rack I am far more than satisfied.

I've owned it for 5 years. It's survived two moves, and still looks like new -- sturdy, shiny, and holds tons of clothes. You can even do pull-ups on it!

Best features:
* height adjustable: 48" to 60"
* hanging rod adjustable: 48" to 72"
* collapsible; you can pop the top hanging rod, and the two vertical pods will fold down to horizontal.
* sturdy; doesn't wobble a bit (plastic cloth racks do)
* can be used to transport large boxes! Stack the boxes on top of the two parallel rods at the bottom and there you go!

Search for the perfect belt is over!

2012-04-30T23:36:00.000-07:00

Reversible, reasonably priced, nice colors, and doesn't crack.

PowerPoint Mistakes -- from a usability standpoint

2012-04-23T13:08:00.001-07:00

I saw a presentation today. Some of its fonts were really tiny.
That prompted me to google "powerpoint mistakes". Got many hits, including this funny and informative YouTube video by Don McMillan.

Interestingly, these sources often only tell you what to, or what not to do, but don't really tell you why.
For example, some would tell you "don't read the slides", "use a minimal font size of 24", "don't use more than 7 bullets on a slide", "don't use dark color text on dark background", etc.

But what are the common rationale behind all these dos and don'ts?
Usability -- that is whether it is easy for your audience to "use" (read, understand, see, etc.) your slides.

If you think about your slides' usability as the first priority, then I expect you can naturally fix many of your slides' problems. For example, you won't:

use fonts smaller than 24, because audience in the last row of a room can't see the text
use more than 7 bullets, slide gets visually complex, straining your audience's eyes
use dark color text on dark background, because your audience will find it hard to read
read slides, because your audience will find you boring,
and the list goes on...

I think it's much better to only remember one thing -- the principle of good slide usability -- rather than having to remember a laundry list of dos and don'ts.

Usability is one focus of within the research area of Human-Computer Interaction. I think it also applies to slide design.

Aggregate bitwise OR in SQLite

2012-03-27T20:46:00.001-07:00

The following SQL snippet applies bitwise OR on all the values in numbers table's value column. This example supports up to 8 bits; you can easily extend it to support more.

CREATE TEMP TABLE numbers(value INTEGER);
INSERT INTO numbers VALUES(2);
INSERT INTO numbers VALUES(3);
INSERT INTO numbers VALUES(15);

select
((sum(value&1)>0) << 0) +
((sum(value&2)>0) << 1) +
((sum(value&4)>0) << 2) +
((sum(value&8)>0) << 3) +
((sum(value&16)>0) << 4) +
((sum(value&32)>0) << 5) +
((sum(value&64)>0) << 6) +
((sum(value&128)>0) << 7)
from numbers;

The Best, Easy, Delicious Desserts for Potluck Parties

2012-01-14T16:05:00.000-08:00

I didn't like potluck parties. I didn't like how my home-cook dishes look like. They're just not appealing.

One day, I decided I had to fix this --- to learn to make one thing that I could bring. Something that's delicious, easy to make, *and* unlikely that other people would bring.

So I found this 5-star Brownie Chocolate Chip Cheesecake recipe.
It needs very few ingredients, and you basically just mix them together, and it tastes fantastic. Brownie bottom layer, plus cream cheese top layer filled with chocolate chips. Yummy. None of my friends could resist. (Mr. V, you should try it, but don't eat so much. Watch your tummy.)

And I'm using this springform pan. Sturdy and doesn't leak. Hasn't failed me for 1.5 years!

This Christmas, I learned another recipe: Rice Krispies Treat. It needs only 3 ingredients, and 10 min to make (really 10 min, unlike other recipes that lie). Delicious. A good dessert to make when I don't have much time. There's Kellogg's version, but I like my friend's ingredient proportion better:

1/4 cup butter
4 cups marshmallows
5 cups Rice Kripies cereal

I use one of these nicely-built pyrex containers (3 QT) to hold the rice kris pies.

Amazon.com Widgets

Pittsburgh (Being Grad Students)

2012-01-13T01:37:00.000-08:00

Before I came to Pittsburgh for grad student, someone told me there's nothing to do here. And I would freeze to death.

Well, it's certainly different from Hong Kong. But it's a great city. It's medium sized, and you can find a bit of everything, without the craziness of big cities.

First off, CMU has a diverse student population, with students coming from every parts of the world. I haven't counted, but I'm sure my friends come from more than 20 counties. (Just counted, it's roughly 25!)

Pittsburgh has a fantastic football team (Steelers) and an ice hockey team (Penguins). Ahem, and a baseball team, the Pirates. Sorry to say, but the Pirates hasn't been doing that well. Yet, I have only been to Pirates games. Their tickets are cheap.

Whoever says Pittsburgh is cold lies. Don't listen to him. Or her. I have traveled to other cities in the states that are much colder. Try those cities north of Pittsburgh closer to the lake, and you'll know what real cold is! And of course there are even colder ones further north. And it actually doesn't snow that much in the winter. But I never have to shovel snow.

The nicest thing about Pittsburgh is perhaps its cheap housing. As a grad student, I could have afford to buy a house. Though I didn't. And I regret. Many of my friends bought those. Good investment indeed. The Pittsburgh housing market wasn't affected much at all in 2008. People here really buy houses to live in them.

More later.

Steamer instead of iron (Being Grad Students)

2012-01-11T13:59:00.000-08:00

No one likes to iron clothes. It takes so much effort and space. Clearing space in my tiny apartment so I can open up the ironing board is already a chore. Then you warm up the iron. All you want is to just iron one shirt!
Then lately, I learned of a much better alternative -- using a steamer! No ironing board needed. It's fast to heat up the steam (and it only uses water). It has a small footprint. And since it stands upright, you can hang clothes on it. Plus, you can use it to sterilize curtains and other things. Handy.

Decorating student apartment (Being Grad Students)

2012-01-09T11:33:00.000-08:00

If you're like me, a grad student renting an apartment, chances are that you have some rather bare (and boring) walls, because your landlord (or landlady) warned you not to hammer any nails, or they will take a big chuck out from your security deposit.

How do you hang a clock then? What about some frame photos?

I found some fantastic "command hooks" by 3M which can be removed from the wall cleanly. No glue left behind. They also won't pulled wallpaper (unless it is peeling anyways). 3M also sells strips for hanging pictures, among other "command" products.

Thumbs up.

Choose conferences with nice locations (Being Grad Students)

2012-01-08T09:59:00.000-08:00

One of the nicest being a grad student is you get to travel, often "for free", to conference held at great places (e.g., Sydney, Paris, Hong Kong, etc.).

Picking such a conference could motivate you to work hard for a paper. :-) So choose your conference wisely. After all. If you have some good work that you can submit to similar conferences that have comparable reputation/prestige, which conference would you send it to? The nicer one of course.

Just remember to check with you advisor that it's OK with him/her though.

Cut your own hair (Being Grad Students)

2012-01-07T22:00:00.000-08:00

When I was little, my father cut my hair. Then I came to the states for grad school -- no one cut my hair anymore!

Going to a hair salon in the US can be so expensive. The costs add up really quickly. So I thought about doing it myself; I remembered my father saying "it's a piece of cake" cutting my hair. How bad could it be if I did it myself?

Very bad, when I first started, a few years back. I bought a cheap Wahl hair clipper (like this one), and I hated it, even though it's a 5 star product.
1) It has a long cord, which often get in the way
2) It's really awkward holding it to cut the side or the back of my head -- just try using your right hand to scratch your left ear from behind your head, and you'll know what I mean.
3) It's a big hassle having to keep changing the "length guides" for different hair lengths.

Apparently, you need something different when you cut your own hair. I found this Philips hair clipper recently, which solved all the above problems!!
1) It's cordless.
2) It has a rotating head, so you can hold it sideway when you cut your hair on the side or the back of your head
3) It has a "zoom ring" that lets you change the cutting length from 30mm (about 1.18 in) down to super close shave. No need to fiddle with the millions of "length guides"! (Well, technically, there are still two guides, one for the longer lengths, one for shorter lengths.)

I love this clipper. Great for a quick fix.

For those of you (men) who haven't cut your own hair before. It's very easy (assuming you use this Philips clipper, and you just want to style your hair a little). Here's how:

1a) Pick the longest length you want your hair to be (e.g., 30mm), and run the clipper through your whole head.
1b) If you want to style the top part of your hair (e.g., for parting, or styling), leave that part longer (or don't cut that at all) for later.
2) Pick a shorter one (e.g., 25cm) and run it along the areas close to the ear, and the back of your neck.
3) Repeat step 2, with even shorter length if desired. The key is to "blend" different lengths of hairs.
4) To add "layers" to your hair (to make your hair more "choppy" and to fine tune it to frame your face), use a pair of scissors, and cut "into" your hair like this.

Of course, if you need a really clean cut, or fancy style, you'll have to go to a hair salon.

Grocery shopping, folding t shirt (Being Grad Students)

2011-01-05T11:48:00.000-08:00

1. When grocery shopping, instead of doing all the math in your head about which brand of food (or toilet paper if you may) is cheaper, you can often look at the tiny text close to the prominent sale price. For food, it's often stated as $XXX per ounce (and for toilet paper, $YYY per roll). This save you some brain cells, and some gold coins in your wallet.

2. Fold a t shirt in 2 sec. (This makes the chore slightly more bearable.)

http://youtu.be/An0mFZ3enhM

Creating decorators in the Prefuse framework

2009-08-13T20:03:00.000-07:00

I'm working on some graph visualization using Prefuse, and I want to allow the use to select nodes in the graph by dragging a marquee. In addition, I want to show a tiny black square indicator at the top-left corner of a selected node. The image on the left is what I eventually got.

I wanted to use the Decorator design pattern provided by Prefuse to add the square indicators. Conceptually, the steps are:

programatically selects a subset of the nodes in the graph
applies the desired decoration on those subset of nodes

But in the actual implementation, these basic steps unfold into much more complex steps:

creates a "focus group", of type "DefaultTupleSet", to the "visualization" object, using the method visualization.addFocusGroup(); you'll need to give this newly created group a name. This group (which is a "tuple set") is precisely the data structure that you use to store the selected nodes
look at this example on how to create a control that lets you draw the rectangle marquee, which will put the selected nodes into the "focus group" that you created in step 1.
now comes the nontrivial part -- creating the "decorators", which are objects created to decorate the selected nodes. Programmatically, you call visualization.addDecorators() to create this new group of decorators. Again, you give this decorator group a name.
creates a layout action to continously relocate the indicator so that it always stay at the top-left of the node
creates a renderer to render the tiny square. I think we can use a ShapeRenderer instead so this step could be skipped.

NEVER USE "graph.nodes.xxx.yyy.zzz" AS A GROUP'S NAME. IT BREAKS PREFUSE. I haven't had the time to figure out why, probably because of some of its internal parsing error, but this kind of names break Prefuse. I learned about this the hard way; wasted 4+ hours.

Using BitSet or BitVector to store a set of integers

2009-05-25T14:53:00.000-07:00

When programming with Java, we typically use HashSet to store a set of Integer, if we want to keep track of which Integer has been used and which hasn't. When we need to store millions of Integers, HashSet can be too slow, and it may take up too much memory.

Java's BitSet and BitVector from COLT are great alternative; they are MUCH faster and consumes way less memory. For example, to store 20 million integers, both BitSet and BitVector only use about 2.5MB and take only, respectively, 0.67s and 0.27s (which means BitVector is the fastest) to set and get all of the values once. using HashSet, it takes ~962MB, and 6.3s. This is more than 380 times of memory saving, and ~20 times speed up.

One "disadvantage" for BitVector is that it's size can't dynamically scale, but you can always fake it by manually growing or shrinking the set via setSize(...)

I can't figure out why Java's HashSet implementation uses so much RAM; I tried to reduce the size to 2 million, and that uses 100MB, approximately 1/10 of the size when having 20 millions. So this means indeed the Java implementation is memory-hungry. I'm sure there's some way to cut down the memory consumption, but I haven't spent the time to investigate.

Speed Comparison of: (1) Java's built-in HashMap, (2) Trove's TIntIntHashMap, and (3) Colt's OpenIntIntHashMap

2009-05-09T22:29:00.000-07:00

Finally get to do a comparison among several popular Java HashMap implementations:

Java's built-in HashMap
Trove's implementation for primitive types
Colt's implementation for primitive types

Not too surprisingly, Java's built-in implementation is the slowest, and consumes the largest amount of memory among the three, since it stores objects instead of primitive types (e.g., instead of storing the number 3 as an int, it stores it as an Integer). What WAS surprisingly, however, was that Colt's implementation was SO MUCH faster than Java's implementation. For example, for a int->int hashmap, running 10,000,000 puts() with Colt took less than 1/3 of the time. Running 10,000,000 gets() took less than 1/6, and 10,000,000 containsKey() less than 1/5! This is some substantial timing saving. This time-saving advantage is still there, even when we use a int->float[] hashmap.

Memory-wise, the primitive implementations from Trove and Colt don't save us that much. For int->int hashmap, both Trove and Colt save us about 30% of memory, but for int->float[] (each float[] value has length 5) save us less than 10%. Nevertheless, there's still some saving.

Since I need to deal with large data sets, where it's common to use hashmaps with millions of entries, I'll use the Colt implementation from now on, for both it's impressive time saving and (slighly) lower memory consumption.

Here's the code I used for evaluating int->float[] hashmap. To get the timing and memory consumption for one implementation, comment out the other two that you DON'T want. You will also need to pass in these arguments to the VM, so enough heap memory will be alocated "-Xms1200M -Xmx1200M".


import java.util.HashMap;

import cern.colt.map.OpenIntIntHashMap;
import cern.colt.map.OpenIntObjectHashMap;
import gnu.trove.TIntIntHashMap;
import gnu.trove.TIntObjectHashMap;


public class Compare {
 
 public static void main(String args[]){
  
  System.out.println("1st line: time used(s)\n2nd line: heap memory used so far(MB)");
  
  int n = 10000000;
  
  long startTime = System.nanoTime(); 
  long startHeapSize = Runtime.getRuntime().freeMemory();

  
  // BEGIN: benchmark for Java's built-in hashmap
  System.out.println("\n===== Java's built-in HashMap =====");
  HashMap jIntIntMap = new HashMap();

  System.out.println("\n-- " + n + " puts(key, value) --");
  startTime = System.nanoTime(); 
  for (int i = 0; i < n; i++) { jIntIntMap.put(i,new float[]{0f,1f,2f,3f,4f}); }
  System.out.println( (System.nanoTime() - startTime) / 1000000000.0 );
  System.out.println( (startHeapSize - Runtime.getRuntime().freeMemory()) /1048576.0  );  
  
  System.out.println("\n-- " + n + " gets(key) --");
  startTime = System.nanoTime(); 
  for (int i = 0; i < n; i++) { jIntIntMap.get(i); }
  System.out.println( (System.nanoTime() - startTime) / 1000000000.0 );
  System.out.println( (startHeapSize - Runtime.getRuntime().freeMemory()) /1048576.0  );  

  System.out.println("\n-- " + n + " containsKey(key) --");
  startTime = System.nanoTime(); 
  for (int i = 0; i < n; i++) { jIntIntMap.containsKey(i); }
  System.out.println( (System.nanoTime() - startTime) / 1000000000.0 );
  System.out.println( (startHeapSize - Runtime.getRuntime().freeMemory()) /1048576.0  );  
  // END  
  
  
  // BEGIN: benchmark for Trove's TIntIntHashMap
  System.out.println("\n===== Trove's TIntIntHashMap =====");
  TIntObjectHashMap tIntIntMap = new TIntObjectHashMap();

  System.out.println("\n-- " + n + " puts(key, value) --");
  startTime = System.nanoTime(); 
  for (int i = 0; i < n; i++) { tIntIntMap.put(i,new float[]{0f,1f,2f,3f,4f}); }
  System.out.println( (System.nanoTime() - startTime) / 1000000000.0 );
  System.out.println( (startHeapSize - Runtime.getRuntime().freeMemory()) /1048576.0  );  
  
  System.out.println("\n-- " + n + " gets(key) --");
  startTime = System.nanoTime(); 
  for (int i = 0; i < n; i++) { tIntIntMap.get(i); }
  System.out.println( (System.nanoTime() - startTime) / 1000000000.0 );
  System.out.println( (startHeapSize - Runtime.getRuntime().freeMemory()) /1048576.0  );  

  System.out.println("\n-- " + n + " containsKey(key) --");
  startTime = System.nanoTime(); 
  for (int i = 0; i < n; i++) { tIntIntMap.containsKey(i); }
  System.out.println( (System.nanoTime() - startTime) / 1000000000.0 );
  System.out.println( (startHeapSize - Runtime.getRuntime().freeMemory()) /1048576.0  );  
  // END     
  
  // BEGIN: benchmark for Colt's OpenIntIntHashMap
  System.out.println("\n===== Colt's OpenIntIntHashMap =====");
  OpenIntObjectHashMap cIntIntMap = new OpenIntObjectHashMap();

  System.out.println("\n-- " + n + " puts(key, value) --");
  startTime = System.nanoTime(); 
  for (int i = 0; i < n; i++) { cIntIntMap.put(i,new float[]{0f,1f,2f,3f,4f}); }
  System.out.println( (System.nanoTime() - startTime) / 1000000000.0 );
  System.out.println( (startHeapSize - Runtime.getRuntime().freeMemory()) /1048576.0  );  
  
  System.out.println("\n-- " + n + " gets(key) --");
  startTime = System.nanoTime(); 
  for (int i = 0; i < n; i++) { cIntIntMap.get(i); }
  System.out.println( (System.nanoTime() - startTime) / 1000000000.0 );
  System.out.println( (startHeapSize - Runtime.getRuntime().freeMemory()) /1048576.0  );  

  System.out.println("\n-- " + n + " containsKey(key) --");
  startTime = System.nanoTime(); 
  for (int i = 0; i < n; i++) { cIntIntMap.containsKey(i); }
  System.out.println( (System.nanoTime() - startTime) / 1000000000.0 );
  System.out.println( (startHeapSize - Runtime.getRuntime().freeMemory()) /1048576.0  );  
  // END    
  
  
 }

}

How to write a great research paper

2009-01-23T12:33:00.000-08:00

A great guide by Simon Peyton Jones

Embedding web browser in Java application

2008-12-03T13:40:00.000-08:00

I found this NativeSwing library an excellent library for embedding a web browser in a Java application, seemingly easier to use and much better documented that JDIC.

Primitive Java Collections (primitive hashset, etc.)

2008-11-25T22:16:00.000-08:00

Java's built-in collections such as hashmap, hashset, etc are not efficient when you have a lot of data (e.g., 1 million of items), it's because everything in the hashmap (i.e. its keys and values) and hashset (i.e. its values) are stored as objects, which are expensive to store and access. So if you want to store, say, 1 million int in a hashset, use a primitive hashset instead, such as Trove, or COLT. They seem to be some of the best primitive collections. But I haven't compared their performace (memory consumption and access speed) so I don't know which one is better.

Downloading full CiteSeerX data

2008-11-18T18:49:00.000-08:00

Here is, in my opinion, the easiest way to download the full dataset from CiteSeerX. (Note that CiteSeer is the older version, which is no longer updated.)

Steps for downloading the full dataset from CiteSeerX:

Download and extract the "Demo" from http://www.oclc.org/research/software/oai/harvester.htm
Go to the directory of the extracted files, type the following command to download the full dataset of CiteSeerX to the file "citeseerx_alldata.xml"
java -classpath .;oaiharvester.jar;xerces.jar org.acme.oai.OAIReaderRawDump http://citeseerx.ist.psu.edu/oai2 -o citeseerx_alldata.xml

I also wanted to thank Dr. Lee Giles from the CiteSeerX project for pointing me at the right directions to obtain the data and recommending me to try some Perl harvesters (though I didn't eventually use them).

People won't buy what they don't understand

2008-02-22T13:00:00.000-08:00

"People won't buy what they don't understand"
A great sentence! That's why we need to write plain English. :-)