Swimming in bits

Ruby SQLite3 memory tuning

2015-12-22T15:29:00.002-05:00

0. Motivation

SQLite database is designed as an embedded db, the default settings of memory footprint, page size are biased to workload with moderate data sizes. In a recent project, I used sqlite to implement some path generators on a graph dataset, in order to scale the program without worrying about out of memory issues, e.g. 3-hop paths in a 100K graph would go beyond 20G easily, external memory algorithms need to be implemented, writing a SQL and writing my own c code basically will have similar performance. However, both ruby and sqlite are not supposed to do this type of work under the default settings. The post contains some settings one needs to do in order to have better performance.

1. Ruby

One should set the GC using the following VARs. (ruby 1.9+, 2.0)

export RUBY_HEAP_MIN_SLOTS=8000000
export RUBY_FREE_MIN=327680
export RUBY_GC_MALLOC_LIMIT=512000000

On details of Ruby GC, and the effect of these environment vars, see this page:
http://tmm1.net/ruby21-rgengc/

Other useful links

reference setting in real world rails apps: https://gist.github.com/jjb/7389552

2. SQLite

to use more memory and avoid hitting disk, one should set the following PRAGMA
cache_size: basically the number of b-tree pages can be held in memory.
temp_store: temporary results in memory db or in file

this post is nice in this topic:
http://www.codificar.com.br/blog/sqlite-optimization-faq/

My app is read-only on IDBs, so the synchronous PRAGMA is not useful. But even for R/W applications, be careful to set it.

In Ruby SQLite3 binding, the PRAGMAs are included in the Database class.
After getting the db instance, one can just set:

db.cache_size = 512000 # 500M
db.temp_store = 2 # memory

Other useful links on this topic:

ruby sqlite3 doc:

sqlite3 pragma doc:

https://www.sqlite.org/pragma.html

Generate zipf distribution in C++

2014-04-07T21:00:00.000-04:00

Standard library and Gnu scientific library GSL, do not provide a way to get the probability mass function of the zipf distribution directly. By looking around from wiki and stackoverflow post, implemented a couple lines of code for zipf pmf:


#include 

 

// when one doesn't know the population size

// assume the population is infinity

// then the generalized harmonic number

// can be approximated as zeta function

double get_pmf_zipf(int rank, double skew) {

 // use zeta function, one can approximate the zipf pmf

 return pow(rank, -1*skew)/gsl_sf_zeta(skew);

}

 

// when one knows the population

// we can calculate the generalized harmonic number

double get_pmf_zipf(int rank, int N, double skew) {

 double base = 0;

 for (int k = 1; k <= N; k++) base += pow(k, -1*skew);

 return pow(rank, -1*skew)/base;

}

bold math formula in latex

2013-10-02T17:34:00.005-04:00

make it bold:

from this url: http://tex.stackexchange.com/questions/25428/how-to-bold-math-mode-characters

Try the bm package:

\documentclass{article}
\usepackage{bm}
\begin{document}
$\bm{\langle \rangle}$
\end{document}

remove margins of gnuplot, subfigures in latex

2013-06-19T19:18:00.001-04:00

remove margin of gnuplot generated eps is useful when one wants to squeeze more figures in a subfigure block.

here's how to do it:

1. plot the eps using gnuplot, say x.eps

2. use eps2eps : eps2eps x.eps x_compact.eps
see more here: http://webthreeoh.wordpress.com/2013/02/24/removing-margins-in-gnuplot/

3. the quality of eps2eps may be not ideal because of the compression, to control it:
try: eps2eps -r600 -dEPSCrop x.eps x_compact.eps
see more here: http://electron.mit.edu/~gsteele/pdf/

4. you may want to automate it, in a gnuplot script, you can call system function as well,
try include this at the end of gnuplot script:
`eps2eps -r600 -dEPSCrop x.eps x_compact.eps`

see more here: http://stackoverflow.com/questions/12846717/using-awk-or-other-shell-command-inside-gnuplot-function

5. finally use a subfigure, you may want to squeeze space:
try:
\begin{figure}
\vspace{-15pt}
\subfigure[fig1]{
\includegraphics[totalheight=0.13\textheight]{fig1.eps}
\label{fig:label1}
}
\subfigure[fig2]{
\includegraphics[totalheight=0.13 \textheight]{fig2.eps}
\label{fig:label1}
}
\label{fig:subfig1}
\caption{Title here}
\vspace{-15pt}
\end{figure}
see more from subfigure package document.

Links to make a conference poster,

2013-06-03T15:40:00.000-04:00

Some useful links:

using latex:
The LaTeX beamerposter package
http://www-i6.informatik.rwth-aachen.de/~dreuw/latexbeamerposter.php

using ppt:
http://colinpurrington.com/tips/academic/posterdesign

paper size:
http://en.wikipedia.org/wiki/ISO_216

some instructions:
http://www.icde2011.org/node/103

trademarks:
http://www.trademarks.umd.edu/marks/inst.cfm

linqs guides:
https://wiki.umiacs.umd.edu/linqswiki/index.php/Research:poster:makeposter

quite note: generate a er chart and schema with schemaSpy

2013-04-21T21:46:00.002-04:00

working on some complicated legacy db recently. wants to generate the er diagram to get to know details.

the mysql-workbench works very well, however it's slow in my setting. also the schema has little foreign key constrains as it is almost read-only db. the er reverse engineering feature in workbench is not that useful.

did some google, found the tool: schemaspy, it's standalone jar and easy to use.
http://schemaspy.sourceforge.net/

for mysql:
java -jar schemaSpy_5.0.0.jar -cp mysql-driver/mysql-connector-java-5.1.24-bin.jar -t mysql -host localhost -db dbname -u user -p pwd -o outputdir

how to setup an mpi cluster (mpich2, and openmpi)

2013-04-20T18:11:00.000-04:00

quick memo of some good posts:

to set up the mpi cluster, follow these steps:

1. use keygen to configure the ssh environment in the cluster
https://source.ggy.bris.ac.uk/wiki/Configure_ssh_for_MPI
if all machine shares the home directory, it will make life much easier.

2. if one wants to use MPICH2, follow this link to compile and install
download, 1.4.1 should be fine: http://www.mpich.org/downloads/
(http://www.mpich.org/static/tarballs/1.4.1/)
http://heather.cs.ucdavis.edu/~matloff/MPI/NotesMPICH.NM.html
http://source.ggy.bris.ac.uk/mediawiki/index.php?title=Install_and_configure_MPI&redirect=no

3. write a host file, and run it in the cluster with ease
(note, mpd is dropped, it uses hydra instead, which will be run using mpiexec)
for sophisticated host configuration, see this very nice post:
http://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager

etc. if one don't have permissions to install things everywhere, but just local machine
try to install in local machine, and use .bash_profile / .bash_rc to compile mpi to one's shared directory. so remote machine can use with ease from shared directory without install it.

etc. for debugging:
this one might be nice.
http://heather.cs.ucdavis.edu/~matloff/pardebug.html

For OpenMPI cluster, step 1,2 is the same, difference are:
d1. host file
hostfile format is different from mpich2:
instead of : machine_a:2

it uses: machine_a slots=2

here's some example
https://wiki.thayer.dartmouth.edu/display/computing/Linux+MPI#LinuxMPI-HostFile

d2. orted: command not found error
error message may occur if one try to run on remote machine
e.g.

sh: orted: command not found
--------------------------------------------------------------------------
A daemon (pid 6797) died unexpectedly with status 127 while attempting
to launch so we are aborting.

There may be more information reported by the environment (see above).

This may be because the daemon was unable to find all the needed shared
libraries on the remote node. You may set your LD_LIBRARY_PATH to have the
location of the shared libraries on the remote nodes and this will
automatically be forwarded to the remote nodes.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that the job aborted, but has no info as to the process
that caused that situation.
--------------------------------------------------------------------------
mpirun: clean termination accomplished

to solve it,
-> make sure, MPI_HOME/bin is in PATH, MPI_HOME/lib is in LD_LIBRARY_PATH

and, make sure when run the mpirun / mpiexec, using absolute path to run it:
/path/to/mpirun -np 2 ...
see a thread here: link

plotting time series using gnuplot

2013-04-17T00:48:00.003-04:00

This post by Ron Peterson is really good. Thanks for the great code!
http://linuxgazette.net/126/peterson.html

Also It's worth pointing out the time format definition follows the following rules:

http://gnuplot.sourceforge.net/docs_4.2/node274.html

Time Series timedata Format Specifiers
Format	Explanation
`%d`	day of the month, 1-31
`%m`	month of the year, 1-12
`%y`	year, 0-99
`%Y`	year, 4-digit
`%j`	day of the year, 1-365
`%H`	hour, 0-24
`%M`	minute, 0-60
`%s`	seconds since the Unix epoch (1970-01-01 00:00 UTC)
`%S`	second, 0-60
`%b`	three-character abbreviation of the name of the month
`%B`	name of the month

change the separator / delimiter :

set datafile separator ","

Following text are all from original post by Ron Peterson:

Plotting time series data with Gnuplot

By Ron Peterson

Introduction

Good systems administrators log stuff. Lots of stuff. A lot of the information we collect consists of time series data: a set of numerical values assocated with a sequence of discrete time values.

There are any number of tools to help the diligent sysadmin monitor this data visually as it is collected. A good many of them are built using Tobias Oetiker's excellentRRDTool. Some noteworthy examples include Cacti, Cricket, and Smokeping. There are many others.

That's all well and good as long as you know what you want to monitor. However, sometimes you'd just like to do some quick ad hoc visualization. As you might surmise, most Linux systems provide a myriad of visualization tools (Grace and GRI come to mind). In this article, I'll introduce you to Gnuplot, focusing specifically on how to plot time series data.

Prepare some data

Gnuplot without data is like gravy without potatoes. So before we get to the gravy, let's make some potatoes. Let's say for the sake of argument, or at least for the purpose of giving the rest of the article something to talk about, I include the following line in my system's crontab file:

*/1 * * * * root /bin/cat /proc/loadavg 2>&1 | /usr/bin/logger -p local3.info -t CRON-LOADAVG

If you're like me, and have configured your system's syslog.conf as follows:

local3.*  /var/log/cron.log

...then you will find all local3 facility messages in their own special file. Because we're telling 'logger' to tag all of our load average data, it will be easy to extract this information from the rest of our logfile clutter. A simple 'grep CRON-LOADAVG /var/log/cron.log > load.dat.1' should do nicely. This will give us a file that looks like so:

Mar 19 00:30:02 ahost CRON-LOADAVG: 0.40 0.78 1.19 11/296 3690
Mar 19 00:31:01 ahost CRON-LOADAVG: 3.54 1.55 1.41 4/311 3997
Mar 19 00:32:01 ahost CRON-LOADAVG: 2.68 1.59 1.43 2/278 4142
...

Now let's extract just the data we want:

cat load.dat.1 | tr -s ' ' ' ' | cut -d' ' -f1,2,3,6 > load.dat.2

The translate command 'tr' squishes multiple spaces into a single space, so that we can expect more consistent behaviour from the 'cut' command. In this case, the translate command 'tr' is superfluous, but I think it's a good habit nonetheless. With any luck, our data now looks something like:

Mar 19 00:30:02 0.40
Mar 19 00:31:01 3.54
Mar 19 00:32:01 2.68
...

That's almost perfect. Unfortunately, our gnuplot example will expect two space delimited columns of input, so we need to replace the spaces delimiting our timestamp components with some other character, like a hyphen.

perl -pe 's/(.*?)\s(.*?)\s(.*)/$1-$2-$3/;' load.dat.2 > load.dat.3

This isn't a Perl article, so I won't bore you with the details of what this command is doing. In the interest of pedagogy though, I think it's helpful to illustrate how sausages are sometimes made; even if it does make me look like a butcher. Our data now looks like:

Mar-19-00:30:02 0.40
Mar-19-00:31:01 3.54
Mar-19-00:32:01 2.68
...

Plot it

Now it's time for the gravy. First I'll give you a taste, and then I'll explain the recipe. Create a file with the following contents, excluding the line numbers. Call it 'plot-load.conf'. Edit the date range on line six to include the extents of your data.

1  set terminal png size 1200,800
2  set xdata time
3  set timefmt "%b-%d-%H:%M:%S"
4  set output "load.png"
5  # time range must be in same format as data file
6  set xrange ["Mar-25-00:00:00":"Mar-26-00:00:00"]
7  set yrange [0:50]
8  set grid
9  set xlabel "Date\\nTime"
10 set ylabel "Load"
11 set title "Load Averages"
12 set key left box
13 plot "load.dat.3" using 1:2 index 0 title "ahost" with lines

If you run the following command, you should end up with a file called 'load.png'. Use your favorite image viewer to take a look. Hopefully nothing too alarming shows up.

cat plot-load.conf | gnuplot

The first line of our gnuplot command file says to create a PNG file, and gives its dimensions. PNG is only one of a myriad possible output formats. The second line says our X axis represents time data. The third line uses standard date format specification (see 'man date') to indicate what our data file's timestamp data looks like. We must use the same format in line six, where we indicate our graph's start time and end time. You can omit this, but I find it's useful to anchor the endpoints, particularly when plotting multiple data sources in a single graph. Line seven sets the plot limits of our Y axis.

Line 13 deserves a little bit of extra attention. The name of our data source comes first. The 'using 1:2' bit means to extract data from columns one and two of our data source. The 'index 0' bit means to use the first data set in the file. Data sets are delimited by pairs of blank records. Our file was simple. It only comprised col1 and col2 of data set zero in the following pseudo data file.

# data set zero
col1 col2 col3 col4
col1 col2 col3 col4
col1 col2 col3 col4


# data set one
col1 col2 col3 col4
col1 col2 col3 col4
col1 col2 col3 col4
col1 col2 col3 col4


# data set two
col1 col2 col3 col4
col1 col2 col3 col4
col1 col2 col3 col4

Asuuming we had multiple data sets in a single file (perhaps we want to compare load averages from multiple hosts), one way we could combine this data into a single graph would be to expand our line 13 as follows:

plot "load.dat.3" using 1:2 index 0 title "ahost" with lines, \
plot "load.dat.3" using 1:2 index 1 title "bhost" with lines, \
plot "load.dat.3" using 1:2 index 2 title "chost" with lines

Conclusion

Potatoes are nice, but as Trotsky once noted, they are "the classic symbol of poverty". Knowing how to quickly whip up some time series plots is useful, but Gnuplot is capable of far more than I've even hinted at in this article. Hopefully I've managed to whet your appetite to learn even more.

Best.

freebase basics

2013-04-14T23:26:00.001-04:00

Where to start:
Freebase wiki: http://wiki.freebase.com/

Terminologies:

Conceptual level:
In freebase, ontology is called Schema

Is A : Type / unary predicate / category / class
Has A : Property / predicate / relation

Implementation/Physical level:

A Domain is a collection of Types (share a Namespace, /type/domain).
A Base is a kind of Domain a way of grouping related types and their permissions.
A Namespace is a directory of uniquely named objects. Namespaces are organized in a hierarchy (like a path). It is used to manage keys and ids. It's more or like a table name of logical/physical database tables of freebase, e.g. namespace:

/type (primitive type, e.g. /type/object, /type/dated_integer),
/authority, (e.g. /authority/iata, /authority/gnis, /authority/musicbrainz, /authority/imdb ... like a datasource)
Common top level domains, e.g. /common, /people, /business, /books
/user, /base,
/guid, /wikipedia/en_id, /lang, /wikipedia, /en (human readable ids), /wikipedia/en
internal: /freebase

Misc:

a Domain has many administrators, who has the permission to add Property

Data & Tool:

Schemas: http://schemas.freebaseapps.com/
Data: https://developers.google.com/freebase/data

JavaCC to generate C++ code

2013-03-26T22:22:00.000-04:00

JavaCC has a developing branch that is able to generate C++ code.
svn address: https://svn.java.net/svn/javacc~svn

There is little examples so far. Some useful links from the mailing list:
how to use, just set the following in the beginning:
-output_language=c++
and write code fragments in jj file using c++
http://java.net/projects/javacc/lists/users/archive/2011-03/message/0

must read, most of it still true:
http://java.net/projects/javacc/lists/users/archive/2011-02/message/10

there's a example in the branch in the following directory
examples/JavaGrammars/cpp
the example parses java1.1 grammar

An example for a simple Adder.


//99+42+0+15

options {

JAVA_UNICODE_ESCAPE = true;

OUTPUT_LANGUAGE = "c++";

NAMESPACE = "adder_parser";

OUTPUT_DIRECTORY = "gen";

//STATIC = false; // ignored by c++

}



// ignored by c++ version

PARSER_BEGIN(Adder)

struct sampleCustomStruct {

int left_operand;

int right_operand;

};

PARSER_END(Adder)



// define the lexical analyzer

SKIP : { " "}

SKIP : { "\n" | "\r" | "\r\n" }

TOKEN : { < PLUS : "+" > }

TOKEN : { < NUMBER : (["0"-"9"])+ > }



// define the parser

int eval():

{

int i; 

  int value;

}

{

value = numberHanlder()

{ printf("value = %d\n", value); }

(

<PLUS>

i = numberHanlder()

{ value += i; }

{ printf("value = %d\n", value); }

)* <EOF>

{ return  value; }

}



// a sub routine in the parser

int numberHanlder():

{

// note it should be a pointer

Token* t;

}

{

t = <NUMBER>

{ return atoi(t->image.c_str()); } 

}

Several Notes:
1. PARSER_BEGIN - PARSER_END
The code in this section will be copied directly into the Adder class.
So #include clause will cause error.
It's possible to use this section define nested class, which may be useful in some case.

2. Token -> Token*
All reference to Token in Java should be declared as Token* in c++.
The token.image is an instance of basic_string, which is subclass of std string.

3. #include some external file or self-defined struct
use PARSER_INCLUDES in options section.
In the .h file, one can include more other headers.

4. Error Handling
Different from Java, no exception handling.
In the generated parser, one can do the following


ErrorHandler errorHandler;

parser.setErrorHandler(&errorHandler);

To set a error handler. It has two methods in ErrorHandler, one is parser error, another is lexical error. Inherit that class, can change its behavior.

Notes about query plans

2013-03-04T02:10:00.001-05:00

Left deep and right deep query plan tree are different for nested joins as well as hash joins:

about the nested join:

The classic paper mentioned it:
Goetz Graefe: Query Evaluation Techniques for Large Databases. ACM Comput. Surv. 25(2): 73-170 (1993), link

Figure 4 shows prototypical left-deep,
right-deep, and bushy plans for a join of four inputs. Left-deep and right-deep plans are different because join algorithms use their two inputs in different ways; for example, in the nested loops join algorithm, the outer loop iterates over one input (usually drawn as left input) while the inner loop iterates over the other input. The set of bushy plans is the most general as it includes the sets of both left-deep and right-deep plans. These names are taken from [Graefe and DeWitt 1987]; left-deep plans are also called "linear processing trees" [Krishnamurthy, Boral, and Zaniolo 1986] or "plans with no composite inner" [Ono and Lohman 1990].

about the hash join:

This is a really good post
http://blogs.msdn.com/b/craigfr/archive/2006/08/10/687630.aspx

Quote it here for cache:

These terms refer to the shape of the query plan as illustrated by this figure:

The shape of the join tree is particularly interesting for hash joins as it affects the memory consumption.
In a left deep tree, the output of one hash join is the build input to the next hash join. Because hash joins consume their entire build input before moving to the probe phase, in a left deep tree only adjacent pairs of hash joins are active at the same time. For example, in the above picture, we begin by building the hash table for HJ1. When HJ1 begins probing, we use the output of HJ1 to build the hash table for HJ2. When HJ1 is done probing, we can release the memory used by its hash table. Only then do we begin probing HJ2 and building the hash table for HJ3. Thus, HJ1 and HJ3 are never active at the same time and can share the same memory grant. The total amount of memory we need is max(HJ1 + HJ2, HJ2 + HJ3).
In a right deep tree, the output of one hash join is the probe input to the next hash join. All of the hash joins must build their complete hash tables before we can begin probing. All of the hash joins are active at once and cannot share memory. When we do begin probing, rows flow up the entire tree of hash joins without blocking. Thus, the total amount of memory we need is HJ1 + HJ2 + HJ3.

Ruby Regexp links

2013-02-14T01:55:00.000-05:00

a quite note to some articles:

Regular Expressions in ruby has a very good summary
http://strugglingwithruby.blogspot.com/2009/05/regular-expressions-in-ruby.html

. any character except newline
[ ] any single character of set
[^ ] any single character NOT of set
* 0 or more previous regular expression
*? 0 or more previous regular expression (non-greedy)
+ 1 or more previous regular expression
+? 1 or more previous regular expression (non-greedy)
? 0 or 1 previous regular expression
??
| alternation
( ) grouping regular expressions
^ beginning of a line or string
$ end of a line or string
{m,n} at least m but most n previous regular expression
{m,n}? at least m but most n previous regular expression (non-greedy)
\1-9 nth previous captured group
\A beginning of a string
\b backspace(0x08)(inside[]only)
\b word boundary(outside[]only)
\B non-word boundary
\d digit, same as[0-9]
\D non-digit
\S non-whitespace character
\s whitespace character[ \t\n\r\f]
\W non-word character
\w word character[0-9A-Za-z_]
\z end of a string
\Z end of a string, or before newline at the end
\/ forward slash

How to get YARN log

2013-02-11T06:28:00.002-05:00

quick memo

use cmd:

yarn logs

usage: general options are:
-applicationId ApplicationId
-appOwner AppOwner
-containerId ContainerId
-nodeAddress NodeAddress

postgresql notes

2013-02-04T06:30:00.001-05:00

Postgresql is famous for its high extensibility and nice performance.

Quick memo:

To define the functions, procedures, and triggers, the manual is the bible:

Refer to Chp 39 for a overview of the language, other language is possible to use, though trickier
http://www.postgresql.org/files/documentation/pdf/9.2/postgresql-9.2-US.pdf

Aggregation functions:
http://www.postgresql.org/docs/8.4/static/functions-aggregate.html

lots of ideas in db are coined several decades ago. Hive UDF/UDAF/UDTF vs. postgresql aggregation functions e.g. xmlagg, array_agg, User defined types and functions, really little difference.

metis, graph partitioning

2013-02-02T16:42:00.001-05:00

METIS from UMN is a great library to do the graph partitioning (also mesh partition, and etc.)
It can do k-way or bisection very fast. (millions vertex graph in seconds)
http://glaros.dtc.umn.edu/gkhome/metis/metis/overview

quick notes to use:

1. input file format
line 1: # of vertex, # of edges, mode_code
the mode_code, 001 means edge weight graph, 011 means edge vertex weight graph,
details, need to refer to manual: http://glaros.dtc.umn.edu/gkhome/fetch/sw/metis/manual.pdf
NOTE: # of edges should be count as once between any pair of vertices.

line 2: is the node 1 (don't need to specify node id 1 in the line)
just follow adj_v_i edge_1_v_weight adj_v_i adj_v_j edge_1_j_weight ...

% started line is comments

2. gpmetis -objtype=vol inputfile #_of_partition
objective function type: vol is the total communication volume minimization cut (in k-way)
another default option is -objtype=cut, which is minimize # of edge cut.

3. output file
each line_no indicates v_id, line_content is the partition id.

getting used to git, quick note from svn perspective

2013-01-27T20:46:00.000-05:00

used git for years, but not everyday basis, understand the difference between distributed version control, but still had some gotcha when using it recently.

Some good links about Git-Svn crash
http://git.or.cz/course/svn.html
http://stackoverflow.com/questions/10169328/git-your-branch-is-ahead-of-origin-master-by-1-commit

On Mac, the Github for mac is very handy.
http://mac.github.com/

This GIT BOOK rocks, of course, the github help
http://git-scm.com/book
https://help.github.com

Here's another trick, if you want to go to previous commit, but keep all the changes:
http://stackoverflow.com/questions/4114095/git-revert-to-previous-commit-how

# This will destroy any local modifications
# Don't do it if you have uncommitted work you want to keep
git reset --hard 0d1d7fc32

# Alternatively, if there's work to keep:
git stash
git reset --hard 0d1d7fc32
git stash pop
# This saves the modifications, then reapplies that patch after resetting.
# You could get merge conflicts, if you've modified things which were
# changed since the commit you reset to

Quicknote for Markdown Format

2013-01-27T19:12:00.000-05:00

There might be many good tutorials, the following one is nice:

markdown the basics
http://daringfireball.net/projects/markdown/basics

and wysiwyg editor
http://markable.in/editor/

And also the Sublime 2 has a plugin called Markdown Preview, which is kinda handy to format the styles.

ssh into virtualbox from mac

2013-01-27T16:50:00.000-05:00

just a quick note.

there're several ways to ssh into the local virtualbox instance.

1. is setting the Networking setting to Bridged other than NAT that is set as default. By doing so, the Bridged network setting will let the virtual box instance act like a physical machine and allocate ip address from the current network. So a ifconfig you can get the ip address.
-> this video is good enough to set it up : http://www.youtube.com/watch?v=5BsShkcweIs

2. but the Bridged network setting is not always working, due to different network infra one is using. Another way to do it is to set another network adapter (Adapter 2, e.g.) to Host-only Adapter. By doing so, a local network will be built in the host machine, which is always available no matter the network infra. So I'd like to recommend this setting.
-> this post is well written but somewhat out of date: http://muffinresearch.co.uk/archives/2010/02/08/howto-ssh-into-virtualbox-3-linux-guests/

One thing to be aware is the vboxnet0 may not exist if going into the vm setting directly. If so, one needs to set it in the virtual box setting itself, which can be accessed in Preference .. > network, and click the +.

Sublime 2, Package Control problem on Ubuntu 12.04 LTS

2013-01-25T15:54:00.000-05:00

First of all, Package Control, and Sublime 2 are awesome tools. I used them for everyday coding in Mac, and Ubuntu nowadays. Basically because the environment is heterogeneous, e.g. ruby, txt, html, js, css, java, groovy, shell, c++. In one data analytics research project, normally I will use several of those tools to make life easier. So drop the eclipse for a while.

I encountered the problem when using Package Control of Sublime 2 in Ubuntu 12.04.

The package control install sub pane/window just don't show up, after the installation. You can find it in the Control+Shift+P pane, but run it, gives you nothing. If I open Ctrl+` console, the error is /tmp/tempxyz is not there.

In the beginning, I thought it was the permission problem, as the ubuntu I'm running lack of sudo permission. But it is not, as I can create the tmp files without using sudo.

Then, I check the Preference: Package Control Setting : Default, and change the repository url from https to http, according to some forum, and it worked to show up the list of packages. However, when I try to download some plugin, it fails again, the error is the same in Ctr+` output, saying that /tmp/tempxyz is not there. But this time, I cannot just fix https to http, not to mentioning doing so it will bring in the men in the middle problem to run arbitrary python code!

(I tried, actually, but it failed again, replace the url formed in ~/.config/sublime-text-2/Packages/Package Control/Package\ Control.py to http always didn't solve the problem)

So I enable the debug mode in Package Control Setting, then the problem comes out. It is not the /tmp/ folder permission problem. But rather, the 'wget' call caused error. The option: --content-on-error causes the wget cmd failed.

To solve the problem, go to
~/.config/sublime-text-2/Packages/Package Control/Package\ Control.py

Change the url construction part, drop that parameter. Everything works fine to install plugins.

quick memo of mounting sftp in mac os x

2012-08-09T01:10:00.000-04:00

get a easier management of remote files using sftp on mac:

install macfuse: http://www.tuxera.com/mac/macfuse-core-10.5-2.1.9.dmg
install macfusion app: http://macfusionapp.org/

if the first link is dead, using osxfuse: http://osxfuse.github.com/ by clicking compliance with macfuse.

open macfusion app, setup the details of ssh connection. and click mount. if error occurs, try to view the log.

Get the newest Octave to run in ubuntu

2012-03-21T08:24:00.002-04:00

just for a reminder here,

octave, a useful tool for quick prototyping ml algorithms. ubuntu10.04 32 official depository version is 3.2. to use the newest version 3.6.x since 2012:

wget http://ubuntuone.com/6WX6polYRNXeJuBqVsCZvi
sudo apt-get build-dep octave3.2
sudo dpkg -i octave_3.6.1-1_i386.deb

try to use: octave

find other linux distributions and built images
octave wiki

Nokogiri & Char encodings

2011-12-26T03:15:00.001-05:00

Personally, Nokogiri is the best library for ruby to look into html structures, especially css-selector support rocks. However, when dealing with i18n webpages, char encoding is always painful for beginners.

for a reminder, mark this page:
http://stackoverflow.com/questions/2572396/nokogiri-open-uri-and-unicode-characters

the following one works

difference is, open-uri read first and then feed into Nokogiri,

if feed directly, will result in hard encoding pain

# encoding: UTF-8
require 'nokogiri'
require 'open-uri'

html = open('http://www.biblegateway.com/passage/?search=Mateo1-2&version=NVI')
doc = Nokogiri::HTML(html.read)
doc.encoding = 'utf-8'
h52 = doc.css('h5')[1].text
puts h52, h52.encoding, h52 == "Genealogía de Jesucristo"
#=> Genealogía de Jesucristo
#=> UTF-8
#=> true

When inserting into mysql without going through rails, but mysql gem, set the UTF8 specifically is mandatory:

this page describes this in detail:
http://wiki.a2hosting.com/index.php/UTF-8_MySQL_connections_with_Ruby

db = Mysql.init
db.options(Mysql::SET_CHARSET_NAME, 'utf8')
db.real_connect('host', 'user', 'password', 'database', 3306, '/usr/local/lib/mysql.sock')
db.query("SET NAMES utf8")

java generic stuff memo

2011-11-07T21:00:00.000-05:00

1. all generic derived classes share one static object

-> as jvm will not separately construct generic classes

e.g. Employee, is the template

Employee, Employee, are the same class Employee in JVM

thus, assuming Employee has a static field, say counter,

then increase the counter in Employee, then Employee will have the updated counter too. (actually, only one counter)

see: http://www.angelikalanger.com/GenericsFAQ/FAQSections/TypeParameters.html#Is there one instances of a static field per instantiation of a parameterized type?

2. need tricks to get the runtime template class object

say Employee has a function, Class getEmployeeType(),

it's not trivial to implement the function, and get the Class.

- one way to walk around is : Class getEmployeeType(Class curr_class)

- another way: http://www.artima.com/weblogs/viewpost.jsp?thread=208860

tricky modulo operator

2011-11-07T20:59:00.001-05:00

in Ruby:

197%5 = 2

-197%5 = 3

in Java:

197%5 = 2

-197%5 = -2

... ...

compile uuid ossp to dll and used it in VS

2010-04-15T23:20:00.005-04:00

Project these days requires cross-platform feature (.Net 4.0, Android 2.1, in house Linux and etc ...). Thus the common used components, we plan to use standard C. Thanks to so many open source libraries, they save us a lot of work. However, compiling and using *nix libraries to windows is a kind of craftsmanship.

In the project, we need a uuid library. The only free & open & reliable solution probably is OSSP UUID.

I started with the newest version uuid-1.6.2.tar.gz with mingw @ a windows xp sp3 pc. I did not use cygwin, due to the license problem.

1. routine job

./configure

make

make check

It does not generate any dynamic library, but a static one, libuuid.a

2. generate dll

well, by reading the stdout make process, i tried

2.1 get dll & def

gcc -shared -o uuid-ossp.dll uuid.o uuid_md5.o uuid_sha1.o uuid_prng.o uuid_mac.o uuid_time.o uuid_ui64.o uuid_ui128.o uuid_str.o -Wl,--output-def,uuid.dll.def

2.2 get lib

dlltool -d uuid.dll.def --dllname uuid-ossp.dll --output-lib uuid-ossp.lib

2.3 test it with the command line tool uuid_cli.c

gcc -I . uuid_cli.c -o uuid.exe uuid_ossp.lib

everything works so well, the command line uuid.exe generated really works.

3. decide how to use and what do we need

sadly, this library, i could not find related API documents.

what i need is just to generate the uuid, by reading uuid_cli.c, i got some hint, here's a simple example.c, after cropping the uuid_cli.c to generate a version 1 uuid.

#include "uuid.h"
#include <stdio.h>

int main(int argc, char *argv[])
{
uuid_t *uuid;
uuid_rc_t rc;
char *vp;
size_t n;
unsigned int version;

version = UUID_MAKE_V1;


if ((rc = uuid_create(&uuid)) != UUID_RC_OK)
 printf("uuid_create error");

if ((rc = uuid_make(uuid, version)) != UUID_RC_OK)
 printf("uuid_make error");

vp = NULL;
if ((rc = uuid_export(uuid, UUID_FMT_STR, &vp, &n)) != UUID_RC_OK)
 printf("uuid_export error");
if (vp)
printf("%s\n", vp);
free(vp);

if ((rc = uuid_destroy(uuid)) != UUID_RC_OK)
 printf("uuid_destroy error");

return 0;
}

The code works well, and can link to the dll well. The only header file we need it uuid.h, no aother uuid*.h needed

4. used in Visual Studio

in the visual studio project, configure the project by using the uuid-ossp.lib, and include the uuid.h in the search path. copy the example.c into the project.

compile and run, oops ...

4.1 uuid.h

the compiler complains the uuid.h include <unistd.h>, it can not find

sure, the file is in mingw include path. i tried, to include related mingw headers into the project. but i failed, with lots of compilation errors.

but you can safely delete this line from the uuid.h, i tried, everything works the same

4.2 heap error

cool, now, the compiler has no complain, but when run, the example.c get a runtime error, in the line

free(vp);

I dig this for a while, and realized that , the vp is malloc-ed and free-ed by different memory malloc & free implementations.

the workable program in mingw, should changed to the following, then, example.c can work fine now:

#include "uuid.h"
#include <stdio.h>

int main(int argc, char *argv[])
{
uuid_t *uuid;
uuid_rc_t rc;
char *vp;
size_t n;
unsigned int version;

version = UUID_MAKE_V1;
n = 37;
vp = (char*)malloc(n);

if ((rc = uuid_create(&uuid)) != UUID_RC_OK)
    printf("uuid_create error");

if ((rc = uuid_make(uuid, version)) != UUID_RC_OK)
    printf("uuid_make error");

vp = NULL;
if ((rc = uuid_export(uuid, UUID_FMT_STR, &vp, &n)) != UUID_RC_OK)
    printf("uuid_export error");
if (vp)
  printf("%s\n", vp);
free(vp);

if ((rc = uuid_destroy(uuid)) != UUID_RC_OK)
    printf("uuid_destroy error");

return 0;
}

cool, everything works now.