Developer's Note

Command to Compare Data Files: comm

noreply@blogger.com (Thiam Teck (1983 - ?)) — Sat, 15 Jan 2022 15:51:00 +0000

`diff` is great command to compare the changes among 2 versions of a file, but for comparing data file and extract data from it, `comm` command is more suitable for the job.

For demo purpose, we will prepare 2 sorted data file as below:

left.txt:


123
456
789

right.txt:


789
888
999

`comm` command work by suppressing lines that exist in one of the file, or both file, where -1 suppress lines unique to the left file, -2 suppress lines unique to the right file, and -3 suppress lines appear in both files.

if no options being specify, it will display data in 3 columns:

> comm left.txt right.txt
123
456
                789
        888
        999

To show only lines that unique to left file, we shall suppress the line unique to right file, and line that appear in both files:

> comm -23 left.txt right.txt
123
456

For showing lines that only appear on the right, we will surpress line unique to the left file, and those appear in both files:

> comm -13 left.txt right.txt
888
999

To extract lines that appear in both files, just suppress line that unique to left and right file:

> comm -12 left.txt right.txt
789

And to exlcude only line that appear on the left, command below will show data in 2 column, one of it is the line commons for both files:

> comm -1 left.txt right.txt
        789
888
999

We believe by now you shall be able to use `comm` command for comparing data file effectively. Thanks for reading.

Linux Scripts to Create Table Partition for Next Month Automatically

noreply@blogger.com (Thiam Teck (1983 - ?)) — Mon, 09 Aug 2021 13:53:00 +0000

In this article we are going to show you a method of create new monthly partition for MySQL table without stored procedure, with the REORGANIZE PARTITION from MySQL/MariaDB.

(Click here to skip the article and straight to the scripts.)

For demo purpose, below are the DML of the table to be partitioned, and the ALTER statement to start with 2 partitions: "p_old" and "p_future". Please change the p_old partition date to the first of next month.

Below is the template of ALTER statement to REORGANIZE PARTITION "p_future" into a partition of next month ('p_202109' for example, as of writing), and 'p_future' for records after next month.

And here is the shell script that did some simple string substitute based on the template, and execute it on your MySQL/MariaDB server. Please do not forget to change the username, password and database name to match you database server.

Finally, you may run this each month to create a new partition for next month with command:

./create_partition.sh

If you are to schedule it via crontab, ensure you modify the script to use full path for the template file.

Thank you.

Offload your SSD (on Windows) and SD Card (on Linux)

noreply@blogger.com (Thiam Teck (1983 - ?)) — Sun, 25 Aug 2019 16:41:00 +0000

As many of us aware, both SSD and SD Card wearing out upon write operation. So to extend life span of the storage, we may look into reduce the write operation.

(With wear leveling, most modern SSD have lifespan that is long enough where you might want to skip this post. But no harm to know something new, right ?)

For SSD, you may consider putting the Windows's TEMP folder into a RAM disk. RAM disk, as the name implies, it is a virtual drive that reserve and utilise the space on your RAM. There are many RAM disk software out there, and I'm using ImDisk's RamDisk Configuration.

You can define the space of the RAM disk, and set the TEMP folders to the RAM Disk with this tool:

(ImDisk RamDisk Configuration)

For SD Card that store your OS (i.e. Raspberry Pi), we will put the `/var/log` on RAM with log2ram. This will make the logs stay in RAM and only flush to storage once a day by default (configurable).

After done installation and reboot your Raspberry Pi, you shall see the screen below:

(log2ram: new mounting point after installation and restart)

This shall extend the lifespan of your SD Card.

Thanks for reading.

Get 30GB of Samba Storage on Cloud

noreply@blogger.com (Thiam Teck (1983 - ?)) — Fri, 16 Aug 2019 15:32:00 +0000

OK, it is less than 30GB to be exact, as the OS took some space too.

The whole idea started from the offer from Google Cloud with an "always free" plan which include 1 f1-micro instance with 30GB standard persistent storage. (Anyway, you might still get charge for egress too)

So if you already have some running VPS or cloud instance, you may just use that too.

However, as you might experienced, most cloud provider have strong focus on network security. And the steps to define firewall rules, etc are different among providers.

Fortunately, ZeroTier come into rescue. We can use ZeroTier to link our home PC with the cloud instance just like there are in the same LAN (or Global Area Network, as advertise by ZeroTier).

So here go the list of guide that I had followed, hope you enjoy it:

How to set up a free micro VPS on Google Cloud Platform (I had choose Centos 7 as the OS, just personal preference)
Getting Started with Software-Defined Networking and Creating a VPN with ZeroTier One (I skip the VPN related steps)

Install the ZeroTier client on you home PC and join the same ZeroTier network (I did not read any guide for this, ZeroTier client for Windows is simple enough to install)

How to Install Samba4 on CentOS 7 for File Sharing on Windows

Additional Ideas:

If you are have Windows 10 Pro at home, you may also consider to use NFS too.
You may experiment on mount object storage to the VM via FUSE for additional storage. (Do share with me the result and experience)

Chrome unable to load Google sites

noreply@blogger.com (Thiam Teck (1983 - ?)) — Wed, 29 Apr 2015 14:53:00 +0000

Recently I had been facing problem loading Google sites such as its search engine, Gmail, Youtube, etc.

It is either take extra ordinary long time to load, or just failed with `ERR_QUIC_PROTOCOL_ERROR`.

I did a search and found I am not alone, and below are the solution [1]:

Open a new tab in Chrome, and type `chrome://flags/` into address bar, then press Enter.
Search for `Experimental QUIC protocol`
Disable it.
Restart Chrome. Done.

It had been about a week, and the error never happen again.

[1]: https://productforums.google.com/forum/#!topic/chrome/xKFfhDvhjU8

Set Transfer Encoding of Attachment for Axis2

noreply@blogger.com (Thiam Teck (1983 - ?)) — Sat, 05 Oct 2013 05:51:00 +0000

When sending SOAP request with attachment, Axis2 handle the attachment via Axiom, and the default transfer encoding for binary file are binary.

In order to change the transfer encoding to something else, we will utilize the org.apache.axiom.attachments.ConfigurableDataHandler instead of the usual javax.activation.DataHandler.

Below is the sample code:

Axis2 + Rampart: Generate Nonce and Created when Plain Text Password was Used

noreply@blogger.com (Thiam Teck (1983 - ?)) — Tue, 12 Jun 2012 16:36:00 +0000

Axis2 support WS-Security via Apache Rampart module. I had develop the SOAP client based on tutorial : UsernameToken Authentication with Rampart

However, the default implementation of Apache Rampart will not generate Nonce when plain text password was used.

After some google, I found this page talk about a configuration called "addUTElements".

So below are the modified configuration fragment:

MySQL Random Words Based on Lorem Ipsum Text

noreply@blogger.com (Thiam Teck (1983 - ?)) — Sat, 07 Apr 2012 03:18:00 +0000

Below is a MySQL function that create random text based on 100 words from Lorem Ipsum text.

Revert to the old Google Reader? Let's enhance the new one

noreply@blogger.com (Thiam Teck (1983 - ?)) — Fri, 04 Nov 2011 18:09:00 +0000

IMHO, The redesign of Google Reader to integrate with Google+ do not impress me.

I am agree with the review of ex-Product Manager of Google Reader. The space allocated for reading is just too small, and I personally like the light blue theme than the current grey scale page.

I failed to google any workable way to restore the old Google look and feel. However, I found the ways to enhance the Google Reader (for Firefox).

Below is the original new Google Reader look and feel:

Below is the modified look and feel:

Below are the steps:
1. Install Greasemonkey plugin
2. Install script "Google Reader - Compact Design" (this script make interface more compact thus give you more space to read)
3. Install Stylish plugin
4. Install user style "Google Reader Revert" (to get back the light blue theme)

So far this 2 scripts do not conflict with each other. Feel free to try.

JQuery UI: Datepicker with Month and Year only

noreply@blogger.com (Thiam Teck (1983 - ?)) — Sat, 19 Mar 2011 06:27:00 +0000

This can be done by hiding the calendar via some CSS trick:

JQuery UI: eliminate "stop running this script?" warning in IE

noreply@blogger.com (Thiam Teck (1983 - ?)) — Sat, 19 Feb 2011 06:20:00 +0000

In IE, when the Javascript on the page take too long duration to execute, the browser will popup a warning of "Stop running this scripts?"

In order to eliminate the warning, we need to hint the browser that the script is not infinitely running, by having some pause via Javascript setTimeout method.

However, for Javascript framework such as JQuery that use selector and method chaining extensively, Just setTimeout alone will not help much.

We will need to slice the elements list from selector into few smaller batches, before chain it to the next method.

Below is the sample code snippet:

Feel free to share if you have a more elegant approach to this problem. :)

Introducing feed2PDF

noreply@blogger.com (Thiam Teck (1983 - ?)) — Sat, 27 Nov 2010 14:18:00 +0000

As the name implied, feed2PDF is an application that convert ATOM/RSS feed into PDF files. It is built on top of Joliprint, with enhancement of merging PDF generated from each feed item into 1 PDF file.

Hope you found this useful.

Split String with Empty Trailling Tokens in Java

noreply@blogger.com (Thiam Teck (1983 - ?)) — Sun, 08 Aug 2010 04:57:00 +0000

The "-1" do the tricks:

String[] stringTokens= dataString.split(",", -1);

Print Directory Tree with Groovy (with closure recursion)

noreply@blogger.com (Thiam Teck (1983 - ?)) — Sun, 25 Apr 2010 06:47:00 +0000

While looking for a script that display directory tree, I came across this Groovy script.

The code look just fine but it don't work (I test it with Groovy 1.7). After some google, I found that this is due to the closure as a first class object in Groovy, it can't be call within the closure itself.

And below is the solution:

def listRecursivly

listRecursivly = { file, arg ->
    println "${'|  ' * arg}`- ${file.getName()}\\"
    file.eachFile() {f ->
        if(f.directory) {
            listRecursivly(f,(arg + 1))
        } else {
            println "${'|  ' * (arg + 1)}`- ${f.getName()}"
        }
    }
}

listRecursivly (new File("."),0)

The little trick here is to define and initialize the closure in 2 separate lines.

Scan File with AVG 8.5 in Windows Live Messenger

noreply@blogger.com (Thiam Teck (1983 - ?)) — Fri, 15 Jan 2010 13:53:00 +0000

It is a bit tricky:

1. Create a batch file (said ScanAVG.bat) in your AVG installation directory:
2. Edit the batch file:


@echo off
"C:\Program Files\AVG\AVG8\avgscanx.exe" /SCAN=%1 /HEUR /ARC /PUP /CLEAN

3. In Live Messenger, Tools > Options > File Transfer
4. Check "Scan files for viruses using:"
5. Fill in the path to your batch file. (i.e. "C:\Program Files\AVG\AVG8\ScanAVG.bat")

Reference: http://forums.techguy.org/general-security/707127-solved-avg-8-free-windows-2.html

Send Message to ActiveMQ via WGET

noreply@blogger.com (Thiam Teck (1983 - ?)) — Sat, 09 Jan 2010 01:26:00 +0000

In ActiveMQ website, it does show the steps of consuming message from ActiveMQ via WGET.

In order to produce message and send into ActiveMQ queue, here is the command:

wget --post-file=message.txt http://localhost:8161/demo/message/MyQueue?type=queue

Where the message.txt is something like this:

body=the test message you would like to submit.

Gant: OutOfMemoryError

noreply@blogger.com (Thiam Teck (1983 - ?)) — Mon, 23 Nov 2009 14:28:00 +0000

Today just hit an OutOfMemoryError when using Gant build script to compile my code.

Google for way to set heap size for Gant, but in the end found the solution by set it in JAVA_OPTS.

set JAVA_OPTS="-Xmx512M"

Redirect Tomcat console output to log file

noreply@blogger.com (Thiam Teck (1983 - ?)) — Mon, 07 Sep 2009 01:47:00 +0000

We know using proper logging library instead of System.out is important. But when there are a legacy application that still using the System.out and System.err, it will be good to be able to pipe it into file for troubleshooting.

The solution is set the swallowOutput="true" in Tomcat context of your webapp. Then this will pipe the stdout and stderr into the logger configured in the same context.

ERROR 1 (HY000) at line 1: Can't create/write to file

noreply@blogger.com (Thiam Teck (1983 - ?)) — Mon, 15 Jun 2009 14:48:00 +0000

MySQL SELECT statement allowed us to pipe the query result in to a file via SELECT ... INTO OUTFILE. It work just fine for me most of the time, until recently I hit this error:

ERROR 1 (HY000) at line 1: Can't create/write to file '\home\myuser\my_output_file.txt' (Errcode: 2) mv: cannot stat `/home/myuser/my_output_file.txt': No such file or directory

The first thing come to my mind is the problem of file permission, so I grant 777 permission to my output directory. But the problem still exist.

Then I check the Mysql username I use, and found that it was granted with FILE permission correctly.

Then I start to google on this error code, and most of the search result point to similars causes.

So I read again the documentation of MySQL, and I found this:

The SELECT ... INTO OUTFILE 'file_name' form of SELECT writes the selected rows to a file. The file is created on the server host, so you must have the FILE privilege to use this syntax.

And the keyword is "server host". I made wrong assumption that it will write file to the server that execute the SELECT ... INTO OUTFILE.

And the workaround is simple, just change my SELECT statement by remove the OUTFILE portion. And pipe the query to file from the MySQL command line

mysql -u myuser -ppassword -h remotehost mydb < myquery.sql > my_output_file.txt

Reading documentation is bored but important :P

Passwordless SSH login via Public Key Authentication

noreply@blogger.com (Thiam Teck (1983 - ?)) — Sun, 19 Apr 2009 11:17:00 +0000

SSH login with password authentication is often a problem for shell script that run without human interaction. One of the option by simulating user input via some Expect script. While another option is to setup certification based authentication.

Below is the steps:

1. SSH to client host (said with username 'myuser')
2. Generate keys at client host, press 'Enter' while it prompt for key location and passphrase.:


$ ssh-keygen -t rsa

3. Upload the public key generated (~/.ssh/id_rsa.pub) to remote host home directory (said username 'remoteuser')
3. SSH to the remote host with same username ('remoteuser'), create the .ssh folder if not exist


$ cd ~
$ mkdir .ssh
$ chmod 700 .ssh

4. Import public key into authorized_keys


$ cat id_rsa.pub >> ~/.ssh/authorized_keys
$ chmod 600 ~/.ssh/*

Now when the 'myuser' at client host SSH to remote host with username 'remoteuser', it will not prompt for password anymore.

Duplicate log in log4j

noreply@blogger.com (Thiam Teck (1983 - ?)) — Thu, 05 Mar 2009 14:54:00 +0000

If you have log4j configured as below, you will notice that log entry from "yourLogger" will duplicate in appender


log4j.rootLogger=info, stdout
log4j.logger.yourLogger=info, stdout

To solve this, just change your configuration to:


log4j.rootLogger=info, stdout
log4j.logger.yourLogger=info, stdout
log4j.additivity.yourLogger=false

Reading List: Dec 2008 Wrapup

noreply@blogger.com (Thiam Teck (1983 - ?)) — Thu, 01 Jan 2009 08:09:00 +0000

IO performance - Tar vs. File vs. Byte streams
Benchmark that show TAR file processing as a good alternative for better performance.

Google Has Open-Sourced Their C++ Mocking Framework

Top Java Developers Offer Advice to Students
11 Top Java developers sharing their experience.

Axis2 WSDL2Java error for SOAP Encoding

noreply@blogger.com (Thiam Teck (1983 - ?)) — Sat, 27 Dec 2008 09:30:00 +0000

Recently had hit the SchemaCompilationException when using Axis2 wsdl2java to generate client stub from a WSDL file.

After some google, had found this page is describe same problem as what I face. And the reason for the exception is due to Axis2 do not support SOAP Encoding.

Fortunately after a while, my colleague had found the solution. Instead of using ADB for data binding, use XMLBeans. Below is example of the command:

wsdl2java -uri myService.wsdl -d xmlbeans

Reading List: Nov 2008 Wrapup

noreply@blogger.com (Thiam Teck (1983 - ?)) — Sun, 30 Nov 2008 01:00:00 +0000

Microsoft joins AMQP Working Group
Microsoft already joins AMQP, so when will Sun?

More Weapons for the War on Complexity: SpringSource Acquires Groovy/Grails Leader
After acquire Covalent, SpringSource acquires G2One.

Interview with Jason Hunter of MarkMail.org
Interview with Jason Hunter about the MarkMail.org and its architecture to scale to its terabytes data.

Martin Fowler see a Thaw in Frozen Thinking about Data Storage
Besides the dominant RDBMS, Martin Fowler point out that RISC RDBMS and document-oriented distributed database can be an alternative for certain scenario.

Java: convert UCS2 to UTF-8

noreply@blogger.com (Thiam Teck (1983 - ?)) — Fri, 14 Nov 2008 13:16:00 +0000

While convert UCS2 into UTF-8, I hit UnsupportedEncodingException. So I use UTF-16 instead and it work well (at least till the time I write this post):


public String ucs2ToUTF8(byte[] ucs2Bytes) throws UnsupportedEncodingException{

    String unicode = new String(ucs2Bytes, "UTF-16");
   
    String utf8 = new String(unicode.getBytes("UTF-8"), "Cp1252");
   
    return utf8;
}