Sun will rise

My Chinese Teacher

2013-05-05T06:31:00.001-07:00

My high school Chinese teacher, Mr. Zhang Jun Qiu, passed away in his fifties this week. He's a role model and I respect him so much. I never expected a few words in QQ chat last year was the last communication with him...

Rest In Peace, my teacher.

dd on devices the difference between Linux and Soalris

2013-04-22T03:11:00.002-07:00

People like to use dd to do the micro benchmark because it is simple. However, we should be careful about what is being measured by dd. This can be complicated than we imagine.

Let's take a look on a simple scenario: single dd writes on devices with bs=16k

    # dd if=/dev/zero of=dev_name bs=16k count=655360

The difference between block I/O and raw I/O and The difference between Solaris and Linux

The device name can be block devices or raw devices. For I/O on block devices, there's a cache layer in OS kernel. For I/O on raw devices, I/O bypasses the kernel's buffer cache.

The device names are different between Linux and Solaris:
- Solaris block device: /dev/dsk/...
- Solaris raw device: /dev/rdsk/...
- Linux block device: /dev/sdxxx, /dev/mapper/xxxx, etc.
- Linux raw device: /dev/raw/xxxx (raw binding on block device)

What do they look like in iostat? (Don't look at performance numbers, I tested on different disks), I'm illustrating the behavior of single dd write on block devices and raw devices.

Solaris block device (bs=16k):
                    extended device statistics
    r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0 13006.1    0.0  101.6 133.5 256.4   10.3   19.7 100 100 c5t20360080E536D50Ed0

Linux block device (bs=16k):
Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00 14098.00    0.00  115.00     0.00 57868.00 1006.40   140.86 1325.40    0.00 1325.40   8.70 100.00

Solaris raw device (bs=16k):
                     extended device statistics
    r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0 2805.0    0.0   43.8  0.0  0.9    0.0    0.3   3  87 c5t20360080E536D50Ed0

Linux raw device (bs=8k):
Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00    0.00 3645.00     0.00 29160.00 16.00     0.63    0.18    0.00    0.18   0.17  62.90

For "single dd write" on block devices, look at "actv" column (Solaris) and "avgqu-sz" column (Linux), which means that average number of transactions actively being serviced (removed from the wait queue but not yet completed) are much larger than 1. On Solaris, the "actv" limit is controlled by kernel parameter ssd_max_throttle or sd_max_throttle, the default is 256. If the limit is hit, the I/O request will be queued ("wait" in iostat). The upper layer (e.g. ZFS filesystem) may also limit the I/Os sent for each device .

For "single dd write" on raw devices, the "actv" or "avgqu-sz" is never larger than 1 (unless the backend of dev is not a real device, e.g. a regular file), which means that only when previous data transfer is completed, the next data can be send to the device. In this regard, Solaris and linux behave similarly. In addition, since this is I/O on raw devices, the I/O size in iostat is equal to application write size. While modern CPU and memory subsystem is very fast, "single dd write" on raw devices becomes more like a disk subsystem latency testing.

On the other hand, Solaris and Linux has different caching implementations for dd writes on block devices. From above iostat output, you can see Solaris write() splits the data into 8k trunks then send data to the I/O driver, while Linux can smartly merge small I/Os ("wrqm" column in iostat). This means that for "dd writes on block devices" Linux usually performs better than Solaris. However, this is not very common in Solaris, in most cases, application I/O writes are on the file system which can consolidate writes or on raw devices which is usually used and optimized by databases. Below is iostat during the write testing on a file in a ZFS filesystem.

$ dd if=/dev/zero of=TEST bs=16k count=655360
                    extended device statistics
    r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0  154.0    0.0  154.0  0.0 10.0    0.1   64.7   2 100 c7t0d0

We can see the above I/O write size is 154.0/154.0=1MB/s.

What you can expect for single dd write performance

Below is my test result on a modern server, yours may be different.

2.5" 10KPM internal disk without using internal hardware raid:
single dd writes on block device bs=16k: 160MB/s.
single dd writes on raw device bs=16k: 160MB/s (drive write-cache enabled)
1.3MB/s (drive write-cache disabled)

A low-end SAN storage:
single dd writes on block device bs=16k: depends on OS, LUN disk layout, raid level etc.
single dd writes on raw device bs=16k: a little more than 40MB/s (with some tweak on storage settings can get 75MB/s)

Additional notes:
-----------------------
- On linux, raw I/O is similar as O_DRIECT, GNU dd has a "oflag=direct" option for block device.
- Also tested on internal hardware raid0 volume of internal disk, the performance is similar as above internal disk testing result. A benefit of using internal hardware raid is for "oflag=dsync" writes, if you configure the internal raid controller cache as write-back enabled, then you will get much better performance.

CLOSE_WAIT

2013-04-21T20:53:00.001-07:00

It's not the first time to hear that people complain CLOSE_WAIT state remains on the system.
It is because that the peer closes the connection (close the socket explicitly or the peer process is terminated), but your side does not take correct action on this connection.

So, what happens if the peer closes the connection?

- If your side does not take correct actions, the peer will NOT result in bad state and the peer's connection state will be cleaned up after tcp_fin_wait_2_flush_interval .

- If your side is doing write() or send() on the connection, your side will receive SIGPIPE signal ("broken pipe", see signal.h man page). The default action for SIGPIPE is exiting the application. So your application probably need change this default behavior.

- If your side is doing read() or recv() on the connection, your side will receive the return code "0". Your side must handle this situation.

- If your side is waiting for POLLIN event via select() or poll() or port_get(), then it will fire the event and consequently your recv() code will return "0", thus you can handle this situation in your application.

- If your side does not take any action on the connection, CLOSE_WAIT will remains on your side until you exit or restart your side application. (The tcp_keepalive_interval or tcp_keepalive_abort_interval in tcp/ip settings does not help on this)

A typical mistake in application is as below, see below example:
....
recvbytes=recv(sockfd, buf, BUFSIZE, 0);
if (recvbytes < 0) {
             perror("recv error");
             close(sockfd);
             .....
}
....

this is buggy, the correct way is " if (recvbytes <= 0) {"
"0" means the peer has closed the connection.

It' strange that Linux man page clearly says "recv() returns 0" means the peer has performed an orderly shutdown but current Solaris man page says nothing about it.

Be careful to use valloc and libmalloc on Solaris 10

2013-04-20T04:37:00.001-07:00

If you application uses valloc(), be careful to use with libmalloc.so on Solaris 10.
Solaris 10's libmalloc (at least for several versions I tested) does not have implementation of valloc(), then valloc() is called from libc but free() is called from libmalloc, this will cause core dump.

You can use DTrace to check it. For example:
~/tmp$ dtrace -qn 'pid$target::valloc:entry {ustack();}' -c ./a.out

To check if you libmalloc has valloc:
$ elfdump /usr/lib/libmalloc.so|grep valloc

Four Years

2013-04-20T04:28:00.001-07:00

Four years ago today, my colleague called me while I was washing dishes after dinner. He said: "our company is sold! check your email now..."
Yes, I cannot forget it for ever. The bitterness is still deeply in my heart.

Build Mesos on Solaris

2013-01-30T00:53:00.000-08:00

Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks. The purpose to build it is that I want to have experiments on Spark , a popular framework for cluster computing (e.g. big data analysis).
Mesos integrated a lots of third party software, the build process on Solaris is not very smooth. At first, I tried Solaris Studio (SunCC), but had some troubles. To save time, I decided to use gcc.
The build environment is: Solaris 11 11/11 sparc; gcc is 4.5.2 from Solaris 11 IPS repository; mesos 0.9.0.

Preparation Steps:

- need to build automake-1.11.6 (automake-1.11.2 from Solaris 11 is not enough for mesos 0.9.0)
wget http://ftp.gnu.org/gnu/automake/automake-1.11.6.tar.gz
CC=cc CXX=CC ./configure --program-suffix=-1.11
gmake;gmake install (in /usr/local/bin)

- modify configure.ac
in "solaris" section:
    CC=cc
    CXX=CC
    CFLAGS="$CFLAGS"
    CXXFLAGS="$CXXFLAGS"
    LIBS="$LIBS -lsocket -lnsl -lproject -lproc -lresolv -lsendfile -lxnet"

then in "JAVA_LDFLAGS" section, add something:
...
    elif test "$OS_NAME" = "solaris"; then
      JAVA_LDFLAGS=""
      for arch in sparc; do
        dir="$JAVA_HOME/jre/lib/$arch/server"
        if test -e "$dir"; then
          # Note that these are libtool specific flags.
          JAVA_LDFLAGS="-L$dir -R$dir -ljvm"
          break;
          fi
      done
    fi
...

- execute autoconf.

- run configure:
./configure CC=gcc CXX=g++ CFLAGS="-m32 -pthreads" CXXFLAGS="-m32 -pthreads" JAVA_HOME=/usr/java --prefix=/opt/mesos

Porting issues and solutions:

1) process.cpp in libprocess
compile process.cpp and future.hpp failed because syntax error in assembly.
using CC -S, then cc -c process.s, we found it use "pause" instruction.
Solution: use smt_pause()

2) process.cpp in libprocess
ssize_t length = sendfile(s, fd, offset, size);
=>
ssize_t length = sendfile(s, fd, &offset, size);

3)pid.cpp in libprocess
for gethostbyname2_r, it is not availabe on Solaris, has to modify the codes to use gethostname_r

4)port_posix.h, atomic_pointer.h
for macros and memory barriers.

5)getpwuid_r in zookeeper.c
getpwuid_r(uid, &pw, buf, sizeof(buf), &pwp))
smiliar as gethostbyname2

6)recordio.c in zookeeper
redefined htonll:
recordio.h:int64_t htonll(int64_t v);
On solaris, the prototype is: uint64_t htonll(uint64_t hostlonglong);
linux doesn't have htonll
Solution: comment out htonll

7)mt_adapter.c in zookeeper
atomic ops: fetch_and_add

8)cli.c in zookeeper
ctime_r(&tctime, tctimes)
Solaris requires: char *ctime_r(const time_t *clock, char *buf, int buflen);

9)mesos
./common/utils.hpp:359:17: error: ‘NAME_MAX’ was not declared
Solution: define it #define NAME_MAX 255

10)slave/solaris_project_isolation_module.hpp
Solaris project implementation is not completed, so got compiling error.
Workaround: skip it by modifying macro definition

11) protobuf is compiled in 64-bit by default.
Solution: reconfigure in thirdparty/protobuf according to config.log and add -m64 in CFLAGS and CXXFLAGS: (it's better to add -m32 in top level of mesos configure).

12) /usr/lib/python2.6/pycc complained:
cc: No valid input files specified, no output generated
because the src file is c++: native/proxy_executor.cpp
Workaround (by looking at /usr/lib/python2.6/pycc, pycc and pyCC is same):
in ~/Downloads/mesos-0.9.0/src/python
$ PYCC_CC=g++ PYCC_CXX=g++ LDFLAGS="-lnsl -lresolv -lsendfile -lsocket" python setup.py build
(It seems on Solaris 11.1 there's no such issue).

13) gmake test
need -lxnet

14) got runtime error:
if (errno != EINPROGRESS) {...
in fact, errno is 0 here.
Reason: not with "-pthreads" in build libprocess.
Solution: also add "-pthreads" in configure.ac so that it take effects for each thread having a private copy of errno.

Other notes:

- Building mesos 0.9.0 fails with gcc 4.7 on Linux. Most current linux versions come with this gcc version.
- I also built mesos 0.9.0 on Solaris 11.1 x64, the code modifications are less because the platform is x64.
- The modified files are now put into github.

Micro Benchmark MongoDB 2.2 performance on Solaris

2012-10-29T02:28:00.000-07:00

I 100% agree on the statement by MongoDB:

“MongoDB does not publish any official benchmarks. We recommend running application performance tests on your application's work-load to find bottleneck and for performance tuning.”

However I don't have real world workload, so I just tried some micro benchmarks to observe the behaviors of MongoDB and OS. Although the result numbers mean nothing, but I would share some findings here.

1) Using JS Benchmark Harness

MongoDB provides JS Benchmarking Harness as a QA baseline perf measurement tool, not designed to be a "benchmark". This is a good start for having a first look at MongoDB performance. The harness is very easy to setup. However, there are a few things to be considerded.

The sample code on that web page is really really a micro benchmark. I tested it against MongoDB 2.2 for Solaris x64 and got suboptimal result comparing against Linux version. After analyzing the workload characteristics, it is more like a multi-threaded malloc and small TCP/IP packet ping-pong testing.

By passing LD_PRELOAD_64=libmtmalloc.so to starting mongod, I got the performance on Solaris parallel to Linux. If the test client and sever are on separate systems, I may also need disable nagle algorithm: $ sudo ndd -set /dev/tcp tcp_naglim_def 1

The harness also has an interesting feature: RAND_INT [ min , max , multiplier ], it looks like we are able to only touch a fix fraction of data during the testing. Two things need be considered here:

I looked at the current harness implementation, RAND_INT is translated to rand(), this is not really random for big (millions of records) data sets. The fix is using lrand48() instead.
MongoDB uses mmap to cache data, like many other databases, it is still a page-level cache rather than row-level cache. So if your record size is small, RAND_INT [ 1, 10000000, 10 ] doesn't make you only touch 1/10 data, rather it makes you touch all the data.

2) Using YCSB.

YCSB is an extensive load testing tool. But its tests codes for mongodb is a little outdated. I need modify a little bit to add more writeConcern type.

YCSB's testing driver has some limitations:

You can set read/write proportion, but they are in same thread context, which means writes can block reads. So I prefer to put them in separate simultaneous jobs in testing.
The “recordcount” also implicitly set the max Id of data to be tested. when testing mongodb, the small number means only a few data files are mapped into the memory during transaction phase. So setting “recordcount”in transaction phase is not the right way to test against only small portion of the data.

3) Solaris related stuff.

The Solaris version of mongodb 2.2 has a large binary size compared to Linux, although it nearly does not affect the performance, but I don't like it. A quick check on its build info got “GCC 4.4 on snv_89 January 2008”, too old. This should be fixed by adding GCC option "-fno-function-sections" and "-fno-data-sections".

When starting mongod for Solaris, a warning message shows: “your operating system version does not support the method that MongoDB uses to detect impending page faults. This may result in slower performance for certain use cases”. After browsing the source codes, I found processinfo support is not there. So I added the Solaris support, currently the functions that count are ProcessInfo::blockInMemory() and ProcessInfo::blockCheckSupported().

The mongodb source code says “madvise not supported on solaris yet”, this is funny. Solaris certainly supports madvise. But madvise() is only useful when you understand your workloads. So I don't think this piece of code of calling madvise() is important.

ZFS and UFS.

==========
Since mongodb uses mmap(), it leaves a lot of things to the OS file system. UFS is a traditional file system, it uses traditional page cache (cachelist ) for caching file data. ZFS has quite a lot features beyond a file system, ZFS has its own ARC cache. The physical memory usage can be inspected using mdb ::memstat command:

# echo "::memstat"|mdb -k
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                     293720              1147    7%
ZFS File Data               85347               333    2%
Anon                       138902               542    3%
Exec and libs                1638                 6    0%
Page cache                  27118               105    1%
Free (cachelist)          3036514             11861   73%
Free (freelist)            576129              2250   14%

Total                     4159368             16247
Physical                  4159367             16247

In my test, ZFS is very good performance in data loading. However, because ZFS has its own cache, if the data is not mmaped, it will be searched firstly from cachelist then ZFS cache, if it's not there, then data is read from the disk into ARC cache, then data is mapped into mongod process address space as page cache. Using ZFS need more memory and when all data cannot fit in physical memory, there would be fights for memory between cachelist and ZFS cache. Tweaking ZFS parameters (manually set ARC cache size, adjust "primarycache" property, etc) did not help in my tests. For read intensive workload, using SSD as 2nd-level ARC cache will help. In addition, depending the workload and data characteristics, adjusting ZFS recordsize or disabling ZFS prefetching may worth a try.

An interesting madvise option for UFS is MADV_WILLNEED, when this option is set, the system will try to pull all data into the memory (quick warm) while during this period the mongod could not response to clients. So if your whole dataset can fit into the physical memory and you can stand the short period of unresponsive to outside during startup, you can consider using it because it warms fast and get peak performance quickly.

Build CouchDB 1.2 on Solaris 11 (SPARC)

2012-08-22T08:03:00.000-07:00

I want to play a little bit with Document-oriented DB. The first try was MongoDB, however, building MongoDB on SPARC is a horrible experience: MongoDB is written for x86/x64; in addition, it uses scons as its build tool and scons doesn't work with IPS-based Solaris Studio 12.3 (I need hack scons to make it work), in fact in MongoDB's source code, it never consider other compilers but VC and GCC. Therefore, I decided to try CouchDB. Allthough CouchDB has a lot of dependencies, but they are friendly for porting.

1) ICU (International Components for Unicode)
This is easy on Solaris 11:
# pkg install developer/icu (this will also cause library/icu to be installed)
Note: the ICU library 4.6 is built with Sun CC, so the C++ code that will link to it also should be built with Sun CC.

2) Build Erlang
Solaris 11 11/11 IPS repository provides Erlang 5.6.5, this is too old. CouchDB 1.1.x or later requires Erlang >= 5.7.3. Although Erlang website says they do daily build for Solaris Sparc (may be because Ericsson was Sun's shop), but why don't they publish the binary package for Solaris? Fortunately, the build procedure is not that difficult:

- Grab the source code from Erlang website. I chose 14B04 which version is 5.8.5.

- ./configure --prefix=/opt/local (I use default gcc-45 from Solaris 11 IPS repository, since it is said erlang uses some GCC features (label vars?).
Ignore configure warnings unless configure fails.

- fix erts/emulator/drivers/common/inet_drv.c:
ifreq.ifr_hwaddr.sa_data would cause compiling error because ifreq does not have that member. Since Solaris 11 has defined both SIOCGIFHWADDR and SIOCGENADDR, we only need SIOCGENADDR.


//#ifdef SIOCGIFHWADDR

//          if (ioctl(desc->s, SIOCGIFHWADDR, (char *)&ifreq) < 0)

//              break;

//          buf_check(sptr, s_end, 1+2+IFHWADDRLEN);

//          *sptr++ = INET_IFOPT_HWADDR;

//          put_int16(IFHWADDRLEN, sptr); sptr += 2;

//          /* raw memcpy (fix include autoconf later) */

//          sys_memcpy(sptr, (char*)(&ifreq.ifr_hwaddr.sa_data), 
IFHWADDRLEN);

//          sptr += IFHWADDRLEN;

//#elif defined(SIOCGENADDR)

#ifdef SIOCGENADDR

- After gmake install, prepend /opt/local/bin to PATH.

3) Build SpiderMonkey 1.8.5
- Download source code from here.
- It's interesting that Mozilla's project need autoconf-2.13, so I had to build this old version.
Get the source from GNU webiste, then:
./configure --prefix=/usr/local --program-suffix=-2.13

- Run autoconf in SpiderMonkey's source tree: to generate configure script.

 cd  js-1.8.5/js/src
 /usr/local/bin/autoconf-2.13

- This time I used Sun CC from Solaris Studio 12.3 which is freely available.
CC=cc CXX=CC ./configure

- gmake; sudo gmake install (in /usr/local)

4) Build CouchDB 1.2
Before proceeding, I need to fix something:

- apache-couchdb-1.2.0/src/couchdb/priv/Makefile.in:
replace "-Wall -Werror" with "-v" because I use Sun CC.

- cd /usr/local/lib; ln -s libmozjs185.so.1.0 libmozjs185-1.0.so
(because configure.ac always choose libmozjs185-1.0 instead of libmozjs185 due to existence of libmozjs185-1.0.a)

- apache-couchdb-1.2.0/src/snappy/google-snappy/snappy-stubs-internal.h:
solaris does not have byteswap.h, I replaced it with byteorder.h and defined a few macros.

...

#else

//#include 

#include 

#define bswap_16(x) BSWAP_16(x)

#define bswap_32(x) BSWAP_32(x)

#define bswap_64(x) BSWAP_64(x)

....


- configure


 ./configure CC=cc CXX=CC LDFLAGS="-R /usr/local/lib" --with-erlang=/opt/local/lib/erlang/usr/include --with-js-lib=/usr/local/lib/ --with-js-include=/usr/local/include/js/ --prefix=/opt/couchdb1.2

- gmake

- Run test:
$ export PATH=/usr/perl5/5.12/bin:$PATH
$ gmake check
Wait for a while, all tests should be successful.

- sudo gmake install

- create a script couchdb.sh in /opt/couchdb1.2/bin (since I don't want the couchdb output to be at arbitrary place).
#!/usr/bin/bash
BIN_DIR=$(dirname $0)
cd $BIN_DIR
./couchdb -o ../var/log/couchdb.stdout -e ../var/log/couchdb.stderr ${1+"$@"}

- modify /opt/couchdb1.2/etc/couchdb/default.ini, change bind_address to let couchdb accessible from anywhere
bind_address = 0.0.0.0

- start couchdb: /opt/couchdb1.2/bin/couchdb.sh -b

Now, enjoy CouchDB!

UPDATE (Sept 2012):

- I also tested Erlang 15B01 and latest 15B02, erts/emulator/drivers/common/inet_drv.c has already been fixed. But for building couchdb, we need to add something to /opt/local/lib/erlang/usr/include/erl_driver.h:
#if defined(__sun)
#include <unistd.h> /* for ssize_t */
#endif

- Running couchdb testsuite in browser (Erlang 14B04 and 15B01) caused erlang process crash. I posted the core dump analysis and temporary workaround in erlang mailing list. Using 15B02 doesn't have this problem, but still should be careful when the Erlang application on Solaris reloads crypto.so, see discussions here. One solution might be adding "-z nodelete" LDFLAGS in lib/crypto/c_src/sparc-sun-solaris2.11/Makefile.

root my phone

2012-06-27T03:52:00.000-07:00

Compared to PC users, the smart phone users have far less freedom. I bought an android 4.0.3 phone recently. My phone was pre-installed many apps that I don't like and could not delete, in addition I could not install apps from Goolge Play because this phone is sold in China market. To solve these problems, I have to get root permission on the phone.

After a weekend study, I realized that a major way is to flash the phone using some 3rd party ROMs. I don't like this way . With further study, I searched out a Linux security bug by chance. This bug also impacted Android. A hacker has exploited it for android. This is really good news to me. However, it's not easy to figure out the offsets for my phone: I know nothing about ARM assembly; run-as is statically linked, stripped and symbols are obfuscated, making it difficult to understand the binary; I installed binutils-arm-linux-gnueabi on my linux desktop but arm-linux-gnueabi-objdump does not give me useful info.

In the end, I found 2 resources helped me a lot:
- android run-as source code
- IDA Disassembler 6.2 demo for Linux (I should thank this great tool)
These two resources helped me understand the assembly codes of run-as. In addition, analyzing run-as binary of Transformer Prime 4.0.3 helped me how to find the offsets for my phone because the Transformer Prime's offsets are already known.

The remaining is simple:
- Download and install android sdk
- run "android" from cmd line, add platform-tools for using adb.
- setup udev (for linux), follow this guide. After making changes, /etc/init.d/udev restart
Enable debugging mode and disabling fastboot on the phone. Connect to my phone using "adb shell" from Linux:
- push "mempodroid" to /data/local/tmp on the phone.
- run the magic "mempodroid"using the offsets that I figured out, become root!
- remount /system of the phone in read-write mode:
mount -o remount,rw /system
- deleted un-wanted apps in /system/app, /system/delapp, be careful when being root! (before doing this, I had removed as many apps as possible from app manager of the phone). For safety, I backed up them to sdcard using "cat" in adb shell.
- To make Google Play work, I copied the following apks to /system/app:
GoogleLoginService.apk
GoogleServicesFramework.apk
OneTimeInitializer.apk
Vending.apk
These apks could be accquired from cyanogenmod website.
- Type "reboot" from adb shell.

It's much better now. I prefer "temporary root", it's safer.

Optimize fonts and memory on Solaris 11 desktop

2011-12-01T18:22:00.001-08:00

Solaris 11 was released on Nov. 9th, 2011, this is the result of 7 years of huge engineering efforts since Solaris 10. Although Solaris is mainly designed for servers nowadays, Solaris is also good for desktop usage: ZFS, timeslider, DTrace, etc. Solaris 11 comes with GNOME 2.30 which is a very stable version (I'm disappointed with GNOME 3 shell or Ubuntu Unity) and Nimbus theme (created by Sun) which is my favorite theme.

I made some changes to the default settings of Solaris 11 desktop to suit my needs.

Fonts

The most important part for the user experience on the desktop might be fonts. Different people have difference preferences on fonts and the same font has different effects on various font-size and screen resolutions, therefore I need take some time to test different fonts and find the appropriate ones for my desktop. In the end, I chose the following fonts for my desktop screen (20.1", 1600x1200 LCD): "Nimbus Sans" for English font, "WenQuanYi Micro Hei" for Chinese font. Below are the steps for setup:

Change Desktop Appearance

Right click the desktop -> Desktop Appearance... -> Fonts, set the Application, Document, Desktop, Window title font to "Nimbus Sans", size "10" and set the Fixed width font to "Liberation Mono". (Note: do not use Chinese fonts in Desktop Appearance, because some Chinese fonts at some sizes make "i" looks "l")

Install Chinese fonts.

Solaris 11's default Chinese font is "AR PL ShanHeiSun Uni" (文鼎PL細上海宋Uni), which is a free, long history, high-quality font on Unix/Linux. But I want to use a more modern font. Solaris 11 software repository include "WenQuanYi Zen Hei" font ( To view which fonts are available, open Package Manager -> System -> Fonts) , but I personally feel it does not fit my screen. "WenQuanYi Micro Hei" is a better option.To install it:

Download from here
Unpack the downloaded file
mkdir -p ~/.fonts
Copy the "wqy-microhei.ttc" file to ~/.fonts
(optional) Run "fc-cache" in terminal.
To verify: run "fc-list | grep WenQuanYi", you should see that.

Change the system fonts configuration
The "Sans-serif", "Serif" and "Monospace" fonts are mapped to the default system fonts in /etc/fonts, to customize it, I did:

mkdir -p ~/.fonts.conf.d/
cp /etc/fonts/conf.d/60-latin.conf ~/.fonts.conf.d/
cp /etc/fonts/conf.d/65-nonlatin.conf ~/.fonts.conf.d/
edit ~/.fonts.conf.d/60-latin.conf
move Nimbus fonts to the front in each section so that "serif" maps to "Nimbus Roman No9 L", "sans-serif" maps to "Nimbus Sans L", and "monospace" maps to "Liberation Mono".
edit ~/.fonts.conf.d/65-nonlatin.conf
add "WenQuanYi Micro Hei" before "WenQuanYi Zen Hei" for both "serif" and "sans-serif", and add "WenQuanYi Micro Hei Mono" before "AR PL ShanHeiSun Uni" for "monospace".
To verify: run "fc-match sans", "fc-match monospace" and "fc-match serif" in terminal.

Memory
My x86 workstation has very limited memory, so I need to make changes to limit the unnecessary memory use. Here are options that you can do to save the memory:

Desktop Appearance -> Visual Effects, choose "None"
System -> Preferences -> File Management, disable "Preview"
System -> Preferences -> File Indexing, disable "indexing"
System -> Preferences -> Startup Applications, disable anything you don't want
Only keep the necessary Applets on your task bar
Check your and root cron jobs (sudo crontab -l), disable unnecessary jobs with "crontab -e"
Check the SMF services using "svcs" command, disable unnecessary services with "svcadm disable" command

Cloud Foundry

2011-11-16T18:12:00.001-08:00

A little background
Cloud Foundry project is an open-source platform-as-a-service (PaaS) project initiated by VMware. CloudFoundry.com is a VMware hosted, managed and supported PaaS environment. Micro Cloud Foundry is a complete version of Cloud Foundry that runs in a virtual machine on a developer’s Mac or PC.

Try Cloud Foundry
There are 2 ways: install it from github or use Micro Cloud Foundry virtual machine image provided by cloudfoudry.com. The second method is relatively simple and requires an CloudFoundry.com account. Because I already have an account, I used the second method.
The testing was not very smooth by following Micro Cloud Foundry Quick Start guide. The major problem was the network issue: the VM image uses the bridged network, however, in my env, it bound to the wrong network physical adapter (you should be careful about this especially when you have multiple network adapters or you have Virtualbox installed). The cloud foundry setup didn't pointed out this problem. This caused me to use vcap.me as the domain.
To fix above issue, I need use the virtual network editor to change the binding from automatic to a dedicated network adapter. The VMware player doesn't install the virtual network editor by default, I had to follow this guide to manually extract the installation file (VMware-player-xxxx.exe /e folder-name) , find the network.cab and extract the vmnetcfg.exe and copy it to VMWare Player folder. (C:\Program Files (x86)\VMware\VMware Player).
Another issue is VCAP ROUTER: 404: after setting up the micro cloud, vmc command failed with below message: Error (JSON 404): VCAP ROUTER: 404 - DESTINATION NOT FOUND I tried reconfiguring the domain in micro cloud setup menu, the problem was gone.

Thoughts
As for cloud computing, provisioning VM instances or application instances is relatively easy, the resource management is a challenge.

Build FBReader on Solaris

2011-10-28T01:21:00.000-07:00

Here are my steps.

Environment:

Solaris 11 Express x64

Gnome desktop

gcc-3.4.3

PATH=/usr/sfw/bin:/usr/gnu/bin:/usr/bin:/bin

Step 1: Build the dependencies first

FriBidi-0.10.9 , liblinebreak-2.1 ,

$./configure; make; sudo make install

These libraries will be installed in /usr/local/lib

Step 2: Build FBReader-0.12.10

Need some hacks here.

1) makefiles/arch/desktop.mk

13 CFLAGS = -pipe -fno-exceptions -Wall -Wno-ctor-dtor-privacy -W -DLIBICONV_PLUG -DDO_ICONV_CAST
14 LDFLAGS = -L /usr/local/lib -R /usr/local/lib

2) fbreader/src/formats/fb2/FB2Reader.h

It seems "_P" had been defined somewhere and cause enum declaration errors.

     23 #include <ZLXMLReader.h>
     24
     25 #undef _P
     26
     27 class FB2Reader : public ZLXMLReader {

3) zlibrary/core/src/unix/library/ZLibrary.cpp

Thanks to Rod Evans' great catch, I added an extra dlerror() to make sure it returns NULL before calling dlsym().

    128         void (*initLibrary)();
    129         dlerror();
    130         *(void**)&initLibrary = dlsym(handle, "initLibrary");

4) make; sudo make install

5) /usr/bin/FBReader & (or run it from GNOME menu: start -> Office ->E-book reader)

Have fun.

crazy CSS, thanks DTrace

2011-03-03T02:55:00.000-08:00

I hate many enterprise applications because of very unpleasant user experiences. Here's a recent example.
When I opened an application in firefox browser, my computer nearly hanged. Using basic Solaris commands prstat, mpstat, etc, I found displaying this page caused huge number of (40K) context switches per second.
At first, I thought it might be because of some ugly javascripts. So I first checked the source of this web page, well it contains several big java script files (sum up over 1MB), then I disabled the javascript in browser, however, the problem remained. I also tried offline browsing, it didn't solve the problem, so this is not a networking issue.

What's wrong?

~# truss -c -p 1224 (#1244 is firefox process ID)
^C
syscall               seconds   calls errors
read                     .016    3097     777
write                    .022    2126       1
open                     .000       2
close                    .000       4
fstat                    .000       2
access                   .000       1       1
ioctl                    .005    1145
fdsync                   .003       1
fcntl                    .000       6
lwp_park                 .236   28575     330
lwp_unpark               .177   28340
sigaction                .000      52
...

It looks like lwp_park/lwp_unpark are evils. Use a Dtrace script to find who caused this.

~# cat lwp_park.d
#!/usr/sbin/dtrace -qs
syscall::lwp_park:entry
/pid==$target/
{
@[ustack()]=count();
}
END
{
trunc(@, 10);
}

./lwp_park.d -p 1244

Then switch to the problemed web page in firefox, go back to that terminal running DTrace, type Ctrl-C.
The DTrace output the user stacks for causing both lwp_park() and lwp_unpark() which is interesting. After demangling C++ symbols, I got:

              libc.so.1`__lwp_unpark+0x19
              libnspr4.so`PR_Unlock+0x102
              libxul.so`unsigned TimerThread::RemoveTimer(nsTimerImpl*)+0x7e
              libxul.so`unsigned nsTimerImpl::Cancel()+0x32
              libxul.so`unsigned imgContainer::FrameUpdated(unsigned,nsIntRect&)+0xbd
              libxul.so`void row_callback(png_struct_def*,unsigned char*,unsigned long,int)+0x187
              libxul.so`MOZ_PNG_proc_IDAT_data+0x190
              libxul.so`MOZ_PNG_process_data+0x4ce
              libxul.so`unsigned ReadDataOut(nsIInputStream*,void*,const char*,unsigned,unsigned,unsigned*)+0x8c
              libxul.so`unsigned nsStringInputStream::ReadSegments(unsigned(*)(nsIInputStream*,void*,const char*,unsigned,unsigned,unsigned*),void*,unsigned,unsigned*)+0x47
              libxul.so`unsigned nsPNGDecoder::WriteFrom(nsIInputStream*,unsigned,unsigned*)+0x3a
              libxul.so`unsigned imgContainer::ReloadImages()+0x339
              libxul.so`unsigned imgContainer::GetCurrentFrameRect(nsIntRect&)+0x1a1
              libxul.so`unsigned imgRequest::NotifyProxyListener(imgRequestProxy*)+0x10f
              libxul.so`unsigned imgRequestProxy::Clone(imgIDecoderObserver*,imgIRequest**)+0xe7
              libxul.so`unsigned nsImageLoader::Load(imgIRequest*)+0x52

libxul.so`already_AddRefednsImageLoader::Create(nsIFrame*,imgIRequest*,int,nsImageLoader*)+0x57
              libxul.so`void nsPresContext::SetupBackgroundImageLoaders(nsIFrame*,const nsStyleBackground*)+0x98
              libxul.so`void nsCSSRendering::PaintBackgroundWithSC(nsPresContext*,nsIRenderingContext&,nsIFrame*,const nsRect&,const nsRect&,const nsStyleBackground&,const nsStyleBorder&,unsigned,nsRect*)+0x93f
              libxul.so`void nsCSSRendering::PaintBackground(nsPresContext*,nsIRenderingContext&,nsIFrame*,const nsRect&,const nsRect&,unsigned,nsRect*)+0xea
            24658

              libc.so.1`__lwp_park+0x19
              libc.so.1`cond_wait_queue+0x60
              libc.so.1`cond_wait_common+0x1eb
              libc.so.1`__cond_timedwait+0x66
              libc.so.1`cond_timedwait+0x27
              libc.so.1`pthread_cond_timedwait+0x24
              libnspr4.so`PR_WaitCondVar+0x24a
              libxul.so`unsigned TimerThread::Run()+0x11c
              libxul.so`unsigned nsThread::ProcessNextEvent(int,int*)+0x121
              libxul.so`int NS_ProcessNextEvent_P(nsIThread*,int)+0x2d
              libxul.so`void nsThread::ThreadFunc(void*)+0xa7
              libnspr4.so`_pt_root+0xe7
              libc.so.1`_thrp_setup+0x9b
              libc.so.1`_lwp_start
            26335

The stack of lwp_unpark() is often more interesting, because lwp_park() is usually caused by waiting for a condition variable which does not directly tell you what the application is waiting for. While if there are many lwp_park(), then there are many lwp_unpark() accordingly, so I looked at lwp_unpark() stack. I'm not a browser expert, from the function name, the firefox seemed to do rendering background based on CSS. Then I checked the CSS file which size is 150KB, Damn, there're 571 occurences like below:

{background: url(../xxx/yyy/zzz.png)}  

Then I wanted to check how many times nsCSSRendering::PaintBackground() was called by using another DTrace script.

From:

libxul.so`void  nsCSSRendering::PaintBackground(nsPresContext*,nsIRenderingContext&,nsIFrame*,const  nsRect&,const nsRect&,unsigned,nsRect*)+0xea  

#!/usr/sbin/dtrace -s
pid$target:libxul:__1cOnsCSSRenderingPPaintBackground6FpnNnsPresContext_rnTnsIRenderingContext_pnInsIFrame_rknGnsRect_9AIp7_v_:ea
{
@ =count();
}
Ran this DTrace script, and switched to the problemed web page in firefox. this time DTrace output:
~# ./pid.d -p 1244
dtrace: script './pid.d' matched 1 probe
^C
422

It indicated rendering this web page caused the firefox painted background 422 times (via X window), crazy!!! On the other hand, it looks like firefox's libxul.so should be improved when it renders these images. It caused about 25,000 context switches. Is it because that rendering the images is in asynchronous way and the rendered data are in local memory or on fast network so that context switches occurred so fast?

Workaround: In firefox settings, Preference -> Contents -> Load images automatically -> Exceptions -> add this website to block list.

JiuZhai_HuangLong

2010-10-26T03:17:00.000-07:00

JiuZhai and HuangLong are my wife's dream vacation places for years. This October we finally embarked on the journey. The cost of 3-days trip was very high (even more expensive than some overseas travel), but the view is very beutiful and unique. More photos taken by me can be found here, here.

Sun set in China

2010-09-02T03:26:00.000-07:00

Sun, my favorite brand.
Sun, my favorite logo.
Sun, the best engineering culture I have met with.
Sun, the most important career experience for me.
Sun, set in China on August 31st, 2010.

MeeGo!

2010-07-18T07:30:00.001-07:00

I don't want to use my laptop provided by the company any more: 1) it's 4 years old. 2) overheating problem. I decided to buy a device with my own money. I don't want to invest much because any computer becomes out of dated very soon. The requirement are: light, fast to start, energy efficient. iPad is not my choice because I need do some coding occasionally.

Finally I chose a netbook ViewSonic VNB 102p, it is based on Intel N450 Atom platform, the price is reasonable (less than RMB2000). I evaluated several OSes and installed MeeGo and Ubuntu Remix 10.04, these are designed for netbooks. Amazingly, all devices work out of box on these 2 OSes.

When I started to use this new netbook, I found I spent more time on MeeGo than Ubuntu. It might be because everything is fine on Ubuntu but I need some time to make some applications work on Meego. And Meego looks more simple, faster to start. I installed and customized thunderbird (for mail), openoffice 3.2.1 and skype (for voice chat), compiled Shrew (for VPN) and Shotwell 0.6.1 (for managing photos and online albums). Now I have used it for one month and this netbook can meet my demands for daily use.

Why flying east takes shorter time

2010-06-17T20:17:00.000-07:00

I'm not a frequent traveler. Some years ago I found flying from west of U.S to east U.S. took 1 hour less time than return trip, I didn't find answer at that time. Recently I decided to investigate this by the help of google search.

The answer is simple: WIND . But wait, why do most people in the world find flying east takes shorter time, does the wind over Earth surface always flies east? Then I studied some flights schedule around the equator such as Singapore to Tokyo and some flights in South America, there's no significant difference in flying time between goes and come-backs. After I read some articles, below factors should be considered:

- Wind and Earth Rotation or Prevailing Winds . You know the directions of the winds in different areas of the world.

- The locations of most cities with flights. Open the world map, you can find the much more cities are located between lattitude 30 and 60, in this area, the direction of wind is from the west to the east.

- You should consider long distance flights (>3 hours), otherwise the airplane may not spend most of time at cruise speed, and many other factors (such as reserving time for air traffic control) can affect the flying time.

- Coriolis effect. Yes, it affects the direction of the wind and it also affects the trails of airplanes, i.e. when the airplane flies from low lattitude to high lattitude, the plane will be deflected to the east. However, Coriolis itself nearly does not affect the differences of flying time between goes and come-backs. For example, if you fly northwest from Singapore to North Eurpoe, the real flying distance may be longer than you imagine from the map, the return trip is also longer.

Am I right? I'm not 100% sure. I'm just trying to explain the reason :)

10 years later, Beijing

2010-05-13T23:31:00.000-07:00

It has been nearly 10 years since I moved away from Beijing. In end of April 2010,I took a vacation with my family in Beijing. A lot of changes took place in 10 years: more people, more subways, soaring house prices... We visited some famous places, the Square, the Great Wall, Beijing Zoo, etc. Nice feeling.I love vacations more and more...

Strugling with the partition table

2010-04-18T03:31:00.000-07:00

I hate to use MS Windows: 1) insecure 2) dog slow after patching for years. Recently I decided to try Solaris Express or the new OpenSolaris. The first step is to leave some space for Solaris partition, so I deleted a Windows partition.

Now the partition table looks like:

| Free (for solaris) | Ubuntu | Windows | extened partition including several ntfs and linux swap|

Unfortunately, after I installed the solaris, both the existing Windows and Ubuntu failed to boot. For Windows, the error message is: "Windows\System32\ntoskrnl.exe is missing", for Ubuntu, it jumped to grub command directly.

I guess the reason was probably the partition number changed. I logged into Solaris, run format -> fdisk command, the result is below:

      Partition   Status    Type          Start   End   Length    %
      =========   ======    ============ =====   ===   ======   ===
          1                 Linux native   2551 5099    2549     21
          2                 IFS: NTFS      5100 7011    1912     16
          3                 EXT LBA      7012 12160    5149     42
          4       Active    Solaris2          1 2550    2550     21

then use ntfscat to show the content of boot.ini of Windows partition:

bash-4.0# ntfscat /dev/dsk/c1d0p2 boot.ini
[boot loader]
timeout=5
default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS
[operating systems]
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Home Edition" /noexecute=optin /fastdetect
C:\Ubuntu81.img="Ubuntu 8.10"

You can see the partition number "1" is not correct since the Windows partition number has changed to "2". So to fix it: make a new boot.ini (also need unix2dos command to add "/r" for Windows), then overide it on Windows:
ntfscp /dev/dsk/c1d0p2 ./newboot.ini boot.ini
Also, change the /rpool/boot/grub/menu.lst to correct the partition numbers to:
...
title Windows
        rootnoverify (hd0,1)
        chainloader +1
title Ubuntu8.1
        rootnoverify (hd0,0)
        chainloader +1
...

Reboot the system, I can start Windows normally.

Next, for Ubuntu problem, I need to get the content of Ubuntu grub menu.lst. Solaris currently does not have built-in supports for linux ext2/ext3 file system. I need to install some packages by following the guide, then, mount the Ubuntu partition:
mount -F ext2fs /dev/dsk/c1d0p1 /mnt
and got the content of Ubuntu grub menu.lst:

...
title           Ubuntu 8.10, kernel 2.6.27-14-generic
uuid            2786d26c-beb2-43e6-93a8-1cb3be921f23
kernel          /boot/vmlinuz-2.6.27-14-generic root=UUID=2786d26c-beb2-43e6-93a8-1cb3be921f23 ro quiet splash
initrd          /boot/initrd.img-2.6.27-14-generic
quiet
...

Reboot the system, select to boot Ubuntu in Solaris's grub menu, I was prompted to enter grub command. I tried to manually input: root (hd0,0) and the content of "Kernel....", "initrd....", then boot command, however, the system failed to boot up in the process.

So there must be somthing wrong with the Ubuntu's grub (My Ubuntu's grub was installed in Ubuntu's partition not the MBR), it seems I need to re-install the grub:
grub> root (hd0,0)
grub> install /boot/grub/stage1 (hd0,0) /boot/grub/stage2 p
grub> boot

O.K. The problem is resolved.

Lessons:
Don't delete a partition then create a new partition, it would possibly cause the partition numbers to change, making all things suck.

Shanghai Flower Port

2010-04-03T21:47:00.000-07:00

This spring comes so late. At China traditional QingMing holiday, my family went to Shanghai Flower Port. The park is famous for tulips and other flowers. It's located at faraway and seaside suburb of Shanghai, about 70km from downtown. The ticket price is RMB100, and free for little children. In this park, I can feel much fresher air and see beatuiful flowers.

For more photos, see here.

Send SOS to twitter in one button

2010-03-17T00:56:00.000-07:00

Nowadays location based applications become more and more popular. Maybe the simplest location based app is sending emergency message with your location. For example, if you go to a place with potential risk, when the accident happens, you might not have enough time to send SMS or make a call to tell where you are exactly.
My little project ttlite is for this: you can preset your SOS message and turn on GPS on your phone, in case the accident happens, you just push one button to send SOS with your location to twitter. You have known about twitter's great power in spreading messages instantly, right?
It is here: http://code.google.com/p/ttlite/

This winter in Shanghai

2010-02-13T05:10:00.000-08:00

Last night it was very cold, this morning I found the wall behind the curtain was wet because of the cendensed moisture in the warm room.
It's sunny today, I went to a park nearby with my family.Suddenly we found ice slush, it's the first time I saw it in Shanghai. Very interesting. The photos are here:

JavaFX colorlines game (on JavaFX 1.2)

2010-02-09T01:49:00.000-08:00

One of good things at Sun is that there are always new technologies to play with, even if it's not related to my daily job. In November, 2008, I did a simple implementation of colorlines game with JavaFX, of course it runs on JavaFX 1.0.

Over one year passed, I know there's a lot of changes from JavaFX 1.0 to JavaFX 1.2. So today I decided to upgrade it. Fortunately, I don't need change one line of codes, it passed compilation with JavaFX1.2 on Netbeans 6.8. However, it had problem during runtime on Windows XP: running as applet is O.K., but running as web start did not show anything. Then I changed to ubuntu linux and opensolaris, everything worked fine. I checked the source codes and Java debug console, but could not find any clue, It's so weird.

Anyway, JavaFX 1.2 showed faster than JavaFX 1.0. I put the codes to Google Code. If anybody can point out the problem I will appreciate that very much.

The best tribute I've ever read about Sun Microsystems

2010-02-04T23:57:00.000-08:00

This was written by @jimparkinson on his blog :

Today The Sun will Rise

About this blog (why I am here)

2010-02-04T01:19:00.000-08:00

I had several blogs, SDN China, msn live spaces, javaeye, and Sun blog. Due to various reasons (company rules, privacy protection, censorship, etc), I emptied or discontinued them.
Currently I decide to use blogspot.com as my blog site because it's free (free of charge and free to speak, no censorship), although blogspot.com is blocked by some countries, which means the users need to work around the firewall to read/write blogs on blogspot.com.