EngineeringException

Interactive time series data analysis using Apache Kafka and Druid

2014-12-31T17:49:00.000+09:00

Japanese version is here.

Abstract

Druid is a database system. This is composed of a component for collecting time series data in realtime and a component for analyzing the datasets with queries from users.

Druid says that itself is based on Dremel (VLDB2010) and PowerDrill (VLDB2012). In other words, Druid is like Cloudera Impala. But Druid doesn't ask for high-performance machines. According to Druid Production Cluster Configuration, it seems to require some huge RAM especially.

References

druid-io/druid

Main repository for Druid. This contains source codes of servers and examples for extensions.

druid-io/druid-api

A repository for APF of Druid. This provides some information for creating extensions for parsing data, interpreting query and so on.

metamx/java-util

A library repository for Druid. There is an interface for parser for some reason.

Druid

SIGMOD2014 Parper

Druid: Interactive Queries Meet Real-time Data: Open Source Convention - O'Reilly OSCON, July 20 - 24, 2014 in Portland, OR

OSCON2014 Slide

https://www.youtube.com/results?search_query=druid+real-time

There are some presentation about Druid.

Collecting time series data

Currently, the stable version of Druid is 0.6.160. I use this version. For understanding the details, I refer druid-io/druid at druid-0.6.160 and druid-io/druid-api at druid-api-0.2.14.1. The Druid codes is unstable as we know from commit history. Therefore, for increasing the life of my works I have shied away from writing codes for just Druid. I equip external systems for inability of default of Druid in place of creating extensions for Druid.

For building my system, I use Vagrant and Docker. I got a Vagrantfile from coreos/coreos-vagrant and modified it for setting aside 4GB memory.

diff --git a/Vagrantfile b/Vagrantfile
index 2a1572e..c877612 100644
--- a/Vagrantfile
+++ b/Vagrantfile
@@ -13,7 +13,7 @@ $num_instances = 1
 $update_channel = "alpha"
 $enable_serial_logging = false
 $vb_gui = false
-$vb_memory = 1024
+$vb_memory = 4096
 $vb_cpus = 1

 # Attempt to apply the deprecated environment variable NUM_INSTANCES to

$ vagrant up  # starting VM
$ vagrant ssh  # entering VM
CoreOS (alpha)
core@core-01 ~ $ mkdir zookeeper kafka druid clients
core@core-01 ~ $ which docker
/usr/bin/docker

The following of consoles are in VM.

Building dependent system

Druid is a distributed system composed of some machines. For their coordination, Zookeeper is used. Dockerfile for launching Zookeeper is https://github.com/laclefyoshi/druid_test/blob/master/zookeeper.

$ cd zookeeper
$ docker build -t laclefyoshi/zookeeper:1.0 .
$ docker run -d -p 2181:2181 --name zookeeper laclefyoshi/zookeeper:1.0

Druid caches time series data from Firehose. We have some choices with extensions for Firehose. For example, Storm Trident State is used as Firehose in a book: Storm Blueprints: Patterns for Distributed Real-time Computation.

I use Apache Kafka (Version 0.8) as Firehose. I use foregoing Zookeeper for Kafka and Druid as shared system. Dockerfile for launching Kafka is https://github.com/laclefyoshi/druid_test/tree/master/kafka.

$ cd kafka
$ docker build -t laclefyoshi/kafka:1.0 .
$ docker run -d -p 9092:9092 --link zookeeper:zookeeper --name kafka laclefyoshi/kafka:1.0

Building Druid

More than one Druid service run on a Druid system. A Druid system defines the name, Firehose as a source of time series data, structure of the data and so on.

There are two modes for building Druid system: standalone mode and cluster mode. In standalone mode, Druid system runs on one machine and runs one Druid service. In cluster mode, it runs on a number of machines assigned roles and runs multiple Druid services. I introduce the simple standalone mode. Dockerfile for launching a Druid system and service is https://github.com/laclefyoshi/druid_test/tree/master/druid

$ cd druid
$ docker build -t laclefyoshi/druid:1.0 .
$ docker run -d -p 8083:8083 --link zookeeper:zookeeper --link kafka:kafka --name druid laclefyoshi/druid:1.0

Finally, I set up a container for sending time series data to Kafka. This is called Kafka Producer and sends the following data.

{"timestamp": "2014-12-21T10:17:00", "number": 7, "word": "fox"}

timestamp is a time and date of the data. number is a random number. word is an English word. Dockerfile for launching Kafka Producer is https://github.com/laclefyoshi/druid_test/tree/master/clients.

$ cd clients
$ docker build -t laclefyoshi/clients:1.0 .
$ docker run -d --link zookeeper:zookeeper --link kafka:kafka --name clients laclefyoshi/clients:1.0

Now, I have finished to building containers for using Druid. clients container send time series data to kafka container. And druid container catches and collects the data from kafka container. zookeeper container follows them.

$ docker ps
CONTAINER ID        IMAGE                       COMMAND                CREATED              STATUS              PORTS                    NAMES
4021acc277c7        laclefyoshi/clients:1.0     "bash send_data.sh"    About a minute ago   Up About a minute                            clients
9280e328e97d        laclefyoshi/druid:1.0       "bash ./start_druid_   9 minutes ago        Up 9 minutes        0.0.0.0:8083->8083/tcp   druid
9e17bab05cc2        laclefyoshi/kafka:1.0       "bash ./kafka_create   11 minutes ago       Up 11 minutes       0.0.0.0:9092->9092/tcp   kafka
c0614de5dce0        laclefyoshi/zookeeper:1.0   "./bin/zkServer.sh s   25 minutes ago       Up 25 minutes       0.0.0.0:2181->2181/tcp   zookeeper

Analyzing time series data using Druid

After some minutes of building system, I post the following query to Druid with HTTP POST.

$ cat query.json
{
  "queryType":"timeBoundary",
  "dataSource":"druid_test",
  "intervals":["2014-12-01/2014-12-31"]
}
$ curl -X 'POST' -H 'Content-Type:application/json' -d @query.json http://localhost:8083/druid/v2/?pretty

Then Druid replies the following response.

[ {
  "timestamp" : "2014-12-21T10:17:00.000Z",
  "result" : {
    "minTime" : "2014-12-21T10:17:00.000Z",
    "maxTime" : "2014-12-21T10:51:00.000Z"
  }
} ]

This query is for identifying the time range of data for analysis on a Druid service: druid_test. This name of the service is defined in druid.service in runtime.properties and the designation for timestamp field is defined in the parser configuration in realtime.spec.

Next, I post a query for counting words in each 15 minutes in 30 minutes, so I get 2 results by 1 query. In addition, I get the sorting result in descending order of counts.

$ cat query.json
{
  "queryType": "groupBy",
  "dataSource": "druid_test",
  "granularity": "fifteen_minute",
  "dimensions": ["word"],
  "limitSpec": {
     "type": "default",
     "columns": [
       {"dimension": "count_word", "direction": "DESCENDING"}
      ],
     "limit": 10
  },
  "aggregations": [
    {"type": "count", "name": "count_word"}
  ],
  "intervals": ["2014-12-21T10:20/2014-12-21T10:50"]
} 

$ time curl -X 'POST' -H 'Content-Type:application/json' -d @query.json http://localhost:8083/druid/v2/?pretty
[ {
  "version" : "v1",
  "timestamp" : "2014-12-21T10:15:00.000Z",
  "event" : {
    "count_word" : 4,
    "word" : "lazy"
  }
}, {
  "version" : "v1",
  "timestamp" : "2014-12-21T10:15:00.000Z",
  "event" : {
    "count_word" : 4,
    "word" : "the"
  }
}, {
  "version" : "v1",
  "timestamp" : "2014-12-21T10:15:00.000Z",
  "event" : {
    "count_word" : 3,
    "word" : "over"
  }
}, {
  "version" : "v1",
  "timestamp" : "2014-12-21T10:15:00.000Z",
  "event" : {
    "count_word" : 3,
    "word" : "quick"
  }
}, {
  "version" : "v1",
  "timestamp" : "2014-12-21T10:15:00.000Z",
  "event" : {
    "count_word" : 2,
    "word" : "dog"
  }
}, {
  "version" : "v1",
  "timestamp" : "2014-12-21T10:15:00.000Z",
  "event" : {
    "count_word" : 2,
    "word" : "jumps"
  }
}, {
  "version" : "v1",
  "timestamp" : "2014-12-21T10:15:00.000Z",
  "event" : {
    "count_word" : 1,
    "word" : "fox"
  }
}, {
  "version" : "v1",
  "timestamp" : "2014-12-21T10:30:00.000Z",
  "event" : {
    "count_word" : 6,
    "word" : "brown"
  }
}, {
  "version" : "v1",
  "timestamp" : "2014-12-21T10:30:00.000Z",
  "event" : {
    "count_word" : 6,
    "word" : "quick"
  }
}, {
  "version" : "v1",
  "timestamp" : "2014-12-21T10:30:00.000Z",
  "event" : {
    "count_word" : 4,
    "word" : "fox"
  }
} ]
real    0m0.035s
user    0m0.003s
sys     0m0.001s

The processing the query spends about 0.03 seconds in Druid. According to the results, clients sends one or two data in a minute. Between 10:15 and 10:30, "lazy" and "fox" appear 4 times. Between 10:30 and 10:45, "brown" appears 6 times.

This analysis is very simple counting data but suitable for practical use. For example, if the data is of Web servers, I can get the summary of logs based on IP addresses or User-Agents. For another example, if the data is of SSH servers, I can detect unauthorized invasion by counting the authentication failure.

For more practical use, Druid can interpreting other query other than groupBy. For example, there are sum for getting sum of values of specific fields, topN for getting ranking of counting, filter for filtering the data based on values of specific fields and so on.

Footnote

All system which I want to use as Firehose of Druid don't provide suitable formatted data for Druid. Therefore, in many cases, a system, for example Apache Storm, is needed for reprocessing time series data between the sources of data and Firehose.

text to speech in Google Chrome

2014-01-26T15:21:00.001+09:00

Google Chrome 33 beta provides Web Speech API. I tried to use it on JavaScript console of Google Chrome.

Some voices are provided by Google Chrome and others are provided by the platform (in my case, Mac provides the voices).

The message is needed to bind a text, a voice and the language.

An American speaks English.

> msg.text = "JavaScript (JS) is a dynamic computer programming language."
> msg.voice = voices[0]
> msg.lang = voices[0].lang  # "en-US"

> window.speechSynthesis.speak(msg)

A Japanese speaks English.

> msg.text = "JavaScript (JS) is a dynamic computer programming language."
> msg.voice = voices[7]
> msg.lang = voices[7].lang  # "ja-JP"

A Chinese speaks English.

> msg.text = "JavaScript (JS) is a dynamic computer programming language."
> msg.voice = voices[9]
> msg.lang = voices[9].lang  # "zh-CN"

In this case, the text interpreted as Chinese pinyin. Maybe lang attribute specifies how to interpret the text. But when "en-US" is specified as the lang, the voice changes to an American's voice.

A Japanese speaks Japanese.

> msg.text = "JavaScript（ジャヴァスクリプト）とは、プログラミング言語のひとつである。"
> msg.voice = voices[7]
> msg.lang = voices[7].lang

This case is failure, too. This speaks an escaped Japanese text.

> escape("JavaScript（ジャヴァスクリプト）とは、プログラミング言語のひとつである。")
"JavaScript%uFF08%u30B8%u30E3%u30F4%u30A1%u30B9%u30AF%u30EA%u30D7%u30C8%uFF09%u3068%u306F%u3001%u30D7%u30ED%u30B0%u30E9%u30DF%u30F3%u30B0%u8A00%u8A9E%u306E%u3072%u3068%u3064%u3067%u3042%u308B%u3002"

There are some problems for practical use, but this API will be used for pushing information from Web to people's ears. I think that the Web increasingly resembles TV.

Thanks.

software-defined radio with Gqrx on Mac

2013-12-28T15:34:00.000+09:00

Software-defined radio (SDR) is an inexpensive hobby for people who are interested in radio intercept.

I went to Akihabara last week and bought DVB-T+DAB+FM RTL2832U+R820T. This device is normally a receiver for one-segment broadcasting but is known internationally as a SDR device. I wasted little time in receiving radio.

I used Gqrx for controlling my SDR device. I thought it was painful to build rtl-sdr and GNU Radio and so on from scratch. Gqrx lacks documents, but I could smoothly understand how to use it. Simply stated, I just started the application, specified a frequency and adjusted the volume.

I'm listening to fire radio for Tokyo. It appears that incidents large and small occur every 30 seconds.

Thanks.

updating SSD firmware version in Mac mini with USB flash drive

2013-11-10T20:41:00.000+09:00

I bestirred myself and updated SSD firmware in my Mac.

Mac: Mac mini (Mid 2011)
OS: OS X Mavericks (10.9)
SSD: PLEXTOR PX-128M5P

Current Firmware version: 1.01
New Firmware version: 1.05 (released 2013-05-09)
URL

The instructions for updating firmware is written based on using CD/DVD drive, but as you may know, Mac mini doesn't have a CD/DVD drive. I found the method for updating firmware with an USB flash drive.

Will new Crucial SSD work if i dont...: Apple Support Communities

I downloaded PX-128M5Pro 1.05.iso from the official page and rEFIt-Syslinux.dmg from above URL.

I used the he following process.

$ hdiutil attach rEFIt-Syslinux.dmg
/dev/disk4              FDisk_partition_scheme
/dev/disk4s1            DOS_FAT_32                      /Volumes/SYSLINUX
/dev/disk4s2            Apple_HFS                       /Volumes/rEFIt

$ diskutil list
/dev/disk0
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *128.0 GB   disk0
   1:                        EFI EFI                     209.7 MB   disk0s1
   2:                  Apple_HFS Macintosh SSD           127.2 GB   disk0s2
   3:                 Apple_Boot Recovery HD             650.0 MB   disk0s3
/dev/disk1
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *500.1 GB   disk1
   1:                        EFI EFI                     209.7 MB   disk1s1
   2:                  Apple_HFS Macintosh HD            499.2 GB   disk1s2
   3:                 Apple_Boot Recovery HD             650.0 MB   disk1s3
/dev/disk2
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *640.1 GB   disk2
   1:                        EFI EFI                     209.7 MB   disk2s1
   2:                  Apple_HFS HD-PCTU2                639.8 GB   disk2s2
/dev/disk3
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:                            NO NAME                  *2.1 GB     disk3
/dev/disk4
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:     FDisk_partition_scheme                        *32.6 MB    disk4
   1:                 DOS_FAT_32 SYSLINUX                8.2 MB     disk4s1
   2:                  Apple_HFS rEFIt                   23.2 MB    disk4s2

$ diskutil unmountDisk /dev/disk3  # usb flash drive
Unmount of all volumes on disk3 was successful
$ diskutil unmountDisk /dev/disk4  # rEFIt-Syslinux.dmg
Unmount of all volumes on disk3 was successful

$ sudo dd if=/dev/disk4 of=/dev/disk3 bs=1m
31+1 records in
31+1 records out
32636928 bytes transferred in 13.280089 secs (2457584 bytes/sec)

$ diskutil eject /dev/disk3
$ diskutil eject /dev/disk4


## unplug and plug USB flash drive for remounting it
## /Volumes/SYSLINUX and /Volumes/rEFIt appear


$ hdiutil mount PX-128M5Pro\ 1.05.iso
/dev/disk3                                              /Volumes/SSD FLASH_

$ ls -l /Volumes/SSD\ FLASH_/isolinux/
total 5848
-r-xr-xr-x  1 yoshiyasu  staff   139B 11 20  2010 BOOTMSG.TXT*
-r-xr-xr-x  1 yoshiyasu  staff   2.8M  5  6  2013 BTDSK.IMG*
-r-xr-xr-x  1 yoshiyasu  staff   576B  2 23  2003 F1_HELP.TXT*
-r-xr-xr-x  1 yoshiyasu  staff   506B  8 21  2003 F2_LICEN.TXT*
-r-xr-xr-x  1 yoshiyasu  staff   1.4K  5 10  2003 F_ABOUT.TXT*
-r-xr-xr-x  1 yoshiyasu  staff   591B  6 12  2004 ISOLINUX.CFG*
-r-xr-xr-x  1 yoshiyasu  staff    14K 12 21  2008 isolinux.bin*
-r-xr-xr-x  1 yoshiyasu  staff    20K  9 25  2008 memdisk*

$ cd /Volumes/SYSLINUX
$ ls -l
total 3224
-rwxrwxrwx  1 yoshiyasu  staff    65K  5  3  2007 COMMAND.COM*
-rwxrwxrwx  1 yoshiyasu  staff    15K  3  3  2011 ldlinux.sys*
-rwxrwxrwx  1 yoshiyasu  staff    54K  3  3  2011 menu.c32*
-rwxrwxrwx@ 1 yoshiyasu  staff   237B  4  4  2011 syslinux.cfg*
-rwxrwxrwx  1 yoshiyasu  staff   1.4M  3  3  2011 ubninit*
-rwxrwxrwx  1 yoshiyasu  staff    25K  3  3  2011 ubnkern*
$ rm syslinux.cfg
$ cp /Volumes/SSD\ FLASH_/isolinux/* .
$ mv ISOLINUX.CFG syslinux.cfg


## don't eject USB flash drive but eject other external drives
## push the option key and reboot

## select the boot drive "rEFIt" -> "Boot Legacy OS from SYSLINUX"

I followed instructions, and the updating was finished in a few minutes.

I'm not sure the effect but certainly the SSD firmware revision shows 1.05 on the System Profiler.

Thanks.

flymake and scalastyle

2013-10-05T15:59:00.001+09:00

I've used checkstyle when coding Java. Because I have a greater opportunity to write Scala recently, I found scalastyle: checkstyle for Scala.

Checkstyle collaborates easily with emacs because the outputs are suitable for flymake, but scalastyle doesn't.

The following is an example of default outputs of scalastyle. If flymake gets this outputs, nothing happens.

That's why I wrote a script for converting outputs of scalastyle into a flymake-friendly format. I want to use the script on Linux and Mac, so I used python instead of sed (I know that sed on Mac is not the same as it on Linux)

The following is an example of reformed outputs of scalastyle.sh.

I stored the script into ~/.local/bin/scalastyle.sh and edited .emacs as follows.

That's a good time to migrate from emacs to IDE (the candidate is IntelliJ for now).

Thanks.

scalding with Oozie

2013-09-22T14:29:00.000+09:00

scalding is a great library for Hadoop. Previously, I wrote many UDFs for Pig, but it seemed like such a drag to write Java codes because those UDFs were almost halfway. I could write codes for data processing easy thanks to scalding and Scala.

And now, I use Oozie for scheduling Hadoop applications. For using scalding applications in Oozie, I need to use Java Action. The configuration for Java Action of Oozie is not complicated. I share a practical templates of workflow.xml and configuration.xml.

The following is the directory structure for this workflow.

.
├── lib
│   └── app.jar
└── workflow.xml

A note of caution is file element in workflow.xml. The first part of value delimited "#" specifies a hdfs path for a jar file and the second part decides a alias name. The alias name is used in values of

arg and so on.



Scalding provides the matrix library. I'm not unable to use it effectively, but the library really seems to useful because it can cover some tasks with numpy.



Thanks.

installing CDH into secured servers with Cloudera Manager
2013-09-16T15:47:00.000+09:00

I needed to install CDH4 (Cloudera's Distribution Including Apache Hadoop) into secured (closed) servers which can access only the internal network and a gateway server but can't access the external networks as well as the Internet.

Because I wanted to choose the most comfortable way to install and manage CDH, I decided to use Cloudera Manager (Standard Version). Therefore I could follow the Cloudera's guide / Installation Path A - Automated Installation by Cloudera Manager but I needed to do some works for using Cloudera Manager in the local network.

There are 4 servers in the secured network. Each server runs CentOS 6.4 (64bit). In what follows gateway01 is a gateway server which can access the Internet and the secured network and secured001 is a server in the secured network.

Firstly, I needed to build a local repository for yum (the Cloudera's guide).

(gateway01)$ mkdir cloudera
(gateway01)$ ls cloudera
cloudera-manager-installer.bin  # need to register for getting it: https://www.cloudera.com/content/support/en/downloads.html
(gateway01)$ mkdir cloudera/repo
(gateway01)$ mkdir cloudera/parcel
(gateway01)$ cd cloudera/repo
(gateway01)$ wget http://archive.cloudera.com/cm4/redhat/6/x86_64/cm/4/RPMS/x86_64/cloudera-manager-agent-4.7.1-1.cm471.p0.125.el6.x86_64.rpm
(gateway01)$ wget http://archive.cloudera.com/cm4/redhat/6/x86_64/cm/4/RPMS/x86_64/cloudera-manager-daemons-4.7.1-1.cm471.p0.125.el6.x86_64.rpm
(gateway01)$ wget http://archive.cloudera.com/cm4/redhat/6/x86_64/cm/4/RPMS/x86_64/cloudera-manager-server-4.7.1-1.cm471.p0.125.el6.x86_64.rpm
(gateway01)$ wget http://archive.cloudera.com/cm4/redhat/6/x86_64/cm/4/RPMS/x86_64/cloudera-manager-server-db-4.7.1-1.cm471.p0.125.el6.x86_64.rpm
(gateway01)$ wget http://archive.cloudera.com/cm4/redhat/6/x86_64/cm/4/RPMS/x86_64/enterprise-debuginfo-4.7.1-1.cm471.p0.125.el6.x86_64.rpm
(gateway01)$ wget http://archive.cloudera.com/cm4/redhat/6/x86_64/cm/4/RPMS/x86_64/jdk-6u31-linux-amd64.rpm
(gateway01)$ cd ../parcel
(gateway01)$ wget http://archive.cloudera.com/cdh4/parcels/latest/CDH-4.4.0-1.cdh4.4.0.p0.39-el6.parcel
(gateway01)$ wget http://archive.cloudera.com/cdh4/parcels/latest/manifest.json
(gateway01)$ cd ../../
(gateway01)$ scp -r cloudera secured001:./
(gateway01)$ ssh secured001

(secured001)$ cd ~/cloudera/repo/
(secured001)$ createrepo .
(secured001)$ cd ~/
(secured001)$ sudo cat <<EOF > /etc/yum.repos.d/local.repo
[myrepo]
name=myrepo
baseurl=http://secured001:8081/cloudera/repo
enabled=1
gpgcheck=0
(secured001)$ python -m SimpleHTTPServer 8081 &
(secured001)$ sudo ./cloudera/cloudera-manager-installer.bin --skip_repo_package=1

--skip_repo_package=1 in the last line means that Cloudera Manager don't access the Cloudera repository on the Internet (archive.cloudera.com) and access only the local repository.

Secondly, I accessed http://secured001:7180/ after finishing the installation process of Cloudera Manager and followed the guide. It's not hard to install the components of CDH through the web browser as long as I keep in mind that I need to indicate the local Parcel repository (http://secured001:8081/cloudera/parcel) here.

Just for the record, I failed to install Impala because of missing "impala" user. So I installed it separately.

Thanks.

Hosebird and Kafka: Scala version conflict
2013-05-25T16:34:00.002+09:00

I tried to integrate Hosebird (an official client library for Twitter Firehose) (version 1.4.0) and Kafka (A message queuing server) (0.7.0), but got the below error at runtime.

$ mvn clean package
$ java -cp target/hosebird_kafka-1.0-SNAPSHOT.jar hosebird_kafka.App a b c d
Exception in thread "hosebird-client-io-thread-0" java.lang.NoSuchMethodError: scala.runtime.RichInt.until(I)Lscala/collection/immutable/Range;
        at com.twitter.joauth.UrlEncoder$.apply(UrlEncoder.scala:42)
        at com.twitter.joauth.UrlEncoder.apply(UrlEncoder.scala)
        at com.twitter.hbc.core.endpoint.BaseEndpoint.addQueryParameter(BaseEndpoint.java:98)
        at com.twitter.hbc.core.endpoint.DefaultStreamingEndpoint.addDefaultParams(DefaultStreamingEndpoint.java:48)
        at com.twitter.hbc.core.endpoint.BaseEndpoint.getURI(BaseEndpoint.java:57)
        at com.twitter.hbc.core.HttpConstants.constructRequest(HttpConstants.java:71)
        at com.twitter.hbc.httpclient.ClientBase.run(ClientBase.java:134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
        at java.lang.Thread.run(Thread.java:680)

The reason for the NoSuchMethodError is version conflicts of Scala.

Hosebird (hbc-core-1.4.0) depends on joauth-3.2.3, joauth-3.2.3 depends on  and Kafka (kafka-0.7.0-incubating) depends on scala-library-2.8.0.

Because maven adopts the smaller version when multiple libraries depends on a same library with different versions, this time scala-library-2.8.0 is adopted as this project dependency. In the result, joauth-3.2.3 can't use a method of scala-library-2.9.2.

All versions of joauth in Maven Central Repository depend on scala-library-2.9.2. But I found old joauth in here directory. Apparently joauth-2.0.1 depends on scala-library-2.8.1.

After I installed joauth-2.0.1 in local repository, I added the dependency. I could create and run jar normally.

Thanks.

serjtag patches for avrdude-5.11.1
2013-03-31T15:18:00.000+09:00

I needed to update firmware of AVR-USBasp-B (original USBasp's page) which I bought at Akihabara

avrdude: Warning: Found USB device "USBasp" with old VID/PID! Please update firmware of USBasp!
avrdude: warning: cannot set sck period. please check for usbasp firmware update.

But I don't have another writer (Arduino runs in another project, so I don't use Arduino-ISP). I tried to use FTDI BitBang AVR-Writer with FT232R.

I needed to apply patches to avrdude for using FTDI BitBang AVR-Writer but the original avrdude-serjtag patches don't support avrdude-5.11.1 (latest version). So I edited the patches and created github repository: laclefyoshi/avrdude-serjtag · GitHub.

After building and installing avrdude (read README.md), I succeeded to update  firmware of AVR-USBasp-B.

$ sudo kextunload /System/Library/Extensions/FTDIUSBSerialDriver.kext

$ avrdude -C /opt/usr/avrdude/etc/avrdude.conf -c diecimila -P ft0 -p m8 -v

avrdude: Version 5.11.1, compiled on Mar 30 2013 at 17:18:59
         Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
         Copyright (c) 2007-2009 Joerg Wunsch

         System wide configuration file is "/opt/usr/avrdude/etc/avrdude.conf"
         User configuration file is "/Volumes/Macintosh HD/Users/yoshiyasu/.avrduderc"
         User configuration file does not exist or is not a regular file, skipping

         Using Port                    : ft0
         Using Programmer              : diecimila
avrdude: BitBang OK
avrdude: pin assign miso 3 sck 5 mosi 6 reset 7
avrdude: drain OK
         AVR Part                      : ATMEGA8
         Chip Erase delay              : 10000 us
         PAGEL                         : PD7
         BS2                           : PC2
         RESET disposition             : dedicated
         RETRY pulse                   : SCK
         serial program mode           : yes
         parallel program mode         : yes
         Timeout                       : 200
         StabDelay                     : 100
         CmdexeDelay                   : 25
         SyncLoops                     : 32
         ByteDelay                     : 0
         PollIndex                     : 3
         PollValue                     : 0x53
         Memory Detail                 :

                                  Block Poll               Page                       Polled
           Memory Type Mode Delay Size  Indx Paged  Size   Size #Pages MinW  MaxW   ReadBack
           ----------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
           eeprom         4    20   128    0 no        512    4      0  9000  9000 0xff 0xff
           flash         33    10    64    0 yes      8192   64    128  4500  4500 0xff 0x00
           lfuse          0     0     0    0 no          1    0      0  2000  2000 0x00 0x00
           hfuse          0     0     0    0 no          1    0      0  2000  2000 0x00 0x00
           lock           0     0     0    0 no          1    0      0  2000  2000 0x00 0x00
           calibration    0     0     0    0 no          4    0      0     0     0 0x00 0x00
           signature      0     0     0    0 no          3    0      0     0     0 0x00 0x00

         Programmer Type : ft245r
         Description     : FT232R Synchronous BitBang

 ft245r:  bitclk 230400 -> ft baud 115200
avrdude: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.00s

avrdude: Device signature = 0x1e9307
avrdude: safemode: lfuse reads as EF
avrdude: safemode: hfuse reads as D9

avrdude: safemode: lfuse reads as EF
avrdude: safemode: hfuse reads as D9
avrdude: safemode: Fuses OK

avrdude done.  Thank you.

$ avrdude -C /opt/usr/avrdude/etc/avrdude.conf -c diecimila -P ft0 -p m8 -U flash:w:usbasp.atmega8.2011-05-28.hex:i
avrdude: BitBang OK
avrdude: pin assign miso 3 sck 5 mosi 6 reset 7
avrdude: drain OK

 ft245r:  bitclk 230400 -> ft baud 115200
avrdude: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.00s

avrdude: Device signature = 0x1e9307
avrdude: NOTE: FLASH memory has been specified, an erase cycle will be performed
         To disable this feature, specify the -D option.
avrdude: erasing chip
 ft245r:  bitclk 230400 -> ft baud 115200
avrdude: reading input file "usbasp.atmega8.2011-05-28.hex"
avrdude: writing flash (4700 bytes):

Writing | ################################################## | 100% 2.25s

avrdude: 4700 bytes of flash written
avrdude: verifying flash memory against usbasp.atmega8.2011-05-28.hex:
avrdude: load data flash data from input file usbasp.atmega8.2011-05-28.hex:
avrdude: input file usbasp.atmega8.2011-05-28.hex contains 4700 bytes
avrdude: reading on-chip flash data:

Reading | ################################################## | 100% 1.29s

avrdude: verifying ...
avrdude: 4700 bytes of flash verified

avrdude: safemode: Fuses OK

avrdude done.  Thank you.

$ avrdude -C /opt/usr/avrdude/etc/avrdude.conf -c diecimila -P ft0 -p m8 -U hfuse:w:0xC9:m  # read http://www.fischl.de/usbasp/Readme.txt

$ sudo kextload /System/Library/Extensions/FTDIUSBSerialDriver.kext

I read Atmega8 with USBasp.

$ avrdude -C /opt/usr/avrdude/etc/avrdude.conf -c usbasp -P ft0 -p m8  -v

avrdude: Version 5.11.1, compiled on Mar 30 2013 at 17:18:59
         Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
         Copyright (c) 2007-2009 Joerg Wunsch

         System wide configuration file is "/opt/usr/avrdude/etc/avrdude.conf"
         User configuration file is "/Volumes/Macintosh HD/Users/yoshiyasu/.avrduderc"
         User configuration file does not exist or is not a regular file, skipping

         Using Port                    : ft0
         Using Programmer              : usbasp
         AVR Part                      : ATMEGA8
         Chip Erase delay              : 10000 us
         PAGEL                         : PD7
         BS2                           : PC2
         RESET disposition             : dedicated
         RETRY pulse                   : SCK
         serial program mode           : yes
         parallel program mode         : yes
         Timeout                       : 200
         StabDelay                     : 100
         CmdexeDelay                   : 25
         SyncLoops                     : 32
         ByteDelay                     : 0
         PollIndex                     : 3
         PollValue                     : 0x53
         Memory Detail                 :

                                  Block Poll               Page                       Polled
           Memory Type Mode Delay Size  Indx Paged  Size   Size #Pages MinW  MaxW   ReadBack
           ----------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
           eeprom         4    20   128    0 no        512    4      0  9000  9000 0xff 0xff
           flash         33    10    64    0 yes      8192   64    128  4500  4500 0xff 0x00
           lfuse          0     0     0    0 no          1    0      0  2000  2000 0x00 0x00
           hfuse          0     0     0    0 no          1    0      0  2000  2000 0x00 0x00
           lock           0     0     0    0 no          1    0      0  2000  2000 0x00 0x00
           calibration    0     0     0    0 no          4    0      0     0     0 0x00 0x00
           signature      0     0     0    0 no          3    0      0     0     0 0x00 0x00

         Programmer Type : usbasp
         Description     : USBasp, http://www.fischl.de/usbasp/

avrdude: auto set sck period (because given equals null)
avrdude: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.00s

avrdude: Device signature = 0x1e9307
avrdude: safemode: lfuse reads as DF
avrdude: safemode: hfuse reads as CA

avrdude: safemode: lfuse reads as DF
avrdude: safemode: hfuse reads as CA
avrdude: safemode: Fuses OK

avrdude done.  Thank you.

USBasp is small and good programmer. Recently, sometimes, I want to take time away from Arduino and use  AVR microcontrollers.

Thanks.

controlling digital camera with Arduino and light
2013-02-09T22:56:00.003+09:00

I bought a digital camera: EOS Kiss X6i last year.

This camera has an interface for remote controller. On examination, I knew this plug is 2.5mm stereo mini plug and has very simple mechanism. So I connected the camera and Arduino.

This page provides me with useful information.

Arduino read Analog-Pin 1 (connecting with a light sensor input). If the value of Analog-Pin 1 exceeds a certain value, Arduino outputs Digital-Pin 2 (focus) and 3 (shutter).

I think that taking pictures continuously is to place excessive strain on camera, so I set 30 seconds of delay.

The light sensor can replace other sensors or triggers. For example, Ethernet shield can be connected. I enjoy the extension of my camera's release.

Thanks.

using Source Code Pro in emacs
2012-10-21T09:06:00.002+09:00

I used Anonymous Pro in emacs previously. But I saw Adobe's blog and I was interested in this new font.

Hiragino Kaku Gothic Pro is a font included Mac OS X for Japanese.

There is a very well-recognized distinction between "0", "O", "1", "I" and "l" certainly.

Thanks.

Google Chrome supports MathML
2012-10-13T15:21:00.000+09:00

According to this post, Google Chrome (Canary build) supprts MathML.

The version of Google Chrome Canary build is 24.0.1293.0 currently. The version of Google Chrome Stable build is 22.0.1229.94.

Firstly, I wrote MathML file with LibreOffice formula editor and saved as a MathML 1.01 file (math.mml). I think that MathML is not much better format by hand than LaTeX.

The gist is here.

When I opened this mml file with Google Chrome Stable, it was shown as below. Chrome knew the file is XML document but couldn't read MathML.

When I opened it with Google Chrome Canary, it was shown as below. Chrome could read MathML approximately right.

MathML can be written in XHTML document. After this, the web browser can interpret numeral information and mathematical formula. For example when I open a web page providing statistical information, the web browser will be able to show me the richer information.

Thanks.

measuring the similarity between two images
2012-09-08T21:52:00.000+09:00

When I searched methods for measuring the similarity between two images, I found two implementations: libpuzzle and pHash. libpuzzle specializes in the comparison of images. pHash supports the comparison of images, audios and movies.

These can be installed with homebrew. I compared these libraries.

libpuzzle

The license of libpuzzle is BSD license. It depends on gd library.

$ brew info libpuzzle
libpuzzle: stable 0.11
http://libpuzzle.pureftpd.org/project/libpuzzle
Depends on: gd
Not installed
https://github.com/mxcl/homebrew/commits/master/Library/Formula/libpuzzle.rb

$ brew install libpuzzle
Warning: It appears you have MacPorts or Fink installed.
Software installed with other package managers causes known problems for
Homebrew. If a formula fails to build, uninstall MacPorts/Fink and try again.
==> Installing libpuzzle dependency: libpng
==> Downloading http://downloads.sf.net/project/libpng/libpng15/1.5.12/libpng-1.5.12.tar.gz
######################################################################## 100.0%
==> ./configure --prefix=/opt/homebrew/Cellar/libpng/1.5.12
==> make install
/opt/homebrew/Cellar/libpng/1.5.12: 14 files, 1.0M, built in 15 seconds
==> Installing libpuzzle
==> Downloading http://download.pureftpd.org/pub/pure-ftpd/misc/libpuzzle/releases/libpuzzle-0.11.tar.bz2
######################################################################## 100.0%
==> ./configure --prefix=/opt/homebrew/Cellar/libpuzzle/0.11
==> make install
/opt/homebrew/Cellar/libpuzzle/0.11: 12 files, 104K, built in 17 seconds

I wrote a simple C code using libpuzzle.

$ gcc -L/opt/homebrew/lib -I/opt/homebrew/include -lpuzzle test_libpuzzle.c
$ otool -L ./a.out 
./a.out:
        /opt/homebrew/lib/libpuzzle.1.dylib (compatibility version 2.0.0, current version 2.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)

$ ./a.out tokyo_skytree.jpg tokyo_skytree.jpg
distance: 0

I prepared five pictures from flickr for testing. 

tokyo_skytree.jpg: original (960 × 1280)
tokyo_skytree_500.jpg: reduced (375 × 500)
tokyo_skytree_75.jpg: reduced and cut (75 × 75)
tokyo_skytree_thumbnail.jpg: reduced (75 × 100)
wrong_image.jpg: non-related (640 × 960)

$ time for i in `/bin/ls *.jpg`; do  for j in `/bin/ls *.jpg`; do  echo -ne "$i <-> $j\t:\t"; ./a.out $i $j; done; done

tokyo_skytree.jpg <-> tokyo_skytree.jpg :       distance: 0
tokyo_skytree.jpg <-> tokyo_skytree_500.jpg     :       distance: 0.0981159
tokyo_skytree.jpg <-> tokyo_skytree_75.jpg      :       distance: 0.549193
tokyo_skytree.jpg <-> tokyo_skytree_thumbnail.jpg       :       distance: 0.290368
tokyo_skytree.jpg <-> wrong_image.jpg   :       distance: 0.710145
tokyo_skytree_500.jpg <-> tokyo_skytree.jpg     :       distance: 0.0981159
tokyo_skytree_500.jpg <-> tokyo_skytree_500.jpg :       distance: 0
tokyo_skytree_500.jpg <-> tokyo_skytree_75.jpg  :       distance: 0.547736
tokyo_skytree_500.jpg <-> tokyo_skytree_thumbnail.jpg   :       distance: 0.272873
tokyo_skytree_500.jpg <-> wrong_image.jpg       :       distance: 0.715105
tokyo_skytree_75.jpg <-> tokyo_skytree.jpg      :       distance: 0.549193
tokyo_skytree_75.jpg <-> tokyo_skytree_500.jpg  :       distance: 0.547736
tokyo_skytree_75.jpg <-> tokyo_skytree_75.jpg   :       distance: 0
tokyo_skytree_75.jpg <-> tokyo_skytree_thumbnail.jpg    :       distance: 0.56704
tokyo_skytree_75.jpg <-> wrong_image.jpg        :       distance: 0.716492
tokyo_skytree_thumbnail.jpg <-> tokyo_skytree.jpg       :       distance: 0.290368
tokyo_skytree_thumbnail.jpg <-> tokyo_skytree_500.jpg   :       distance: 0.272873
tokyo_skytree_thumbnail.jpg <-> tokyo_skytree_75.jpg    :       distance: 0.56704
tokyo_skytree_thumbnail.jpg <-> tokyo_skytree_thumbnail.jpg     :       distance: 0
tokyo_skytree_thumbnail.jpg <-> wrong_image.jpg :       distance: 0.756318
wrong_image.jpg <-> tokyo_skytree.jpg   :       distance: 0.710145
wrong_image.jpg <-> tokyo_skytree_500.jpg       :       distance: 0.715105
wrong_image.jpg <-> tokyo_skytree_75.jpg        :       distance: 0.716492
wrong_image.jpg <-> tokyo_skytree_thumbnail.jpg :       distance: 0.756318
wrong_image.jpg <-> wrong_image.jpg     :       distance: 0

real    0m1.118s
user    0m0.963s
sys     0m0.100s

      orig

      500

      75

      thumbnail

      wrong

      orig

      0

      0.0981159

      0.549193

      0.290368

      0.710145

      500

      0.0981159

      0

      0.547736

      0.272873

      0.715105

      75

      0.549193

      0.547736

      0

      0.56704

      0.716492

      thumbnail

      0.290368

      0.272873

      0.56704

      0

      0.756318

      wrong

      0.710145

      0.715105

      0.716492

      0.756318

      0

If the two images are similar, the distance approaches 0 (if these are different, the distance approaches 1). When the different images are compared, it appears that the distance is more than 0.7.

phash

The license of pHash is GPLv3. It depends on CImg for processing images.

$ brew info phash
phash: stable 0.9.4
http://www.phash.org/
Depends on: cimg, ffmpeg, libsndfile, libsamplerate, mpg123
Not installed
https://github.com/mxcl/homebrew/commits/master/Library/Formula/phash.rb
==> Options
--disable-audio-hash
        Disable audio hash
--disable-video-hash
        Disable video hash
--disable-image-hash
        Disable image hash

$ brew install phash --disable-audio-hash --disable-video-hash
Warning: It appears you have MacPorts or Fink installed.
Software installed with other package managers causes known problems for
Homebrew. If a formula fails to build, uninstall MacPorts/Fink and try again.
==> Installing phash dependency: cimg
==> Downloading http://downloads.sourceforge.net/cimg/CImg-1.5.0.zip
######################################################################## 100.0%
/opt/homebrew/Cellar/cimg/1.5.0: 425 files, 31M, built in 6 seconds
==> Installing phash
==> Downloading http://www.phash.org/releases/pHash-0.9.4.tar.gz
######################################################################## 100.0%
==> Patching
patching file src/cimgffmpeg.cpp
==> ./configure --prefix=/opt/homebrew/Cellar/phash/0.9.4 --enable-shared --disable-video-hash --disable-audio-hash
==> make install
/opt/homebrew/Cellar/phash/0.9.4: 10 files, 900K, built in 47 seconds

I wrote a simple C++ code using pHash.

$ g++ -L/opt/homebrew/lib -I/opt/homebrew/include -lpHash test_phash.cpp
$ otool -L a.out 
a.out:
        /opt/homebrew/lib/libpHash.0.dylib (compatibility version 1.0.0, current version 1.0.0)
        /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 56.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)

$ ./a.out tokyo_skytree.jpg tokyo_skytree.jpg
distance: 0

$ time for i in `/bin/ls *.jpg`; do  for j in `/bin/ls *.jpg`; do  echo -ne "$i <-> $j\t:\t"; ./a.out $i $j; done; done

tokyo_skytree.jpg <-> tokyo_skytree.jpg :       distance: 0
tokyo_skytree.jpg <-> tokyo_skytree_500.jpg     :       distance: 8
tokyo_skytree.jpg <-> tokyo_skytree_75.jpg      :       distance: 22
tokyo_skytree.jpg <-> tokyo_skytree_thumbnail.jpg       :       distance: 8
tokyo_skytree.jpg <-> wrong_image.jpg   :       distance: 30
tokyo_skytree_500.jpg <-> tokyo_skytree.jpg     :       distance: 8
tokyo_skytree_500.jpg <-> tokyo_skytree_500.jpg :       distance: 0
tokyo_skytree_500.jpg <-> tokyo_skytree_75.jpg  :       distance: 20
tokyo_skytree_500.jpg <-> tokyo_skytree_thumbnail.jpg   :       distance: 2
tokyo_skytree_500.jpg <-> wrong_image.jpg       :       distance: 30
tokyo_skytree_75.jpg <-> tokyo_skytree.jpg      :       distance: 22
tokyo_skytree_75.jpg <-> tokyo_skytree_500.jpg  :       distance: 20
tokyo_skytree_75.jpg <-> tokyo_skytree_75.jpg   :       distance: 0
tokyo_skytree_75.jpg <-> tokyo_skytree_thumbnail.jpg    :       distance: 20
tokyo_skytree_75.jpg <-> wrong_image.jpg        :       distance: 32
tokyo_skytree_thumbnail.jpg <-> tokyo_skytree.jpg       :       distance: 8
tokyo_skytree_thumbnail.jpg <-> tokyo_skytree_500.jpg   :       distance: 2
tokyo_skytree_thumbnail.jpg <-> tokyo_skytree_75.jpg    :       distance: 20
tokyo_skytree_thumbnail.jpg <-> tokyo_skytree_thumbnail.jpg     :       distance: 0
tokyo_skytree_thumbnail.jpg <-> wrong_image.jpg :       distance: 32
wrong_image.jpg <-> tokyo_skytree.jpg   :       distance: 30
wrong_image.jpg <-> tokyo_skytree_500.jpg       :       distance: 30
wrong_image.jpg <-> tokyo_skytree_75.jpg        :       distance: 32
wrong_image.jpg <-> tokyo_skytree_thumbnail.jpg :       distance: 32
wrong_image.jpg <-> wrong_image.jpg     :       distance: 0

real    0m3.215s
user    0m3.007s
sys     0m0.173s

      orig

      500

      75

      thumbnail

      wrong

      orig

      0

      8

      22

      8

      30

      500

      8      

      0      

      20      

      2      

      30      

      75      

      22      

      20      

      0      

      20      

      32      

      thumbnail

      8      

      2      

      20      

      0      

      32      

      wrong

      30      

      30      

      32      

      32      

      0      

If the two images are similar, the distance approaches 0 (if these are different, the distance is large number). When the different images are compared, it appears that the distance is more than 30.

The result of libpuzzle and pHash are similar. But the processing time of pHash takes 3 times longer than libpuzzle. I will use libpuzzle for my task.

Thanks.

getting unused port number
2012-06-16T17:20:00.001+09:00

A simple way to get unused port number on local machine is creating a socket.

I wrote a Python code.

But in Jython, this behavior is a bit different.

I think that the behavior of Python is correct.

Thanks.

verifying keys with openssl
2012-06-01T23:15:00.000+09:00

For verifying that a private key and a public key are right combination, we can use Modules field of the keys.

encryption - How do you test a public/private keypair? - Stack Overflow

I got a private key of Yahoo! from Yahoo! Axis. And the Axis extension for Google Chrome includes a public key field.

I checked the two keys were right combination.

$ python check_keys.py 
warning [YAxis_Chrome_v1_0_20120520.crx]:  306 extra bytes at beginning or within zipfile
  (attempting to process anyway)
Public key modulus:   D609E9C990D38EFADFB21AFBD227EDBAFCFE4C2E6A9ECE84B0A95DF9914E93DC832B7A70BF07FFE1D29317544EE57B6F249E740707919C364340FFD05F1EB2E2A5C321D8447BFBC4D18FA97C5E6ABB24AF3D60F0C37E46D2C820522EE2BE1ECB0D528B7A2A9C9309EC1E542A06BCFAFF2A3239C0F739F8C54E7BD9C5B52BEA65
Private key modulus:  D609E9C990D38EFADFB21AFBD227EDBAFCFE4C2E6A9ECE84B0A95DF9914E93DC832B7A70BF07FFE1D29317544EE57B6F249E740707919C364340FFD05F1EB2E2A5C321D8447BFBC4D18FA97C5E6ABB24AF3D60F0C37E46D2C820522EE2BE1ECB0D528B7A2A9C9309EC1E542A06BCFAFF2A3239C0F739F8C54E7BD9C5B52BEA65
match!

Unfortunately (?), the keys are right. Thus, we can create certified extensions as Yahoo!.

It seems that the Axis development team uses ant for building the extension. Because the sources and product were in same location, an incident like this occurred. There may not be a person with much knowledge about Google Chrome extensions in the team.

Thanks.

Using socket API with Google Chrome Canary build
2012-05-20T15:40:00.000+09:00

I knew that Google Chrome Canary build supports chrome.experimental.socket API.

I searched some sample codes on the Web, but these can't run on the current version. I don't know whether these codes will be right on the future version. First of all, I wrote executable codes on the current version according to documents and test codes.

laclefyoshi/chrome_socket_sample

This extension is a UDP socket client with chrome.experimental.socket API. I checked this on Google Chrome canary 21.0.1137.1 canary and 21.0.1143.0 canary for Mac. We can send string messages to a given UDP server. udpserver.py is a simple UDP server written in Python.

When the send button pushes, the below function calls.

In the document, the type of second argument of chrome.experimental.socket.write(..) is String. But actually, it is ArrayBuffer.

udpserver.py displays the client info and the sent message.

$ python udpserver.py
127.0.0.1:49623 wrote: Hello!

And the logs are left on the JavaScript console.

We can use socket API directly with Developer tools.

> udp_socket_writer("127.0.0.1", 9998, "Hello!")
undefined
  ▼Object
    bytesWritten: 6

> udp_socket_writer("127.0.0.1", 9998, "こんにちは、ソケット")  // multibyte string is OK, too
undefined
  ▼Object
    bytesWritten: 30

I haven't yet checked server mode and TCP connection. When the socket API becomes stable, I want to make IRC client running on Google Chrome. P2P communication between Google Chromes  isn't a dream.

Thanks.

flymake and checkstyle
2012-04-15T14:37:00.000+09:00

In our Java project, Checkstyle Plugin for Jenkins is used. Because I had never cared about Java coding style too much, I receive warning mails every commitment to svn.

So. I wrote .emacs for displaying warnings of checkstyle with flymake on emacs.

Firstly, I wrote shell script for calling checkstyle as a command. I put this script into /opt/bin/checkstyle.sh.

Finally, I wrote .emacs. I use flymake-cursor.el for displaying messages on minibuffer.

However, I didn't know how importunate coding style is.

Thanks.

defining custom URL using URLStreamHandler
2012-04-01T11:14:00.000+09:00

On Mac, iOS and Android, we can define custom URL schemes and behaviors for their URL. In Java, java.net.URL can manage every URL and we can add/change the behavior by defining and setting java.net.URLStreamHandler.

I wrote a handler for URL which has a protocol: sum:// in Scala. For example, an URL: sum://1,2,3 has a content: 6 (= 1 + 2 + 3).

This completes a registration of a protocol: sum://. We can use sum:// as like regular URL.

scala> :load "SumURLHandler.scala"
scala> val url:URL = new URL("sum://1,2,3,4,5,6,7,8/foo/bar?a=A&b=B#head")

scala > println("ptorotol: %s\nhost: %s\npath: %s\nquery: %s\nref: %s".format(
     | url.getProtocol, url.getHost, url.getPath, url.getQuery, url.getRef))

ptorotol: sum
host: 1,2,3,4,5,6,7,8
path: /foo/bar
query: a=A&b=B
ref: head

scala> val url:URL = new URL("sum://1,2,3,4,5,6,7,8")
scala> val isr:InputStreamReader = new InputStreamReader(url.openStream)
scala> var result:String = ""
scala> while(isr.ready()) {
     | result += isr.read.toChar
     |}

scala> println(result)
36

In Jython:

By using of elements of URL wisely, we can define useful URL in own applications.

Thanks.

identifying whether a binary is 32-bit or 64-bit in itself
2012-03-24T22:22:00.000+09:00

For example, when I use a C library with 32-bit Python interpreter on a 64-bit machine, I need to compile the library as 32-bit binary. 

file command provides such information.

$ file /usr/lib/libpython2.6.so.1.0
/usr/lib/libpython2.6.so.1.0: ELF 32-bit LSB shared object, ARM, version 1 (SYSV), dynamically linked, stripped

$ file /usr/lib/libpython2.6.dylib 
/usr/lib/libpython2.6.dylib: Mach-O universal binary with 2 architectures
/usr/lib/libpython2.6.dylib (for architecture x86_64):  Mach-O 64-bit dynamically linked shared library x86_64
/usr/lib/libpython2.6.dylib (for architecture i386):    Mach-O dynamically linked shared library i386

But here I show some methods for identifying the binary's bit-mode inside program.

C

gcc (llvm-gcc) can build a binary in the 32-bit or 64-bit mode. The choice can be made by adding compile option: -m32 or -m64.

The difference of modes comes in size of pointer. If size of pointer is 4, the binary or interpreter is 32-bit mode. If size of pointer is 8, the binary or interpreter is 64-bit mode. That is, the size of pointer shows memory bandwidth (byte unit).

$ gcc -m32 mode.c
$ file ./a.out 
./a.out: Mach-O executable i386
$ ./a.out 
4

$ gcc -m64 mode.c 
$ file ./a.out 
./a.out: Mach-O 64-bit executable x86_64
$ ./a.out 
8

Python

Python can also get size of pointer with struct module.

$ file `which python2.6`
/usr/bin/python2.6: ELF 32-bit LSB executable, ARM, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, stripped
$ python mode.py
4

$ file `which python2.7`
/usr/bin/python2.7: Mach-O universal binary with 2 architectures
/usr/bin/python2.7 (for architecture x86_64):   Mach-O 64-bit executable x86_64
/usr/bin/python2.7 (for architecture i386):     Mach-O executable i386
$ python mode.py
8

Perl

Perl doesn't have a function for showing size of pointer. But Config module shows it.

$ file `which perl`
/usr/bin/perl: ELF 32-bit LSB executable, ARM, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, stripped
$ perl mode.pl
4

$ file `which perl`
/usr/bin/perl: Mach-O universal binary with 2 architectures
/usr/bin/perl (for architecture x86_64):        Mach-O 64-bit executable x86_64
/usr/bin/perl (for architecture i386):  Mach-O executable i386
$ perl mode.pl 
8

Java / Jython

We usually don't treat pointer in Java. But Java Virtual Machine (JVM) has a value of size of pointer for handling data effectively.

$ javac Mode.java

$ java -d32 Mode  # option:-d32 specifies that JVM runs in 32-bit mode
32

$ java -d64 Mode  # option:-d64 specifies that JVM runs in 64-bit mode
64

$ jython -J-d32  # option:-J-d32 has the same effect as a option:-d32 of java
Jython 2.5.2 (Release_2_5_2:7206, Mar 2 2011, 23:12:06) 
[Java HotSpot(TM) Client VM (Apple Inc.)] on java1.6.0_29
Type "help", "copyright", "credits" or "license" for more information.
>>> from java.lang import System
>>> System.getProperty("sun.arch.data.model")
u'32'

$ jython -J-d64  # option:-J-d64 has the same effect as a option:-d64 of java
Jython 2.5.2 (Release_2_5_2:7206, Mar 2 2011, 23:12:06) 
[Java HotSpot(TM) 64-Bit Server VM (Apple Inc.)] on java1.6.0_29
Type "help", "copyright", "credits" or "license" for more information.
>>> from java.lang import System
>>> System.getProperty("sun.arch.data.model")
u'64'

Thanks.

class for manipulating URL
2012-03-17T22:10:00.001+09:00

A problem of java.net.URL is that is doesn't provide APIs for rewriting components of URL. For example, When I need to normalize hostname of URL (hostname can be wholly uncapitalized), I must get the components of URL, make a String instance as hostname, and make a new URL instance.

So I wrote MyURL class for manipulating URL.

As central feature of this class, query string of URL is held as list of pairs. This makes it easy for us to search query parameters.

I don't provide Pair<L, R> class. But there are many examples on the Web.

Thanks.

open file and search term in vim and emacs
2012-02-25T19:24:00.001+09:00

vim has command line option for searching text when vim opens a file.

$ vim /usr/share/emacs/22.1/etc/TODO +/iff
   # it opens TODO, searches "iff" and moves cursor the first appearance position

But emacs doesn't have the similar command line option apparently. And so, I imitated the vim's option by using -eval: a command line option for evaluating emacs lisp expressions when emacs starts.

$ /usr/bin/emacs  /usr/share/emacs/22.1/etc/TODO -eval '(search-forward-regexp "iff")'

There is a difference in these results. vim moves cursor to starting point of search term. On the other hand, emacs moves cursor to end point of search term. Additionally, vim remember the search term given by command line option, but emacs doesn't.

For improving the emacs's behavior, I need to write a little more complicated command. Doesn't emacs have a system for extending command line option by writing emacs lisp in init file?

Thanks.

another way of creating RAM disc in Mac
2012-02-18T21:38:00.000+09:00

I wrote about a way of creating RAM disc in Mac. But I knew that using discutil is better than mount for creating RAM disc in Mac.

Create a RAM Disk in Mac OS X

$ diskutil erasevolume HFS+ "ramdisk" `hdiutil attach -nomount ram://1165430`
Started erase on disk1
Unmounting disk
Erasing
Initialized /dev/rdisk1 as a 569 MB HFS Plus volume
Mounting disk
Finished erase on disk1 ramdisk

The good thing is the disc is visible on Finder. Of course, the disk free space can be checked easily.

Thanks.

af* command for treating audio files in Mac
2012-02-11T17:06:00.001+09:00

I knew that there are commands for treating audio files in Mac by default.

afinfo

afinfo is a command for showing information of audio file.

$ afinfo happykids.wav 
File:           happykids.wav
File type ID:   WAVE
Data format:     1 ch,  11025 Hz, 'lpcm' (0x00000008) 8-bit unsigned integer
                no channel layout.
estimated duration: 5.576417 sec
audio bytes: 61480
audio packets: 61480
bit rate: 88200 bits per second
packet size upper bound: 1
audio data file offset: 44
optimized
source bit depth: I8
----

afconvert

afconvert is a command for converting format of audio file. The supported formats are showed by option: -hf

$ afconvert -hf
Audio file and data formats:
    '3gpp' = 3GP Audio (.3gp)
               data_formats: 'Qclp' 'aac ' 'aace' 'aach' 'aacl' 'aacp' 
                             'samr' 
    '3gp2' = 3GPP-2 Audio (.3g2)
               data_formats: 'Qclp' 'aac ' 'aace' 'aach' 'aacl' 'aacp' 
                             'samr' 
    'adts' = AAC ADTS (.aac, .adts)
               data_formats: 'aac ' 'aach' 'aacp' 
    'ac-3' = AC3 (.ac3)
               data_formats: 'ac-3' 
    'AIFC' = AIFC (.aifc, .aiff, .aif)
               data_formats: I8 BEI16 BEI24 BEI32 BEF32 BEF64 UI8 'ulaw' 
                             'alaw' 'MAC3' 'MAC6' 'ima4' 'QDMC' 'QDM2' 
                             'Qclp' 'agsm' 
    'AIFF' = AIFF (.aiff, .aif)
               data_formats: I8 BEI16 BEI24 BEI32 
    'amrf' = AMR (.amr)
               data_formats: 'samr' 
    'm4af' = Apple MPEG-4 Audio (.m4a, .m4r)
               data_formats: 'aac ' 'aace' 'aach' 'aacl' 'aacp' 'alac' 
    'caff' = CAF (.caf)
               data_formats: '.mp1' '.mp2' '.mp3' 'QDM2' 'QDMC' 'Qclp' 
                             'Qclq' 'aac ' 'aace' 'aach' 'aacl' 'aacp' 
                             'alac' 'alaw' 'dvi8' 'ilbc' 'ima4' I8 BEI16 
                             BEI24 BEI32 BEF32 BEF64 LEI16 LEI24 LEI32 
                             LEF32 LEF64 'ms\x00\x02' 'ms\x00\x11' 'ms\x001' 
                             'paac' 'samr' 'ulaw' 
    'MPG1' = MPEG Layer 1 (.mp1, .mpeg, .mpa)
               data_formats: '.mp1' 
    'MPG2' = MPEG Layer 2 (.mp2, .mpeg, .mpa)
               data_formats: '.mp2' 
    'MPG3' = MPEG Layer 3 (.mp3, .mpeg, .mpa)
               data_formats: '.mp3' 
    'mp4f' = MPEG-4 Audio (.mp4)
               data_formats: 'aac ' 'aace' 'aach' 'aacl' 'aacp' 
    'NeXT' = NeXT/Sun (.snd, .au)
               data_formats: I8 BEI16 BEI24 BEI32 BEF32 BEF64 'ulaw' 
    'Sd2f' = Sound Designer II (.sd2)
               data_formats: I8 BEI16 BEI24 BEI32 
    'WAVE' = WAVE (.wav)
               data_formats: UI8 LEI16 LEI24 LEI32 LEF32 LEF64 'ulaw' 
                             'alaw' 

There is "MPG3" in a list of supported formats, but Mac doesn't provide any MP3 encoder.

$ afconvert -f MPG3 -d '.mp3' happykids.wav 
Error: ExtAudioFileSetProperty ('cfmt') failed ('fmt?')

$ afconvert -f m4af -d aac happykids.wav

$ afinfo happykids.m4a 
File:           happykids.m4a
File type ID:   m4af
Data format:     1 ch,  11025 Hz, 'aac ' (0x00000000) 0 bits/channel, 0 bytes/packet, 1024 frames/packet, 0 bytes/frame
Channel layout: Mono
estimated duration: 5.576417 sec
audio bytes: 15950
audio packets: 63
audio 61480 valid frames + 2112 priming + 920 remainder = 64512
bit rate: 21806 bits per second
packet size upper bound: 399
audio data file offset: 4096
optimized
format list:
[ 0] format:     1 ch,  11025 Hz, 'aac ' (0x00000000) 0 bits/channel, 0 bytes/packet, 1024 frames/packet, 0 bytes/frame
     Channel layout: Mono
source bit depth: I8
----

afplay

afplay is a command for playback of audio file. Probably, the supported formats are same as afconvert.

I found some weblogs which said that the sound quality of afplay is better than iTunes. But I can't tell the difference.

Thanks.

Removing margins of PDF generated by iBooks Author
2012-01-30T22:20:00.000+09:00

iBooks Author is very solicitous and beautiful Desktop publishing tool. But I have a complaint about this. It is that margins and a watermark break into PDF generated by iBooks Author.

Probably, Apple indicate PDF generated by iBooks Author should not be published because they think that the function of exporting PDF should be used for confirming the content and layout on Mac when we don't have iPad accidentally.

But, I want to make ebooks with iBooks Author and publish them in good-looking and widely-used PDF on the Web, too.

I tried to remove margins from PDF generated iBooks Author in two ways.

pdfcrop

pdfcrop is a software for removing margins from PDF. This is supplied with TeX Live.

pdfcrop automatically recognizes margins in PDF. But pdfcrop without options yields a disappointing result from PDF genetated by iBooks Author because there is a watermark " iBooks Author" in the bottom of it. Therefore, we need to add options for specifying the size of the bottom margin.

$ time pdfcrop --margins "0 0 0 -55" ibooks_pdf.pdf ibooks_pdf_cropped.pdf
PDFCROP 1.32, 2011/08/10 - Copyright (c) 2002-2011 by Heiko Oberdiek.
==> 4 pages written on `ibooks_pdf_cropped.pdf'.

real    2m51.121s
user    2m43.072s
sys     0m0.714s

It took a few minutes to remove margins of 4 pages of PDF.

$ identify ibooks_pdf.pdf
ibooks_pdf.pdf[0] PDF 1024x748 1024x748+0+0 16-bit Bilevel DirectClass 95.8KB 0.010u 0:00.009
ibooks_pdf.pdf[1] PDF 1024x748 1024x748+0+0 16-bit Bilevel DirectClass 95.8KB 0.000u 0:00.009
ibooks_pdf.pdf[2] PDF 1024x748 1024x748+0+0 16-bit Bilevel DirectClass 95.8KB 0.000u 0:00.009
ibooks_pdf.pdf[3] PDF 1024x748 1024x748+0+0 16-bit Bilevel DirectClass 95.8KB 0.000u 0:00.009

$ identify ibooks_pdf_cropped.pdf
ibooks_pdf_cropped.pdf[0] PDF 922x598 922x598+0+0 16-bit Bilevel DirectClass 69.4KB 0.000u 0:00.019
ibooks_pdf_cropped.pdf[1] PDF 922x598 922x598+0+0 16-bit Bilevel DirectClass 69.4KB 0.000u 0:00.019
ibooks_pdf_cropped.pdf[2] PDF 922x598 922x598+0+0 16-bit Bilevel DirectClass 69.4KB 0.000u 0:00.019
ibooks_pdf_cropped.pdf[3] PDF 922x598 922x598+0+0 16-bit Bilevel DirectClass 69.4KB 0.000u 0:00.019

The result is good, but character information of all pages are lost. Thus, we can't select text on the processed PDF because all pages are imaged.

PDF hack

PDF can have information for the paper size (MediaBox) and the display size (CropBox). There is MediaBox in PDF generated by iBooks Author, but there is not CropBox. I added CropBox into PDF with shell command. The size was decided by the result of being processed by pdfcrop.

$ time sed -e 's;/MediaBox;/CropBox [52 75 973 673] /MediaBox;g' ibooks_pdf.pdf > ibooks_pdf_cropped.pdf

real    0m0.004s
user    0m0.002s
sys     0m0.003s

The process was complete in a moment. I confirmed that the processed PDF can be rendered with Preview.app and Google Chrome (Chrome PDF Viewer plugin). But it seems that some application can't reder the processed PDF.

$ identify ibooks_pdf_cropped.pdf 
   **** Warning:  An error occurred while reading an XREF table.
   **** The file has been damaged.  This may have been caused
   **** by a problem while converting or transfering the file.
   **** Ghostscript will attempt to recover the data.

   **** This file had errors that were repaired or ignored.
   **** The file was produced by: 
   **** >>>> Mac OS X 10.7.2 Quartz PDFContext <<<<
   **** Please notify the author of the software that produced this
   **** file that it does not conform to Adobe's published PDF
   **** specification.

ibooks_pdf_cropped.pdf[0] PDF 1024x748 1024x748+0+0 16-bit Bilevel DirectClass 95.8KB 0.010u 0:00.019
ibooks_pdf_cropped.pdf[1] PDF 1024x748 1024x748+0+0 16-bit Bilevel DirectClass 95.8KB 0.000u 0:00.019
ibooks_pdf_cropped.pdf[2] PDF 1024x748 1024x748+0+0 16-bit Bilevel DirectClass 95.8KB 0.000u 0:00.009
ibooks_pdf_cropped.pdf[3] PDF 1024x748 1024x748+0+0 16-bit Bilevel DirectClass 95.8KB 0.000u 0:00.009

I wrote AppleScript droplet and created the project for simplifying the process to crop PDF by sed.

laclefyoshi / cropping_ibooksauthor_pdf / overview — Bitbucket

Thanks.

Outputting data for people with tablib
2012-01-21T12:44:00.000+09:00

In order to show some data in processing to others (that are boss or client in business), I think it tiresome to copy and paste data into Excel sheets or export HTML tables.

I found tablib that is a library for formatting data as tabular form or semistructured form.

== show attributes of Dataset ==
['_Dataset__headers', '__class__', '__delattr__', '__delitem__', '__dict__', '__doc__', '__format__', '__getattribute__', '__getitem__', '__hash__', '__init__', '__len__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setitem__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_clean_col', '_data', '_formatters', '_get_dict', '_get_headers', '_package', '_register_formats', '_separators', '_set_dict', '_set_headers', '_validate', 'add_formatter', 'append', 'append_col', 'append_separator', 'csv', 'dict', 'filter', 'headers', 'height', 'html', 'insert', 'insert_col', 'insert_separator', 'json', 'lpop', 'lpush', 'lpush_col', 'ods', 'pop', 'rpop', 'rpush', 'rpush_col', 'sort', 'stack', 'stack_cols', 'title', 'transpose', 'tsv', 'width', 'wipe', 'xls', 'xlsx', 'yaml']

== output as dict ==
[OrderedDict([('name', 'Alice'), ('age', 21)]), OrderedDict([('name', 'Bob'), ('age', 20)]), OrderedDict([('name', 'Carol'), ('age', 31)]), OrderedDict([('name', 'Taro'), ('age', 10)])]

== output as json ==
[{"name": "Alice", "age": 21}, {"name": "Bob", "age": 20}, {"name": "Carol", "age": 31}, {"name": "Taro", "age": 10}]

== output only female as json ==
[{"name": "Alice", "age": 21}, {"name": "Carol", "age": 31}]

== output as csv ==
name,age,e/o
Alice,21,odd
Bob,20,even
Carol,31,odd
Taro,10,even

== show attributes of Databook ==
['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_datasets', '_package', '_register_formats', 'add_sheet', 'html', 'json', 'ods', 'size', 'wipe', 'xls', 'xlsx', 'yaml']

== output 2 sheets as yaml ==
- !!python/object/apply:collections.OrderedDict
  - - - data
      - - !!python/object/apply:collections.OrderedDict
          - - [name, Alice]
            - [age, 21]
            - [e/o, odd]
        - !!python/object/apply:collections.OrderedDict
          - - [name, Bob]
            - [age, 20]
            - [e/o, even]
        - !!python/object/apply:collections.OrderedDict
          - - [name, Carol]
            - [age, 31]
            - [e/o, odd]
        - !!python/object/apply:collections.OrderedDict
          - - [name, Taro]
            - [age, 10]
            - [e/o, even]
    - [title, null]
- !!python/object/apply:collections.OrderedDict
  - - - data
      - - !!python/object/apply:collections.OrderedDict
          - - [name, Alice]
            - [age, 21]
            - [e/o, odd]
        - !!python/object/apply:collections.OrderedDict
          - - [name, Bob]
            - [age, 20]
            - [e/o, even]
        - !!python/object/apply:collections.OrderedDict
          - - [name, Carol]
            - [age, 31]
            - [e/o, odd]
        - !!python/object/apply:collections.OrderedDict
          - - [name, Taro]
            - [age, 10]
            - [e/o, even]
    - [title, null]

== output 2 sheets as html ==
<h3>Set 0</h3>
<table>
<thead>
<tr><th>name</th>
<th>age</th>
<th>e/o</th></tr>
</thead>
<tr><td>Alice</td>
<td>21</td>
<td>odd</td></tr>
<tr><td>Bob</td>
<td>20</td>
<td>even</td></tr>
<tr><td>Carol</td>
<td>31</td>
<td>odd</td></tr>
<tr><td>Taro</td>
<td>10</td>
<td>even</td></tr>
</table>
<h3>Set 1</h3>
<table>
<thead>
<tr><th>name</th>
<th>age</th>
<th>e/o</th></tr>
</thead>
<tr><td>Alice</td>
<td>21</td>
<td>odd</td></tr>
<tr><td>Bob</td>
<td>20</td>
<td>even</td></tr>
<tr><td>Carol</td>
<td>31</td>
<td>odd</td></tr>
<tr><td>Taro</td>
<td>10</td>
<td>even</td></tr>
</table>

Unfortunately, tablib can't be used in Jython. According to error message, this problem is caused by grammer.py in odfpy.

$ jython
Jython 2.5.2 (Release_2_5_2:7206, Mar 2 2011, 23:12:06)
[Java HotSpot(TM) 64-Bit Server VM (Apple Inc.)] on java1.6.0_29
Type "help", "copyright", "credits" or "license" for more information.
>>> import tablib
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "tablib/__init__.py", line 3, in <module>
    from tablib.core import (
  File "tablib/core.py", line 15, in <module>
    from tablib import formats
  File "tablib/formats/__init__.py", line 6, in <module>
    from . import _csv as csv
  File "tablib/formats/_csv.py", line 6, in <module>
    from tablib.compat import is_py3, csv, StringIO
  File "tablib/compat.py", line 45, in <module>
    from tablib.packages.odf import opendocument, style, text, table
  File "tablib/packages/odf/opendocument.py", line 26, in <module>
    import manifest, meta
  File "tablib/packages/odf/manifest.py", line 24, in <module>
    from element import Element
  File "tablib/packages/odf/element.py", line 28, in <module>
    import grammar
java.lang.ClassFormatError: Invalid method Code length 370403 in class file tablib/packages/odf/grammar$py
 at java.lang.ClassLoader.defineClass1(Native Method)
 at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
 at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
 at org.python.core.BytecodeLoader$Loader.loadClassFromBytes(BytecodeLoader.java:119)

...

java.lang.ClassFormatError: java.lang.ClassFormatError: Invalid method Code length 370403 in class file tablib/packages/odf/grammar$py

When I disabled importing pyodf, tablib can be used in Jython.

$ python
>>> import tablib
>>> print tablib.formats.available
(<module 'tablib.formats._json' from 'tablib/formats/_json.py'>, <module 'tablib.formats._xls' from 'tablib/formats/_xls.py'>, <module 'tablib.formats._yaml' from 'tablib/formats/_yaml.py'>, <module 'tablib.formats._csv' from 'tablib/formats/_csv.py'>, <module 'tablib.formats._tsv' from 'tablib/formats/_tsv.py'>, <module 'tablib.formats._html' from 'tablib/formats/_html.py'>, <module 'tablib.formats._xlsx' from 'tablib/formats/_xlsx.py'>, <module 'tablib.formats._ods' from 'tablib/formats/_ods.py'>)

$ jython
>>> import tablib
>>> print tablib.formats.available
(<module 'tablib.formats._json' from 'tablib/formats/_json.py'>, <module 'tablib.formats._xls' from 'tablib/formats/_xls.py'>, <module 'tablib.formats._yaml' from 'tablib/formats/_yaml.py'>, <module 'tablib.formats._csv' from 'tablib/formats/_csv.py'>, <module 'tablib.formats._tsv' from 'tablib/formats/_tsv.py'>, <module 'tablib.formats._html' from 'tablib/formats/_html.py'>, <module 'tablib.formats._xlsx' from 'tablib/formats/_xlsx.py'>)

Thanks.

	orig	500	75	thumbnail	wrong
orig	0	0.0981159	0.549193	0.290368	0.710145
500	0.0981159	0	0.547736	0.272873	0.715105
75	0.549193	0.547736	0	0.56704	0.716492
thumbnail	0.290368	0.272873	0.56704	0	0.756318
wrong	0.710145	0.715105	0.716492	0.756318	0